<div dir="ltr">And fixed, puppet made a change in sudo file.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On 25 January 2018 at 17:16, CRiMSON <span dir="ltr"><<a href="mailto:crimson@unspeakable.org" target="_blank">crimson@unspeakable.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>I did a quick upgrade this afternoon on a dev machine.<br><br>Jan 25 11:57:07 Updated: glusterfs-libs-3.12.5-2.el7.<wbr>x86_64<br>Jan 25 11:57:08 Updated: glusterfs-client-xlators-3.12.<wbr>5-2.el7.x86_64<br>Jan 25 11:57:08 Updated: glusterfs-3.12.5-2.el7.x86_64<br>Jan 25 11:57:09 Updated: kernel-ml-tools-libs-4.14.15-<wbr>1.el7.elrepo.x86_64<br>Jan 25 11:57:09 Updated: kernel-ml-tools-4.14.15-1.el7.<wbr>elrepo.x86_64<br>Jan 25 11:57:10 Updated: glusterfs-api-3.12.5-2.el7.<wbr>x86_64<br>Jan 25 11:57:10 Updated: glusterfs-fuse-3.12.5-2.el7.<wbr>x86_64<br>Jan 25 11:57:10 Updated: glusterfs-cli-3.12.5-2.el7.<wbr>x86_64<br>Jan 25 11:57:11 Updated: python-perf-4.14.15-1.el7.<wbr>elrepo.x86_64<br>Jan 25 11:57:37 Installed: kernel-ml-devel-4.14.15-1.el7.<wbr>elrepo.x86_64<br>Jan 25 11:57:39 Updated: kernel-ml-headers-4.14.15-1.<wbr>el7.elrepo.x86_64<br>Jan 25 11:57:52 Installed: kernel-ml-4.14.15-1.el7.<wbr>elrepo.x86_64<br>Jan 25 11:57:52 Updated: rubygem-fluent-plugin-viaq_<wbr>data_model-0.0.13-1.el7.noarch<br><br></div>This is all that was upgraded.<br><br></div>But now my storage domains are failing to come up and the host keeps saying it's getting a connection refused. It's all on 1 host.<br><br></div><div>In mom.log I see.<br></div><div><br>2018-01-25 17:10:49,929 - mom - INFO - MOM starting<br>2018-01-25 17:10:49,955 - mom.HostMonitor - INFO - Host Monitor starting<br>2018-01-25 17:10:49,955 - mom - INFO - hypervisor interface vdsmjsonrpcbulk<br>2018-01-25 17:10:50,013 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused<br>2018-01-25 17:10:50,013 - mom - ERROR - Failed to initialize MOM threads<br>Traceback (most recent call last):<br> File "/usr/lib/python2.7/site-<wbr>packages/mom/__init__.py", line 29, in run<br> hypervisor_iface = self.get_hypervisor_interface(<wbr>)<br> File "/usr/lib/python2.7/site-<wbr>packages/mom/__init__.py", line 217, in get_hypervisor_interface<br> return module.instance(self.config)<br> File "/usr/lib/python2.7/site-<wbr>packages/mom/<wbr>HypervisorInterfaces/<wbr>vdsmjsonrpcbulkInterface.py", line 47, in instance<br> return JsonRpcVdsmBulkInterface()<br> File "/usr/lib/python2.7/site-<wbr>packages/mom/<wbr>HypervisorInterfaces/<wbr>vdsmjsonrpcbulkInterface.py", line 29, in __init__<br> super(<wbr>JsonRpcVdsmBulkInterface, self).__init__()<br> File "/usr/lib/python2.7/site-<wbr>packages/mom/<wbr>HypervisorInterfaces/<wbr>vdsmjsonrpcInterface.py", line 43, in __init__<br> .orRaise(RuntimeError, 'No connection to VDSM.')<br> File "/usr/lib/python2.7/site-<wbr>packages/mom/optional.py", line 28, in orRaise<br> raise exception(*args, **kwargs)<br>RuntimeError: No connection to VDSM.<br>[root@lv426 vdsm]# <br><br></div>My vdsm.log is 0 byes (nothing being logged?)<br><br></div>In my engine.log I'm seeing:<br><br>2018-01-25 17:12:11,027-05 INFO [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://lv426.dasgeekhaus.org/127.0.0.1" target="_blank">lv426.dasgeekhaus.org/127.0.0.<wbr>1</a><br>2018-01-25 17:12:11,028-05 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (EE-ManagedThreadFactory-<wbr>engineScheduled-Thread-99) [] Command 'GetCapabilitiesVDSCommand(<wbr>HostName = lv426, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{hostId='a645af84-3da1-<wbr>45ed-bab5-2af66b5924dd', vds='Host[lv426,a645af84-3da1-<wbr>45ed-bab5-2af66b5924dd]'})' execution failed: java.net.ConnectException: Connection refused<br>2018-01-25 17:12:11,028-05 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (EE-ManagedThreadFactory-<wbr>engineScheduled-Thread-99) [] Failure to refresh host 'lv426' runtime info: java.net.ConnectException: Connection refused<br>2018-01-25 17:12:13,517-05 INFO [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://lv426.dasgeekhaus.org/127.0.0.1" target="_blank">lv426.dasgeekhaus.org/127.0.0.<wbr>1</a><br>2018-01-25 17:12:13,517-05 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-<wbr>engineScheduled-Thread-25) [] Command 'GetAllVmStatsVDSCommand(<wbr>HostName = lv426, VdsIdVDSCommandParametersBase:<wbr>{hostId='a645af84-3da1-45ed-<wbr>bab5-2af66b5924dd'})' execution failed: java.net.ConnectException: Connection refused<br>2018-01-25 17:12:13,518-05 INFO [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>PollVmStatsRefresher] (EE-ManagedThreadFactory-<wbr>engineScheduled-Thread-25) [] Failed to fetch vms info for host 'lv426' - skipping VMs monitoring.<br><br></div>All of my network interfaces do come up ok, iptables is turned off so it's not getting in the way.<br><br></div>I'm at a complete loss right now as to what to look at.<br></div>
</blockquote></div><br></div>