Ovirt 4.2, failing to connect to VDSM.
I did a quick upgrade this afternoon on a dev machine. Jan 25 11:57:07 Updated: glusterfs-libs-3.12.5-2.el7.x86_64 Jan 25 11:57:08 Updated: glusterfs-client-xlators-3.12.5-2.el7.x86_64 Jan 25 11:57:08 Updated: glusterfs-3.12.5-2.el7.x86_64 Jan 25 11:57:09 Updated: kernel-ml-tools-libs-4.14.15-1.el7.elrepo.x86_64 Jan 25 11:57:09 Updated: kernel-ml-tools-4.14.15-1.el7.elrepo.x86_64 Jan 25 11:57:10 Updated: glusterfs-api-3.12.5-2.el7.x86_64 Jan 25 11:57:10 Updated: glusterfs-fuse-3.12.5-2.el7.x86_64 Jan 25 11:57:10 Updated: glusterfs-cli-3.12.5-2.el7.x86_64 Jan 25 11:57:11 Updated: python-perf-4.14.15-1.el7.elrepo.x86_64 Jan 25 11:57:37 Installed: kernel-ml-devel-4.14.15-1.el7.elrepo.x86_64 Jan 25 11:57:39 Updated: kernel-ml-headers-4.14.15-1.el7.elrepo.x86_64 Jan 25 11:57:52 Installed: kernel-ml-4.14.15-1.el7.elrepo.x86_64 Jan 25 11:57:52 Updated: rubygem-fluent-plugin-viaq_data_model-0.0.13-1.el7.noarch This is all that was upgraded. But now my storage domains are failing to come up and the host keeps saying it's getting a connection refused. It's all on 1 host. In mom.log I see. 2018-01-25 17:10:49,929 - mom - INFO - MOM starting 2018-01-25 17:10:49,955 - mom.HostMonitor - INFO - Host Monitor starting 2018-01-25 17:10:49,955 - mom - INFO - hypervisor interface vdsmjsonrpcbulk 2018-01-25 17:10:50,013 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused 2018-01-25 17:10:50,013 - mom - ERROR - Failed to initialize MOM threads Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run hypervisor_iface = self.get_hypervisor_interface() File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217, in get_hypervisor_interface return module.instance(self.config) File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 47, in instance return JsonRpcVdsmBulkInterface() File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 29, in __init__ super(JsonRpcVdsmBulkInterface, self).__init__() File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py", line 43, in __init__ .orRaise(RuntimeError, 'No connection to VDSM.') File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in orRaise raise exception(*args, **kwargs) RuntimeError: No connection to VDSM. [root@lv426 vdsm]# My vdsm.log is 0 byes (nothing being logged?) In my engine.log I'm seeing: 2018-01-25 17:12:11,027-05 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to lv426.dasgeekhaus.org/127.0.0.1 2018-01-25 17:12:11,028-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-99) [] Command 'GetCapabilitiesVDSCommand(HostName = lv426, VdsIdAndVdsVDSCommandParametersBase:{hostId='a645af84-3da1-45ed-bab5-2af66b5924dd', vds='Host[lv426,a645af84-3da1-45ed-bab5-2af66b5924dd]'})' execution failed: java.net.ConnectException: Connection refused 2018-01-25 17:12:11,028-05 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-99) [] Failure to refresh host 'lv426' runtime info: java.net.ConnectException: Connection refused 2018-01-25 17:12:13,517-05 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to lv426.dasgeekhaus.org/127.0.0.1 2018-01-25 17:12:13,517-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] Command 'GetAllVmStatsVDSCommand(HostName = lv426, VdsIdVDSCommandParametersBase:{hostId='a645af84-3da1-45ed-bab5-2af66b5924dd'})' execution failed: java.net.ConnectException: Connection refused 2018-01-25 17:12:13,518-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] Failed to fetch vms info for host 'lv426' - skipping VMs monitoring. All of my network interfaces do come up ok, iptables is turned off so it's not getting in the way. I'm at a complete loss right now as to what to look at.
And fixed, puppet made a change in sudo file. On 25 January 2018 at 17:16, CRiMSON <crimson@unspeakable.org> wrote:
I did a quick upgrade this afternoon on a dev machine.
Jan 25 11:57:07 Updated: glusterfs-libs-3.12.5-2.el7.x86_64 Jan 25 11:57:08 Updated: glusterfs-client-xlators-3.12.5-2.el7.x86_64 Jan 25 11:57:08 Updated: glusterfs-3.12.5-2.el7.x86_64 Jan 25 11:57:09 Updated: kernel-ml-tools-libs-4.14.15-1.el7.elrepo.x86_64 Jan 25 11:57:09 Updated: kernel-ml-tools-4.14.15-1.el7.elrepo.x86_64 Jan 25 11:57:10 Updated: glusterfs-api-3.12.5-2.el7.x86_64 Jan 25 11:57:10 Updated: glusterfs-fuse-3.12.5-2.el7.x86_64 Jan 25 11:57:10 Updated: glusterfs-cli-3.12.5-2.el7.x86_64 Jan 25 11:57:11 Updated: python-perf-4.14.15-1.el7.elrepo.x86_64 Jan 25 11:57:37 Installed: kernel-ml-devel-4.14.15-1.el7.elrepo.x86_64 Jan 25 11:57:39 Updated: kernel-ml-headers-4.14.15-1.el7.elrepo.x86_64 Jan 25 11:57:52 Installed: kernel-ml-4.14.15-1.el7.elrepo.x86_64 Jan 25 11:57:52 Updated: rubygem-fluent-plugin-viaq_ data_model-0.0.13-1.el7.noarch
This is all that was upgraded.
But now my storage domains are failing to come up and the host keeps saying it's getting a connection refused. It's all on 1 host.
In mom.log I see.
2018-01-25 17:10:49,929 - mom - INFO - MOM starting 2018-01-25 17:10:49,955 - mom.HostMonitor - INFO - Host Monitor starting 2018-01-25 17:10:49,955 - mom - INFO - hypervisor interface vdsmjsonrpcbulk 2018-01-25 17:10:50,013 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused 2018-01-25 17:10:50,013 - mom - ERROR - Failed to initialize MOM threads Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run hypervisor_iface = self.get_hypervisor_interface() File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217, in get_hypervisor_interface return module.instance(self.config) File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 47, in instance return JsonRpcVdsmBulkInterface() File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", line 29, in __init__ super(JsonRpcVdsmBulkInterface, self).__init__() File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py", line 43, in __init__ .orRaise(RuntimeError, 'No connection to VDSM.') File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in orRaise raise exception(*args, **kwargs) RuntimeError: No connection to VDSM. [root@lv426 vdsm]#
My vdsm.log is 0 byes (nothing being logged?)
In my engine.log I'm seeing:
2018-01-25 17:12:11,027-05 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to lv426.dasgeekhaus.org/127.0.0.1 2018-01-25 17:12:11,028-05 ERROR [org.ovirt.engine.core. vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-99) [] Command 'GetCapabilitiesVDSCommand(HostName = lv426, VdsIdAndVdsVDSCommandParametersBase:{hostId='a645af84-3da1-45ed-bab5-2af66b5924dd', vds='Host[lv426,a645af84-3da1-45ed-bab5-2af66b5924dd]'})' execution failed: java.net.ConnectException: Connection refused 2018-01-25 17:12:11,028-05 ERROR [org.ovirt.engine.core. vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-99) [] Failure to refresh host 'lv426' runtime info: java.net.ConnectException: Connection refused 2018-01-25 17:12:13,517-05 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to lv426.dasgeekhaus.org/127.0.0.1 2018-01-25 17:12:13,517-05 ERROR [org.ovirt.engine.core. vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] Command 'GetAllVmStatsVDSCommand(HostName = lv426, VdsIdVDSCommandParametersBase:{hostId='a645af84-3da1-45ed-bab5-2af66b5924dd'})' execution failed: java.net.ConnectException: Connection refused 2018-01-25 17:12:13,518-05 INFO [org.ovirt.engine.core. vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] Failed to fetch vms info for host 'lv426' - skipping VMs monitoring.
All of my network interfaces do come up ok, iptables is turned off so it's not getting in the way.
I'm at a complete loss right now as to what to look at.
participants (1)
-
CRiMSON