[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

OK, to wrap up this thread and to provide some detail as to how it concluded.... I now have the engine back up and running. The final issue was that I needed to re-create the Synology share that the HE was stored on. I got past all my issues and up to the domain setup stage. Obviously I couldn't install to the same share with the original domain still there so I deleted that (luckily other VMs are in a different share) but the install still failed with a storage domain creation error. So I created a new share and the install could now progress. So I have no idea what happened but I seem to have suffered some sort of failure of a shared folder on my Synology that caused issues with oVirt. I could still mount the folder manually and create/edit/delete files, and the engine was generating 100K's worth of .prob-* files, but it was somehow corrupt? I've copied the ansible error at the end of this mail. The other issue was the IPv6 one which was strange. I didn't see this when I first installed and setup oVirt but perhaps something changed somewhere in our setup. So as Strahil pointed out, I needed the IPV6ADDR entry in my ifcfg-eno1 for the interface being used for the node. The pain is that this is overwritten by the deployment so if that fails, you have to re-add it. So my interface config looks like this now: # Generated by VDSM version 4.30.40.1 DEVICE=eno1 BRIDGE=ovirtmgmt ONBOOT=yes MTU=1500 DEFROUTE=no NM_CONTROLLED=no IPV6INIT=no IPV6ADDR=64:ff9b::c0a8:13d Then I had the strange firewalld issue that I can't explain. I re-installed the node from scratch to resolve that as I'd lost patience. So thanks for all the help and I hope I never have to do that again. :-) Shareef. Ansible storage domain error: 2020-04-16 11:11:55,872+0000 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup : Add NFS storage domain] 2020-04-16 11:11:58,777+0000 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:103 {u'invocation': {u'module_args': {u'comment': None, u'warning_low_space': None, u'gluster fs': None, u'localfs': None, u'managed_block_storage': None, u'data_center': u'Default', u'id': None, u'iscsi': None, u'state': u'unattached', u'wipe_after_delete': None, u'destroy': None, u'fcp': None, u 'description': None, u'format': None, u'nested_attributes': [], u'host': u' ovirt-node-00.phoelex.com', u'discard_after_delete': None, u'wait': True, u'domain_function': u'data', u'name': u'hosted_storage' , u'critical_space_action_blocker': None, u'posixfs': None, u'poll_interval': 3, u'fetch_nested': False, u'nfs': {u'path': u'/volume1/ovirt', u'version': u'auto', u'mount_options': u'', u'address': u'nas- 01.phoelex.com'}, u'timeout': 180, u'backup': None}}, u'msg': u'Fault reason is "Operation Failed". Fault detail is "[Error in creating a Storage Domain. The selected storage path is not empty (probably c ontains another Storage Domain). Either remove the existing Storage Domain from this path, or change the Storage path).]". HTTP response code is 400.', u'exception': u'Traceback (most recent call last):\n File "/tmp/ansible_ovirt_storage_domain_payload_6uM8mE/ansible_ovirt_storage_domain_payload.zip/ansible/modules/cloud/ovirt/ovirt_storage_domain.py", line 792, in main\n File "/tmp/ansible_ovirt_storag e_domain_payload_6uM8mE/ansible_ovirt_storage_domain_payload.zip/ansible/module_utils/ovirt.py", line 621, in create\n **kwargs\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 25168, in add\n return self._internal_add(storage_domain, headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add\n return future.wait() i f wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", l ine 229, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault\n self._raise_error(response, body)\n File "/usr/lib6 4/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error\n raise error\nError: Fault reason is "Operation Failed". Fault detail is "[Error in creating a Storage Domain. The selected s torage path is not empty (probably contains another Storage Domain). Either remove the existing Storage Domain from this path, or change the Storage path).]". HTTP response code is 400.\n', u'changed': Fa lse, u'_ansible_no_log': False} 2020-04-16 11:11:58,877+0000 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:107 Error: Fault reason is "Operation Failed". Fault detail is "[Error in creating a Storage Domain. The selected storage path is not empty (probably contains another Storage Domain). Either remove the existing Storage Domain from this path, or change the Storage path).]". HTTP response code is 400. 2020-04-16 11:11:58,978+0000 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:107 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Error in creating a Storage Domain. The selected storage path is not empty (probably contains another Storage Domain). Either remove the existing Storage Domain from this path, or change the Storage path).]\". HTTP response code is 400."} On Thu, Apr 16, 2020 at 12:14 PM Shareef Jalloq <shareef(a)jalloq.co.uk&gt; wrote: > Actually, you've just raised a point I hadn't thought about. We have an > old Xeon server that is being used to host some ESXi VMs that were needed > while we transitioned to ovirt. Once I have moved those VMs I could > repurpose that as the engine. > > On Thu, Apr 16, 2020 at 11:42 AM Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; > wrote: > >> On April 16, 2020 11:25:20 AM GMT+03:00, Shareef Jalloq < >> shareef(a)jalloq.co.uk&gt; wrote: >> >Is this actually production ready? It seems to break at every step. >> > >> >On Wed, Apr 15, 2020 at 5:45 PM Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; >> >wrote: >> > >> >> On April 15, 2020 5:59:46 PM GMT+03:00, Shareef Jalloq < >> >> shareef(a)jalloq.co.uk&gt; wrote: >> >> >Thanks for your help but I've decided to try and reinstall from >> >> >scratch. >> >> >This is taking too long. >> >> > >> >> >On Wed, Apr 15, 2020 at 3:25 PM Strahil Nikolov >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >wrote: >> >> > >> >> >> On April 15, 2020 2:40:52 PM GMT+03:00, Shareef Jalloq < >> >> >> shareef(a)jalloq.co.uk&gt; wrote: >> >> >> >Yes, but there are no zones set up, just ports 22, 6801 adn 6900. >> >> >> > >> >> >> >On Wed, Apr 15, 2020 at 12:37 PM Strahil Nikolov >> >> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >> >wrote: >> >> >> > >> >> >> >> On April 15, 2020 2:28:05 PM GMT+03:00, Shareef Jalloq < >> >> >> >> shareef(a)jalloq.co.uk&gt; wrote: >> >> >> >> >Oh this is painful. It seems to progress if you have both >> >> >> >> >he_force_ipv4 >> >> >> >> >set and run the deployment with the '--4' switch. >> >> >> >> > >> >> >> >> >But then I get a failure when the ansible script checks for >> >> >> >> >firewalld-zones >> >> >> >> >and doesn't get anything back. Should the deployment flow not >> >be >> >> >> >> >setting >> >> >> >> >any zones it needs? >> >> >> >> > >> >> >> >> >2020-04-15 10:57:25,439+0000 INFO >> >> >> >> >otopi.ovirt_hosted_engine_setup.ansible_utils >> >> >> >> >ansible_utils._process_output:109 TASK >> >[ovirt.hosted_engine_setup >> >> >: >> >> >> >Get >> >> >> >> >active list of active firewalld zones] >> >> >> >> > >> >> >> >> >2020-04-15 10:57:26,641+0000 DEBUG >> >> >> >> >otopi.ovirt_hosted_engine_setup.ansible_utils >> >> >> >> >ansible_utils._process_output:103 {u'stderr_lines': [], >> >> >u'changed': >> >> >> >> >True, >> >> >> >> >u'end': u'2020-04-15 10:57:26.481202', u'_ansible_no_log': >> >False, >> >> >> >> >u'stdout': u'', u'cmd': u'set -euo pipefail && firewall-cmd >> >> >> >> >--get-active-zones | grep -v "^\\s*interfaces"', u'start': >> >> >> >u'2020-04-15 >> >> >> >> >10:57:26.050203', u'delta': u'0:00:00.430999', u'stderr': u'', >> >> >> >u'rc': >> >> >> >> >1, >> >> >> >> >u'invocation': {u'module_args': {u'creates': None, >> >u'executable': >> >> >> >None, >> >> >> >> >u'_uses_shell': True, u'strip_empty_ends': True, >> >u'_raw_params': >> >> >> >u'set >> >> >> >> >-euo >> >> >> >> >pipefail && firewall-cmd --get-active-zones | grep -v >> >> >> >> >"^\\s*interfaces"', >> >> >> >> >u'removes': None, u'argv': None, u'warn': True, u'chdir': >> >None, >> >> >> >> >u'stdin_add_newline': True, u'stdin': None}}, u'stdout_lines': >> >> >[], >> >> >> >> >u'msg': >> >> >> >> >u'non-zero return code'} >> >> >> >> > >> >> >> >> >2020-04-15 10:57:26,741+0000 ERROR >> >> >> >> >otopi.ovirt_hosted_engine_setup.ansible_utils >> >> >> >> >ansible_utils._process_output:107 fatal: [localhost]: FAILED! >> >=> >> >> >> >> >{"changed": true, "cmd": "set -euo pipefail && firewall-cmd >> >> >> >> >--get-active-zones | grep -v \"^\\s*interfaces\"", "delta": >> >> >> >> >"0:00:00.430999", "end": "2020-04-15 10:57:26.481202", "msg": >> >> >> >"non-zero >> >> >> >> >return code", "rc": 1, "start": "2020-04-15 10:57:26.050203", >> >> >> >"stderr": >> >> >> >> >"", >> >> >> >> >"stderr_lines": [], "stdout": "", "stdout_lines": []} >> >> >> >> > >> >> >> >> >On Wed, Apr 15, 2020 at 10:23 AM Shareef Jalloq >> >> >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >> >> >wrote: >> >> >> >> > >> >> >> >> >> Ha, spoke too soon. It's now stuck in a loop and a google >> >> >points >> >> >> >me >> >> >> >> >at >> >> >> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1746585 >> >> >> >> >> >> >> >> >> >> However, forcing ipv4 doesn't seem to have fixed the loop. >> >> >> >> >> >> >> >> >> >> On Wed, Apr 15, 2020 at 9:59 AM Shareef Jalloq >> >> >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >> >> >> wrote: >> >> >> >> >> >> >> >> >> >>> OK, that seems to have fixed it, thanks. Is this a side >> >> >effect >> >> >> >of >> >> >> >> >>> redeploying the HE over a first time install? Nothing has >> >> >changed >> >> >> >in >> >> >> >> >our >> >> >> >> >>> setup and I didn't need to do this when I initially set up >> >our >> >> >> >> >nodes. >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> On Tue, Apr 14, 2020 at 6:55 PM Strahil Nikolov >> >> >> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >> >> >>> wrote: >> >> >> >> >>> >> >> >> >> >>>> On April 14, 2020 6:17:17 PM GMT+03:00, Shareef Jalloq < >> >> >> >> >>>> shareef(a)jalloq.co.uk&gt; wrote: >> >> >> >> >>>> >Hmmm, we're not using ipv6. Is that the issue? >> >> >> >> >>>> > >> >> >> >> >>>> >On Tue, Apr 14, 2020 at 3:56 PM Strahil Nikolov >> >> >> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >> >> >>>> >wrote: >> >> >> >> >>>> > >> >> >> >> >>>> >> On April 14, 2020 1:27:24 PM GMT+03:00, Shareef Jalloq >> >< >> >> >> >> >>>> >> shareef(a)jalloq.co.uk&gt; wrote: >> >> >> >> >>>> >> >Right, I've given up on recovering the HE so want to >> >try >> >> >and >> >> >> >> >>>> >redeploy >> >> >> >> >>>> >> >it. >> >> >> >> >>>> >> >There doesn't seem to be enough information to debug >> >why >> >> >the >> >> >> >> >>>> >> >broker/agent >> >> >> >> >>>> >> >won't start cleanly. >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >In running 'hosted-engine --deploy', I'm seeing the >> >> >> >following >> >> >> >> >error >> >> >> >> >>>> >in >> >> >> >> >>>> >> >the >> >> >> >> >>>> >> >setup validation phase: >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:08,922+0000 DEBUG >> >> >> >> >otopi.plugins.otopi.dialog.human >> >> >> >> >>>> >> >dialog.__logString:204 DIALOG:SEND >> >Please >> >> >> >> >provide >> >> >> >> >>>> >the >> >> >> >> >>>> >> >hostname of this host on the management network >> >> >> >> >>>> >> >[ovirt-node-00.phoelex.com]: >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,831+0000 DEBUG >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >> >> >>>> >> >hostname.getResolvedAddresses:432 >> >> >> >> >>>> >> >getResolvedAddresses: set(['64:ff9b::c0a8:13d', >> >> >> >> >'192.168.1.61']) >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,832+0000 DEBUG >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >> >> >>>> >> >hostname._validateFQDNresolvability:289 >> >> >> >> >ovirt-node-00.phoelex.com >> >> >> >> >>>> >> >resolves >> >> >> >> >>>> >> >to: set(['64:ff9b::c0a8:13d', '192.168.1.61']) >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,832+0000 DEBUG >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >> >plugin.executeRaw:813 >> >> >> >> >>>> >> >execute: >> >> >> >> >>>> >> >['/usr/bin/dig', '+noall', '+answer', >> >> >> >> >'ovirt-node-00.phoelex.com', >> >> >> >> >>>> >> >'ANY'], >> >> >> >> >>>> >> >executable='None', cwd='None', env=None >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,871+0000 DEBUG >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >> >plugin.executeRaw:863 >> >> >> >> >>>> >> >execute-result: ['/usr/bin/dig', '+noall', '+answer', >> >' >> >> >> >> >>>> >> >ovirt-node-00.phoelex.com', 'ANY'], rc=0 >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,872+0000 DEBUG >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >plugin.execute:921 >> >> >> >> >>>> >> >execute-output: ['/usr/bin/dig', '+noall', '+answer', >> >' >> >> >> >> >>>> >> >ovirt-node-00.phoelex.com', 'ANY'] stdout: >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >ovirt-node-00.phoelex.com. 86400 IN A >> >> >192.168.1.61 >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,872+0000 DEBUG >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >plugin.execute:926 >> >> >> >> >>>> >> >execute-output: ['/usr/bin/dig', '+noall', '+answer', >> >' >> >> >> >> >>>> >> >ovirt-node-00.phoelex.com', 'ANY'] stderr: >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,872+0000 DEBUG >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >> >plugin.executeRaw:813 >> >> >> >> >>>> >> >execute: >> >> >> >> >>>> >> >('/usr/sbin/ip', 'addr'), executable='None', >> >cwd='None', >> >> >> >> >env=None >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,876+0000 DEBUG >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >> >plugin.executeRaw:863 >> >> >> >> >>>> >> >execute-result: ('/usr/sbin/ip', 'addr'), rc=0 >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,876+0000 DEBUG >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >plugin.execute:921 >> >> >> >> >>>> >> >execute-output: ('/usr/sbin/ip', 'addr') stdout: >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue >> >> >state >> >> >> >> >UNKNOWN >> >> >> >> >>>> >> >group >> >> >> >> >>>> >> >default qlen 1000 >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > link/loopback 00:00:00:00:00:00 brd >> >00:00:00:00:00:00 >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > inet 127.0.0.1/8 scope host lo >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > valid_lft forever preferred_lft forever >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > inet6 ::1/128 scope host >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > valid_lft forever preferred_lft forever >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 >> >qdisc >> >> >mq >> >> >> >> >master >> >> >> >> >>>> >> >ovirtmgmt state UP group default qlen 1000 >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > link/ether ac:1f:6b:bc:32:6a brd ff:ff:ff:ff:ff:ff >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 >> >> >qdisc >> >> >> >mq >> >> >> >> >state >> >> >> >> >>>> >> >DOWN >> >> >> >> >>>> >> >group default qlen 1000 >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > link/ether ac:1f:6b:bc:32:6b brd ff:ff:ff:ff:ff:ff >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc >> >noop >> >> >> >state >> >> >> >> >DOWN >> >> >> >> >>>> >> >group >> >> >> >> >>>> >> >default qlen 1000 >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > link/ether 02:e6:e2:80:93:8d brd ff:ff:ff:ff:ff:ff >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop >> >> >state >> >> >> >DOWN >> >> >> >> >>>> >group >> >> >> >> >>>> >> >default qlen 1000 >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > link/ether 8a:26:44:50:ee:4a brd ff:ff:ff:ff:ff:ff >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >21: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu >> >1500 >> >> >> >qdisc >> >> >> >> >>>> >noqueue >> >> >> >> >>>> >> >state UP group default qlen 1000 >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > link/ether ac:1f:6b:bc:32:6a brd ff:ff:ff:ff:ff:ff >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > inet 192.168.1.61/24 brd 192.168.1.255 scope >> >global >> >> >> >> >ovirtmgmt >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > valid_lft forever preferred_lft forever >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > inet6 fe80::ae1f:6bff:febc:326a/64 scope link >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > valid_lft forever preferred_lft forever >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >22: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc >> >> >noop >> >> >> >> >state >> >> >> >> >>>> >DOWN >> >> >> >> >>>> >> >group >> >> >> >> >>>> >> >default qlen 1000 >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > link/ether 3a:02:7b:7d:b3:2a brd ff:ff:ff:ff:ff:ff >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,876+0000 DEBUG >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >plugin.execute:926 >> >> >> >> >>>> >> >execute-output: ('/usr/sbin/ip', 'addr') stderr: >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,877+0000 DEBUG >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >> >> >>>> >> >hostname.getLocalAddresses:251 >> >> >> >> >>>> >> >addresses: [u'192.168.1.61', >> >> >u'fe80::ae1f:6bff:febc:326a'] >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,877+0000 DEBUG >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >> >> >hostname.test_hostname:464 >> >> >> >> >>>> >> >test_hostname exception >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >Traceback (most recent call last): >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >File >> >> >> >> >> >>"/usr/lib/python2.7/site-packages/ovirt_setup_lib/hostname.py", >> >> >> >> >>>> >> >line >> >> >> >> >>>> >> >460, in test_hostname >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > not_local_text, >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >File >> >> >> >> >> >>"/usr/lib/python2.7/site-packages/ovirt_setup_lib/hostname.py", >> >> >> >> >>>> >> >line >> >> >> >> >>>> >> >342, in _validateFQDNresolvability >> >> >> >> >>>> >> > >> >> >> >> >>>> >> > addresses=resolvedAddressesAsString >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >RuntimeError: ovirt-node-00.phoelex.com resolves to >> >> >> >> >>>> >64:ff9b::c0a8:13d >> >> >> >> >>>> >> >192.168.1.61 and not all of them can be mapped to non >> >> >> >loopback >> >> >> >> >>>> >devices >> >> >> >> >>>> >> >on >> >> >> >> >>>> >> >this host >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >2020-04-14 09:46:12,884+0000 ERROR >> >> >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >> >> >dialog.queryEnvKey:120 >> >> >> >> >>>> >Host >> >> >> >> >>>> >> >name >> >> >> >> >>>> >> >is not valid: ovirt-node-00.phoelex.com resolves to >> >> >> >> >>>> >64:ff9b::c0a8:13d >> >> >> >> >>>> >> >192.168.1.61 and not all of them can be mapped to non >> >> >> >loopback >> >> >> >> >>>> >devices >> >> >> >> >>>> >> >on >> >> >> >> >>>> >> >this host >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >The node I'm running on has an IP address of .61 and >> >> >> >resolves >> >> >> >> >>>> >> >correctly. >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >On Fri, Apr 10, 2020 at 12:55 PM Shareef Jalloq >> >> >> >> >>>> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >> >> >>>> >> >wrote: >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >> Where should I be checking if there are any >> >> >files/folder >> >> >> >not >> >> >> >> >owned >> >> >> >> >>>> >by >> >> >> >> >>>> >> >> vdsm:kvm? I checked on the mount the HA sits on and >> >> >it's >> >> >> >> >fine. >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> How would I go about checking vdsm can access those >> >> >> >images? >> >> >> >> >If I >> >> >> >> >>>> >run >> >> >> >> >>>> >> >> virsh, it lists them and they were running yesterday >> >> >even >> >> >> >> >though >> >> >> >> >>>> >the >> >> >> >> >>>> >> >HA was >> >> >> >> >>>> >> >> down. I've since restarted both hosts but the >> >broker >> >> >is >> >> >> >> >still >> >> >> >> >>>> >> >spitting out >> >> >> >> >>>> >> >> the same error (copied below). How do I find the >> >> >reason >> >> >> >the >> >> >> >> >>>> >broker >> >> >> >> >>>> >> >can't >> >> >> >> >>>> >> >> connect to the storage? The conf file is already at >> >> >DEBUG >> >> >> >> >>>> >verbosity: >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> [handler_logfile] >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> class=logging.handlers.TimedRotatingFileHandler >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> args=('/var/log/ovirt-hosted-engine-ha/broker.log', >> >> >'d', >> >> >> >1, >> >> >> >> >7) >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> level=DEBUG >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> formatter=long >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> And what are all these .prob-<num> files that are >> >being >> >> >> >> >created? >> >> >> >> >>>> >> >There >> >> >> >> >>>> >> >> are over 250K of them now on the mount I'm using for >> >> >the >> >> >> >Data >> >> >> >> >>>> >domain. >> >> >> >> >>>> >> >> They're all of 0 size and of the form, >> >> >> >> >>>> >> >> /rhev/data-center/mnt/nas-01.phoelex.com: >> >> >> >> >>>> >> >> >> >> >> >_volume2_vmstore/.prob-ffa867da-93db-4211-82df-b1b04a625ab9 >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> @eevans: The volume I have the Data Domain on has >> >TB's >> >> >> >free. >> >> >> >> > The >> >> >> >> >>>> >HA >> >> >> >> >>>> >> >is >> >> >> >> >>>> >> >> dead so I can't ssh in. No idea what started these >> >> >errors >> >> >> >> >and the >> >> >> >> >>>> >> >other >> >> >> >> >>>> >> >> VMs were still running happily although they're on a >> >> >> >> >different >> >> >> >> >>>> >Data >> >> >> >> >>>> >> >Domain. >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> Shareef. >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> MainThread::INFO::2020-04-10 >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>07:45:00,408::storage_backends::373::ovirt_hosted_engine_ha.lib.storage_backends::(connect) >> >> >> >> >>>> >> >> Connecting the storage >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> MainThread::INFO::2020-04-10 >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>07:45:00,408::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >> >> >>>> >> >> Connecting storage server >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> MainThread::INFO::2020-04-10 >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>07:45:01,577::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >> >> >>>> >> >> Connecting storage server >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> MainThread::INFO::2020-04-10 >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>07:45:02,692::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >> >> >>>> >> >> Refreshing the storage domain >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> MainThread::WARNING::2020-04-10 >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>07:45:05,175::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) >> >> >> >> >>>> >> >> Can't connect vdsm storage: Command >> >> >StorageDomain.getInfo >> >> >> >> >with >> >> >> >> >>>> >args >> >> >> >> >>>> >> >> {'storagedomainID': >> >> >> >'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} >> >> >> >> >>>> >failed: >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> (code=350, message=Error in storage domain action: >> >> >> >> >>>> >> >> (u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> On Thu, Apr 9, 2020 at 5:58 PM Strahil Nikolov >> >> >> >> >>>> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >> >> >>>> >> >> wrote: >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >>> On April 9, 2020 11:12:30 AM GMT+03:00, Shareef >> >Jalloq >> >> >< >> >> >> >> >>>> >> >>> shareef(a)jalloq.co.uk&gt; wrote: >> >> >> >> >>>> >> >>> >OK, let's go through this. I'm looking at the >> >node >> >> >that >> >> >> >at >> >> >> >> >>>> >least >> >> >> >> >>>> >> >still >> >> >> >> >>>> >> >>> >has >> >> >> >> >>>> >> >>> >some VMs running. virsh also tells me that the >> >> >> >> >HostedEngine VM >> >> >> >> >>>> >is >> >> >> >> >>>> >> >>> >running >> >> >> >> >>>> >> >>> >but it's unresponsive and I can't shut it down. >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >1. All storage domains exist and are mounted. >> >> >> >> >>>> >> >>> >2. The ha_agent exists: >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >[root@ovirt-node-01 ovirt-hosted-engine-ha]# ls >> >> >> >> >>>> >> >/rhev/data-center/mnt/ >> >> >> >> >>>> >> >>> >nas-01.phoelex.com >> >> >> >> >>>> >> >>> >> >> >\:_volume2_vmstore/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/ >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >dom_md ha_agent images master >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >3. There are two links >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >[root@ovirt-node-01 ovirt-hosted-engine-ha]# ll >> >> >> >> >>>> >> >/rhev/data-center/mnt/ >> >> >> >> >>>> >> >>> >nas-01.phoelex.com >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >>>>\:_volume2_vmstore/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/ha_agent/ >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >total 8 >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >lrwxrwxrwx. 1 vdsm kvm 132 Apr 2 14:50 >> >> >> >> >hosted-engine.lockspace >> >> >> >> >>>> >-> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>/var/run/vdsm/storage/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/ffb90b82-42fe-4253-85d5-aaec8c280aaf/90e68791-0c6f-406a-89ac-e0d86c631604 >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >lrwxrwxrwx. 1 vdsm kvm 132 Apr 2 14:50 >> >> >> >> >hosted-engine.metadata >> >> >> >> >>>> >-> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>/var/run/vdsm/storage/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/2161aed0-7250-4c1d-b667-ac94f60af17e/6b818e33-f80a-48cc-a59c-bba641e027d4 >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >4. The services exist but all seem to have some >> >sort >> >> >of >> >> >> >> >warning: >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >a) Apr 08 18:10:55 ovirt-node-01.phoelex.com >> >> >> >sanlock[1728]: >> >> >> >> >>>> >> >*2020-04-08 >> >> >> >> >>>> >> >>> >18:10:55 1744152 [36796]: s16 delta_renew long >> >write >> >> >> >time >> >> >> >> >10 >> >> >> >> >>>> >sec* >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >b) Mar 23 18:02:59 ovirt-node-01.phoelex.com >> >> >> >> >supervdsmd[29409]: >> >> >> >> >>>> >> >*failed >> >> >> >> >>>> >> >>> >to >> >> >> >> >>>> >> >>> >load module nvdimm: libbd_nvdimm.so.2: cannot open >> >> >> >shared >> >> >> >> >object >> >> >> >> >>>> >> >file: >> >> >> >> >>>> >> >>> >No >> >> >> >> >>>> >> >>> >such file or directory* >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >c) Apr 09 08:05:13 ovirt-node-01.phoelex.com >> >> >vdsm[4801]: >> >> >> >> >*ERROR >> >> >> >> >>>> >> >failed >> >> >> >> >>>> >> >>> >to >> >> >> >> >>>> >> >>> >retrieve Hosted Engine HA score '[Errno 2] No such >> >> >file >> >> >> >or >> >> >> >> >>>> >> >directory'Is >> >> >> >> >>>> >> >>> >the >> >> >> >> >>>> >> >>> >Hosted Engine setup finished?* >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >d)Apr 08 22:48:27 ovirt-node-01.phoelex.com >> >> >> >> >libvirtd[29307]: >> >> >> >> >>>> >> >2020-04-08 >> >> >> >> >>>> >> >>> >22:48:27.134+0000: 29309: warning : >> >> >> >qemuGetProcessInfo:1404 >> >> >> >> >: >> >> >> >> >>>> >> >cannot >> >> >> >> >>>> >> >>> >parse >> >> >> >> >>>> >> >>> >process status data >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >Apr 08 22:48:27 ovirt-node-01.phoelex.com >> >> >> >libvirtd[29307]: >> >> >> >> >>>> >> >2020-04-08 >> >> >> >> >>>> >> >>> >22:48:27.134+0000: 29309: error : >> >> >> >> >virNetDevTapInterfaceStats:764 >> >> >> >> >>>> >: >> >> >> >> >>>> >> >>> >internal >> >> >> >> >>>> >> >>> >error: /proc/net/dev: Interface not found >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >Apr 08 23:09:39 ovirt-node-01.phoelex.com >> >> >> >libvirtd[29307]: >> >> >> >> >>>> >> >2020-04-08 >> >> >> >> >>>> >> >>> >23:09:39.844+0000: 29307: error : >> >> >> >virNetSocketReadWire:1806 >> >> >> >> >: >> >> >> >> >>>> >End >> >> >> >> >>>> >> >of >> >> >> >> >>>> >> >>> >file >> >> >> >> >>>> >> >>> >while reading data: Input/output error >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >Apr 09 01:05:26 ovirt-node-01.phoelex.com >> >> >> >libvirtd[29307]: >> >> >> >> >>>> >> >2020-04-09 >> >> >> >> >>>> >> >>> >01:05:26.660+0000: 29307: error : >> >> >> >virNetSocketReadWire:1806 >> >> >> >> >: >> >> >> >> >>>> >End >> >> >> >> >>>> >> >of >> >> >> >> >>>> >> >>> >file >> >> >> >> >>>> >> >>> >while reading data: Input/output error >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >5 & 6. The broker log is continually printing >> >this >> >> >> >error: >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,438::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >> >> >> >>>> >> >>> >ovirt-hosted-engine-ha broker 2.3.6 started >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,438::broker::55::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >> >> >> >>>> >> >>> >Running broker >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,438::broker::120::ovirt_hosted_engine_ha.broker.broker.Broker::(_get_monitor) >> >> >> >> >>>> >> >>> >Starting monitor >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,438::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Searching for submonitors in >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >/submonitors >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,439::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor network >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,440::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor cpu-load-no-engine >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor mgmt-bridge >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor network >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor cpu-load >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor engine-health >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,442::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor mgmt-bridge >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,442::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor cpu-load-no-engine >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor cpu-load >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor mem-free >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor storage-domain >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor storage-domain >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor mem-free >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,444::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Loaded submonitor engine-health >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,444::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >Finished loading submonitors >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,444::broker::128::ovirt_hosted_engine_ha.broker.broker.Broker::(_get_storage_broker) >> >> >> >> >>>> >> >>> >Starting storage broker >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,444::storage_backends::369::ovirt_hosted_engine_ha.lib.storage_backends::(connect) >> >> >> >> >>>> >> >>> >Connecting to VDSM >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,444::util::384::ovirt_hosted_engine_ha.lib.storage_backends::(__log_debug) >> >> >> >> >>>> >> >>> >Creating a new json-rpc connection to VDSM >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >Client localhost:54321::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >08:07:31,453::concurrent::258::root::(run) START >> >> >thread >> >> >> >> >>>> >> ><Thread(Client >> >> >> >> >>>> >> >>> >localhost:54321, started daemon 139992488138496)> >> >> >> >> >(func=<bound >> >> >> >> >>>> >> >method >> >> >> >> >>>> >> >>> >Reactor.process_requests of >> >> >> >> ><yajsonrpc.betterAsyncore.Reactor >> >> >> >> >>>> >> >object at >> >> >> >> >>>> >> >>> >0x7f528acabc90>>, args=(), kwargs={}) >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >Client localhost:54321::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,459::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) >> >> >> >> >>>> >> >>> >Stomp connection established >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>08:07:31,467::stompclient::294::jsonrpc.AsyncoreClient::(send) >> >> >> >> >>>> >> >Sending >> >> >> >> >>>> >> >>> >response >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,530::storage_backends::373::ovirt_hosted_engine_ha.lib.storage_backends::(connect) >> >> >> >> >>>> >> >>> >Connecting the storage >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:31,531::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >> >> >>>> >> >>> >Connecting storage server >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>08:07:31,531::stompclient::294::jsonrpc.AsyncoreClient::(send) >> >> >> >> >>>> >> >Sending >> >> >> >> >>>> >> >>> >response >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>08:07:31,534::stompclient::294::jsonrpc.AsyncoreClient::(send) >> >> >> >> >>>> >> >Sending >> >> >> >> >>>> >> >>> >response >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:32,199::storage_server::158::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(_validate_pre_connected_path) >> >> >> >> >>>> >> >>> >Storage domain >> >a6cea67d-dbfb-45cf-a775-b4d0d47b26f2 >> >> >is >> >> >> >not >> >> >> >> >>>> >> >available >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:32,199::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >> >> >>>> >> >>> >Connecting storage server >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>08:07:32,199::stompclient::294::jsonrpc.AsyncoreClient::(send) >> >> >> >> >>>> >> >Sending >> >> >> >> >>>> >> >>> >response >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:32,814::storage_server::363::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >> >> >>>> >> >>> >[{u'status': 0, u'id': >> >> >> >> >u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}] >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:32,814::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >> >> >>>> >> >>> >Refreshing the storage domain >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>08:07:32,815::stompclient::294::jsonrpc.AsyncoreClient::(send) >> >> >> >> >>>> >> >Sending >> >> >> >> >>>> >> >>> >response >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:33,129::storage_server::420::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >> >> >>>> >> >>> >Error refreshing storage domain: Command >> >> >> >> >StorageDomain.getStats >> >> >> >> >>>> >> >with >> >> >> >> >>>> >> >>> >args >> >> >> >> >>>> >> >>> >{'storagedomainID': >> >> >> >'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} >> >> >> >> >>>> >failed: >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >(code=350, message=Error in storage domain action: >> >> >> >> >>>> >> >>> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>08:07:33,130::stompclient::294::jsonrpc.AsyncoreClient::(send) >> >> >> >> >>>> >> >Sending >> >> >> >> >>>> >> >>> >response >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:33,795::storage_backends::208::ovirt_hosted_engine_ha.lib.storage_backends::(_get_sector_size) >> >> >> >> >>>> >> >>> >Command StorageDomain.getInfo with args >> >> >> >{'storagedomainID': >> >> >> >> >>>> >> >>> >'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} failed: >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >(code=350, message=Error in storage domain action: >> >> >> >> >>>> >> >>> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >MainThread::WARNING::2020-04-09 >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>08:07:33,795::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) >> >> >> >> >>>> >> >>> >Can't connect vdsm storage: Command >> >> >> >StorageDomain.getInfo >> >> >> >> >with >> >> >> >> >>>> >args >> >> >> >> >>>> >> >>> >{'storagedomainID': >> >> >> >'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} >> >> >> >> >>>> >failed: >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >(code=350, message=Error in storage domain action: >> >> >> >> >>>> >> >>> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >The UUID it is moaning about is indeed the one >> >that >> >> >the >> >> >> >HA >> >> >> >> >sits >> >> >> >> >>>> >on >> >> >> >> >>>> >> >and >> >> >> >> >>>> >> >>> >is >> >> >> >> >>>> >> >>> >the one I listed the contents of in step 2 above. >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >So why can't it see this domain? >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >Thanks, Shareef. >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >On Thu, Apr 9, 2020 at 6:12 AM Strahil Nikolov >> >> >> >> >>>> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >> >> >>>> >> >>> >wrote: >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >> On April 9, 2020 1:51:05 AM GMT+03:00, Shareef >> >> >Jalloq >> >> >> >< >> >> >> >> >>>> >> >>> >> shareef(a)jalloq.co.uk&gt; wrote: >> >> >> >> >>>> >> >>> >> >Don't know if this is useful or not, but I just >> >> >tried >> >> >> >to >> >> >> >> >>>> >> >shutdown >> >> >> >> >>>> >> >>> >and >> >> >> >> >>>> >> >>> >> >start >> >> >> >> >>>> >> >>> >> >another VM on one of the hosts and get the >> >> >following >> >> >> >> >error: >> >> >> >> >>>> >> >>> >> > >> >> >> >> >>>> >> >>> >> >virsh # start scratch >> >> >> >> >>>> >> >>> >> > >> >> >> >> >>>> >> >>> >> >error: Failed to start domain scratch >> >> >> >> >>>> >> >>> >> > >> >> >> >> >>>> >> >>> >> >error: Network not found: no network with >> >matching >> >> >> >name >> >> >> >> >>>> >> >>> >> >'vdsm-ovirtmgmt' >> >> >> >> >>>> >> >>> >> > >> >> >> >> >>>> >> >>> >> >Is this not referring to the interface name as >> >the >> >> >> >> >network is >> >> >> >> >>>> >> >called >> >> >> >> >>>> >> >>> >> >'ovirtmgnt'. >> >> >> >> >>>> >> >>> >> > >> >> >> >> >>>> >> >>> >> >On Wed, Apr 8, 2020 at 11:35 PM Shareef Jalloq >> >> >> >> >>>> >> >>> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >> >> >>>> >> >>> >> >wrote: >> >> >> >> >>>> >> >>> >> > >> >> >> >> >>>> >> >>> >> >> Hmmm, virsh tells me the HE is running but it >> >> >> >hasn't >> >> >> >> >come >> >> >> >> >>>> >up >> >> >> >> >>>> >> >and >> >> >> >> >>>> >> >>> >the >> >> >> >> >>>> >> >>> >> >> agent.log is full of the same errors. >> >> >> >> >>>> >> >>> >> >> >> >> >> >> >>>> >> >>> >> >> On Wed, Apr 8, 2020 at 11:31 PM Shareef >> >Jalloq >> >> >> >> >>>> >> >>> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >> >> >>>> >> >>> >> >> wrote: >> >> >> >> >>>> >> >>> >> >> >> >> >> >> >>>> >> >>> >> >>> Ah hah! Ok, so I've managed to start it >> >using >> >> >> >virsh >> >> >> >> >on >> >> >> >> >>>> >the >> >> >> >> >>>> >> >>> >second >> >> >> >> >>>> >> >>> >> >host >> >> >> >> >>>> >> >>> >> >>> but my first host is still dead. >> >> >> >> >>>> >> >>> >> >>> >> >> >> >> >>>> >> >>> >> >>> First of all, what are these 56,317 .prob- >> >> >files >> >> >> >that >> >> >> >> >get >> >> >> >> >>>> >> >dumped >> >> >> >> >>>> >> >>> >to >> >> >> >> >>>> >> >>> >> >the >> >> >> >> >>>> >> >>> >> >>> NFS mounts? >> >> >> >> >>>> >> >>> >> >>> >> >> >> >> >>>> >> >>> >> >>> Secondly, why doesn't the node mount the NFS >> >> >> >> >directories >> >> >> >> >>>> >at >> >> >> >> >>>> >> >boot? >> >> >> >> >>>> >> >>> >> >Is >> >> >> >> >>>> >> >>> >> >>> that the issue with this particular node? >> >> >> >> >>>> >> >>> >> >>> >> >> >> >> >>>> >> >>> >> >>> On Wed, Apr 8, 2020 at 11:12 PM >> >> >> >> >>>> &gt;&lt;eevans(a)digitaldatatechs.com&gt; >> >> >> >> >>>> >> >>> >wrote: >> >> >> >> >>>> >> >>> >> >>> >> >> >> >> >>>> >> >>> >> >>>> Did you try virsh list --inactive >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> Eric Evans >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> Digital Data Services LLC. >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> 304.660.9080 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> *From:* Shareef Jalloq >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >> >> >>>> >> >>> >> >>>> *Sent:* Wednesday, April 8, 2020 5:58 PM >> >> >> >> >>>> >> >>> >> >>>> *To:* Strahil Nikolov >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >> >> >>>> >> >>> >> >>>> *Cc:* Ovirt Users <users(a)ovirt.org&gt; >> >> >> >> >>>> >> >>> >> >>>> *Subject:* [ovirt-users] Re: ovirt-engine >> >> >> >> >unresponsive - >> >> >> >> >>>> >how >> >> >> >> >>>> >> >to >> >> >> >> >>>> >> >>> >> >rescue? >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> I've now shut down the VMs on one host and >> >> >> >rebooted >> >> >> >> >it >> >> >> >> >>>> >but >> >> >> >> >>>> >> >the >> >> >> >> >>>> >> >>> >> >agent >> >> >> >> >>>> >> >>> >> >>>> service doesn't start. If I run >> >> >'hosted-engine >> >> >> >> >>>> >--vm-status' >> >> >> >> >>>> >> >I >> >> >> >> >>>> >> >>> >get: >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> The hosted engine configuration has not >> >been >> >> >> >> >retrieved >> >> >> >> >>>> >from >> >> >> >> >>>> >> >>> >shared >> >> >> >> >>>> >> >>> >> >>>> storage. Please ensure that ovirt-ha-agent >> >is >> >> >> >> >running and >> >> >> >> >>>> >> >the >> >> >> >> >>>> >> >>> >> >storage >> >> >> >> >>>> >> >>> >> >>>> server is reachable. >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> and indeed if I list the mounts under >> >> >> >> >>>> >/rhev/data-center/mnt, >> >> >> >> >>>> >> >>> >only >> >> >> >> >>>> >> >>> >> >one of >> >> >> >> >>>> >> >>> >> >>>> the directories is mounted. I have 3 NFS >> >> >mounts, >> >> >> >> >one ISO >> >> >> >> >>>> >> >Domain >> >> >> >> >>>> >> >>> >> >and two >> >> >> >> >>>> >> >>> >> >>>> Data Domains. Only one Data Domain has >> >> >mounted >> >> >> >and >> >> >> >> >this >> >> >> >> >>>> >has >> >> >> >> >>>> >> >>> >lots >> >> >> >> >>>> >> >>> >> >of .prob >> >> >> >> >>>> >> >>> >> >>>> files in. So why haven't the other NFS >> >> >exports >> >> >> >been >> >> >> >> >>>> >> >mounted? >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> Manually mounting them doesn't seem to have >> >> >> >helped >> >> >> >> >much >> >> >> >> >>>> >> >either. >> >> >> >> >>>> >> >>> >I >> >> >> >> >>>> >> >>> >> >can >> >> >> >> >>>> >> >>> >> >>>> start the broker service but the agent >> >service >> >> >> >says >> >> >> >> >no. >> >> >> >> >>>> >> >Same >> >> >> >> >>>> >> >>> >error >> >> >> >> >>>> >> >>> >> >as the >> >> >> >> >>>> >> >>> >> >>>> one in my last email. >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> Shareef. >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> On Wed, Apr 8, 2020 at 9:57 PM Shareef >> >Jalloq >> >> >> >> >>>> >> >>> >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >> >> >>>> >> >>> >> >>>> wrote: >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> Right, still down. I've run virsh and it >> >> >doesn't >> >> >> >> >know >> >> >> >> >>>> >> >anything >> >> >> >> >>>> >> >>> >> >about >> >> >> >> >>>> >> >>> >> >>>> the engine vm. >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> I've restarted the broker and agent >> >services >> >> >and >> >> >> >I >> >> >> >> >still >> >> >> >> >>>> >get >> >> >> >> >>>> >> >>> >> >nothing in >> >> >> >> >>>> >> >>> >> >>>> virsh->list. >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> In the logs under >> >> >/var/log/ovirt-hosted-engine-ha >> >> >> >I >> >> >> >> >see >> >> >> >> >>>> >lots >> >> >> >> >>>> >> >of >> >> >> >> >>>> >> >>> >> >errors: >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> broker.log: >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,138::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >> >> >> >>>> >> >>> >> >>>> ovirt-hosted-engine-ha broker 2.3.6 started >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,138::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Searching for submonitors in >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,138::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor network >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,140::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor cpu-load-no-engine >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,140::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor mgmt-bridge >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor network >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor cpu-load >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor engine-health >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor mgmt-bridge >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,142::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor cpu-load-no-engine >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,142::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor cpu-load >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,142::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor mem-free >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor storage-domain >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor storage-domain >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor mem-free >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Loaded submonitor engine-health >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,143::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Finished loading submonitors >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,197::storage_backends::373::ovirt_hosted_engine_ha.lib.storage_backends::(connect) >> >> >> >> >>>> >> >>> >> >>>> Connecting the storage >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,197::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >> >> >>>> >> >>> >> >>>> Connecting storage server >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,414::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >> >> >>>> >> >>> >> >>>> Connecting storage server >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:20,628::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >> >> >>>> >> >>> >> >>>> Refreshing the storage domain >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::WARNING::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:21,057::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) >> >> >> >> >>>> >> >>> >> >>>> Can't connect vdsm storage: Command >> >> >> >> >StorageDomain.getInfo >> >> >> >> >>>> >> >with >> >> >> >> >>>> >> >>> >args >> >> >> >> >>>> >> >>> >> >>>> {'storagedomainID': >> >> >> >> >>>> >'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} >> >> >> >> >>>> >> >>> >failed: >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> (code=350, message=Error in storage domain >> >> >> >action: >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:21,901::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >> >> >> >>>> >> >>> >> >>>> ovirt-hosted-engine-ha broker 2.3.6 started >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:56:21,901::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >> >> >>>> >> >>> >> >>>> Searching for submonitors in >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> agent.log: >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::ERROR::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:57:00,799::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) >> >> >> >> >>>> >> >>> >> >>>> Trying to restart agent >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:57:00,799::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) >> >> >> >> >>>> >> >>> >> >>>> Agent shutting down >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:57:11,144::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) >> >> >> >> >>>> >> >>> >> >>>> ovirt-hosted-engine-ha agent 2.3.6 started >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:57:11,182::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) >> >> >> >> >>>> >> >>> >> >>>> Found certificate common name: >> >> >> >> >ovirt-node-01.phoelex.com >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:57:11,294::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) >> >> >> >> >>>> >> >>> >> >>>> Initializing ha-broker connection >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:57:11,296::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) >> >> >> >> >>>> >> >>> >> >>>> Starting monitor network, options >> >> >> >{'tcp_t_address': >> >> >> >> >'', >> >> >> >> >>>> >> >>> >> >'network_test': >> >> >> >> >>>> >> >>> >> >>>> 'dns', 'tcp_t_port': '', 'addr': >> >> >'192.168.1.99'} >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::ERROR::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:57:11,296::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) >> >> >> >> >>>> >> >>> >> >>>> Failed to start necessary monitors >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::ERROR::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:57:11,297::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) >> >> >> >> >>>> >> >>> >> >>>> Traceback (most recent call last): >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> File >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", >> >> >> >> >>>> >> >>> >> >>>> line 131, in _run_agent >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> return action(he) >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> File >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", >> >> >> >> >>>> >> >>> >> >>>> line 55, in action_proper >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> return he.start_monitoring() >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> File >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >> >> >> >> >>>> >> >>> >> >>>> line 432, in start_monitoring >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> self._initialize_broker() >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> File >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >> >> >> >> >>>> >> >>> >> >>>> line 556, in _initialize_broker >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> m.get('options', {})) >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> File >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", >> >> >> >> >>>> >> >>> >> >>>> line 89, in start_monitor >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> ).format(t=type, o=options, e=e) >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> RequestError: brokerlink - failed to start >> >> >> >monitor >> >> >> >> >via >> >> >> >> >>>> >> >>> >> >ovirt-ha-broker: >> >> >> >> >>>> >> >>> >> >>>> [Errno 2] No such file or directory, >> >[monitor: >> >> >> >> >'network', >> >> >> >> >>>> >> >>> >options: >> >> >> >> >>>> >> >>> >> >>>> {'tcp_t_address': '', 'network_test': >> >'dns', >> >> >> >> >>>> >'tcp_t_port': >> >> >> >> >>>> >> >'', >> >> >> >> >>>> >> >>> >> >'addr': >> >> >> >> >>>> >> >>> >> >>>> '192.168.1.99'}] >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::ERROR::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:57:11,297::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) >> >> >> >> >>>> >> >>> >> >>>> Trying to restart agent >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>>20:57:11,297::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) >> >> >> >> >>>> >> >>> >> >>>> Agent shutting down >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> On Wed, Apr 8, 2020 at 6:10 PM Strahil >> >Nikolov >> >> >> >> >>>> >> >>> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >> >> >>>> >> >>> >> >>>> wrote: >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> On April 8, 2020 7:47:20 PM GMT+03:00, >> >"Maton, >> >> >> >> >Brett" < >> >> >> >> >>>> >> >>> >> >>>> matonb(a)ltresources.co.uk&gt; wrote: >> >> >> >> >>>> >> >>> >> >>>> >On the host you tried to restart the >> >engine >> >> >on: >> >> >> >> >>>> >> >>> >> >>>> > >> >> >> >> >>>> >> >>> >> >>>> >Add an alias to virsh (authenticates with >> >> >> >> >>>> >virsh_auth.conf) >> >> >> >> >>>> >> >>> >> >>>> > >> >> >> >> >>>> >> >>> >> >>>> >alias virsh='virsh -c >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >> >> >>>>>>qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf' >> >> >> >> >>>> >> >>> >> >>>> > >> >> >> >> >>>> >> >>> >> >>>> >Then run virsh: >> >> >> >> >>>> >> >>> >> >>>> > >> >> >> >> >>>> >> >>> >> >>>> >virsh >> >> >> >> >>>> >> >>> >> >>>> > >> >> >> >> >>>> >> >>> >> >>>> >virsh # list >> >> >> >> >>>> >> >>> >> >>>> > Id Name >> >State >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>---------------------------------------------------- >> >> >> >> >>>> >> >>> >> >>>> > xx HostedEngine >> >Paused >> >> >> >> >>>> >> >>> >> >>>> > xx ********** >> >running >> >> >> >> >>>> >> >>> >> >>>> > ... >> >> >> >> >>>> >> >>> >> >>>> > xx ********** >> >> >running >> >> >> >> >>>> >> >>> >> >>>> > >> >> >> >> >>>> >> >>> >> >>>> >HostedEngine should be in the list, try >> >and >> >> >> >resume >> >> >> >> >the >> >> >> >> >>>> >> >engine: >> >> >> >> >>>> >> >>> >> >>>> > >> >> >> >> >>>> >> >>> >> >>>> >virsh # resume HostedEngine >> >> >> >> >>>> >> >>> >> >>>> > >> >> >> >> >>>> >> >>> >> >>>> >On Wed, 8 Apr 2020 at 17:28, Shareef >> >Jalloq >> >> >> >> >>>> >> >>> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >> >> >>>> >> >>> >> >>>> >wrote: >> >> >> >> >>>> >> >>> >> >>>> > >> >> >> >> >>>> >> >>> >> >>>> >> Thanks! >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >> >>>> >> >>> >> >>>> >> The status hangs due to, I guess, the VM >> >> >being >> >> >> >> >>>> >down.... >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >> >>>> >> >>> >> >>>> >> [root@ovirt-node-01 ~]# hosted-engine >> >> >> >--vm-start >> >> >> >> >>>> >> >>> >> >>>> >> VM exists and is down, cleaning up and >> >> >> >restarting >> >> >> >> >>>> >> >>> >> >>>> >> VM in WaitForLaunch >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >> >>>> >> >>> >> >>>> >> but this doesn't seem to do anything. >> >OK, >> >> >> >after >> >> >> >> >a >> >> >> >> >>>> >while >> >> >> >> >>>> >> >I >> >> >> >> >>>> >> >>> >get a >> >> >> >> >>>> >> >>> >> >>>> >status of >> >> >> >> >>>> >> >>> >> >>>> >> it being barfed... >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >> >>>> >> >>> >> >>>> >> --== Host ovirt-node-00.phoelex.com (id: >> >1) >> >> >> >> >status >> >> >> >> >>>> >==-- >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >> >>>> >> >>> >> >>>> >> conf_on_shared_storage : >> >True >> >> >> >> >>>> >> >>> >> >>>> >> Status up-to-date : >> >False >> >> >> >> >>>> >> >>> >> >>>> >> Hostname : >> >> >> >> >>>> >> >>> >ovirt-node-00.phoelex.com >> >> >> >> >>>> >> >>> >> >>>> >> Host ID : 1 >> >> >> >> >>>> >> >>> >> >>>> >> Engine status : >> >> >unknown >> >> >> >> >>>> >stale-data >> >> >> >> >>>> >> >>> >> >>>> >> Score : >> >3400 >> >> >> >> >>>> >> >>> >> >>>> >> stopped : >> >False >> >> >> >> >>>> >> >>> >> >>>> >> Local maintenance : >> >False >> >> >> >> >>>> >> >>> >> >>>> >> crc32 : >> >> >9c4a034b >> >> >> >> >>>> >> >>> >> >>>> >> local_conf_timestamp : >> >523362 >> >> >> >> >>>> >> >>> >> >>>> >> Host timestamp : >> >523608 >> >> >> >> >>>> >> >>> >> >>>> >> Extra metadata (valid at timestamp): >> >> >> >> >>>> >> >>> >> >>>> >> metadata_parse_version=1 >> >> >> >> >>>> >> >>> >> >>>> >> metadata_feature_version=1 >> >> >> >> >>>> >> >>> >> >>>> >> timestamp=523608 (Wed Apr 8 16:17:11 >> >2020) >> >> >> >> >>>> >> >>> >> >>>> >> host-id=1 >> >> >> >> >>>> >> >>> >> >>>> >> score=3400 >> >> >> >> >>>> >> >>> >> >>>> >> vm_conf_refresh_time=523362 (Wed Apr 8 >> >> >> >16:13:06 >> >> >> >> >2020) >> >> >> >> >>>> >> >>> >> >>>> >> conf_on_shared_storage=True >> >> >> >> >>>> >> >>> >> >>>> >> maintenance=False >> >> >> >> >>>> >> >>> >> >>>> >> state=EngineDown >> >> >> >> >>>> >> >>> >> >>>> >> stopped=False >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >> >>>> >> >>> >> >>>> >> --== Host ovirt-node-01.phoelex.com (id: >> >2) >> >> >> >> >status >> >> >> >> >>>> >==-- >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >> >>>> >> >>> >> >>>> >> conf_on_shared_storage : >> >True >> >> >> >> >>>> >> >>> >> >>>> >> Status up-to-date : >> >True >> >> >> >> >>>> >> >>> >> >>>> >> Hostname : >> >> >> >> >>>> >> >>> >ovirt-node-01.phoelex.com >> >> >> >> >>>> >> >>> >> >>>> >> Host ID : 2 >> >> >> >> >>>> >> >>> >> >>>> >> Engine status : >> >> >> >{"reason": >> >> >> >> >"bad >> >> >> >> >>>> >vm >> >> >> >> >>>> >> >>> >status", >> >> >> >> >>>> >> >>> >> >>>> >"health": >> >> >> >> >>>> >> >>> >> >>>> >> "bad", "vm": "down_unexpected", >> >"detail": >> >> >> >"Down"} >> >> >> >> >>>> >> >>> >> >>>> >> Score : 0 >> >> >> >> >>>> >> >>> >> >>>> >> stopped : >> >False >> >> >> >> >>>> >> >>> >> >>>> >> Local maintenance : >> >False >> >> >> >> >>>> >> >>> >> >>>> >> crc32 : >> >> >5045f2eb >> >> >> >> >>>> >> >>> >> >>>> >> local_conf_timestamp : >> >> >1737037 >> >> >> >> >>>> >> >>> >> >>>> >> Host timestamp : >> >> >1737283 >> >> >> >> >>>> >> >>> >> >>>> >> Extra metadata (valid at timestamp): >> >> >> >> >>>> >> >>> >> >>>> >> metadata_parse_version=1 >> >> >> >> >>>> >> >>> >> >>>> >> metadata_feature_version=1 >> >> >> >> >>>> >> >>> >> >>>> >> timestamp=1737283 (Wed Apr 8 16:16:17 >> >> >2020) >> >> >> >> >>>> >> >>> >> >>>> >> host-id=2 >> >> >> >> >>>> >> >>> >> >>>> >> score=0 >> >> >> >> >>>> >> >>> >> >>>> >> vm_conf_refresh_time=1737037 (Wed Apr 8 >> >> >> >16:12:11 >> >> >> >> >>>> >2020) >> >> >> >> >>>> >> >>> >> >>>> >> conf_on_shared_storage=True >> >> >> >> >>>> >> >>> >> >>>> >> maintenance=False >> >> >> >> >>>> >> >>> >> >>>> >> state=EngineUnexpectedlyDown >> >> >> >> >>>> >> >>> >> >>>> >> stopped=False >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >> >>>> >> >>> >> >>>> >> On Wed, Apr 8, 2020 at 5:09 PM Maton, >> >Brett >> >> >> >> >>>> >> >>> >> >>>> &gt;&lt;matonb(a)ltresources.co.uk&gt; >> >> >> >> >>>> >> >>> >> >>>> >> wrote: >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >> >>>> >> >>> >> >>>> >>> First steps, on one of your hosts as >> >root: >> >> >> >> >>>> >> >>> >> >>>> >>> >> >> >> >> >>>> >> >>> >> >>>> >>> To get information: >> >> >> >> >>>> >> >>> >> >>>> >>> hosted-engine --vm-status >> >> >> >> >>>> >> >>> >> >>>> >>> >> >> >> >> >>>> >> >>> >> >>>> >>> To start the engine: >> >> >> >> >>>> >> >>> >> >>>> >>> hosted-engine --vm-start >> >> >> >> >>>> >> >>> >> >>>> >>> >> >> >> >> >>>> >> >>> >> >>>> >>> >> >> >> >> >>>> >> >>> >> >>>> >>> On Wed, 8 Apr 2020 at 17:00, Shareef >> >> >Jalloq >> >> >> >> >>>> >> >>> >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >> >> >>>> >> >>> >> >>>> >wrote: >> >> >> >> >>>> >> >>> >> >>>> >>> >> >> >> >> >>>> >> >>> >> >>>> >>>> So my engine has gone down and I can't >> >> >ssh >> >> >> >into >> >> >> >> >it >> >> >> >> >>>> >> >either. >> >> >> >> >>>> >> >>> >If >> >> >> >> >>>> >> >>> >> >I >> >> >> >> >>>> >> >>> >> >>>> >try to >> >> >> >> >>>> >> >>> >> >>>> >>>> log into the web-ui of the node it is >> >> >> >running >> >> >> >> >on, I >> >> >> >> >>>> >get >> >> >> >> >>>> >> >>> >> >redirected >> >> >> >> >>>> >> >>> >> >>>> >because >> >> >> >> >>>> >> >>> >> >>>> >>>> the node can't reach the engine. >> >> >> >> >>>> >> >>> >> >>>> >>>> >> >> >> >> >>>> >> >>> >> >>>> >>>> What are my next steps? >> >> >> >> >>>> >> >>> >> >>>> >>>> >> >> >> >> >>>> >> >>> >> >>>> >>>> Shareef. >> >> >> >> >>>> >> >>> >> >>>> >>>> >> >> >> >_______________________________________________ >> >> >> >> >>>> >> >>> >> >>>> >>>> Users mailing list -- users(a)ovirt.org >> >> >> >> >>>> >> >>> >> >>>> >>>> To unsubscribe send an email to >> >> >> >> >>>> &gt;users-leave(a)ovirt.org >> >> >> >> >>>> >> >>> >> >>>> >>>> Privacy Statement: >> >> >> >> >>>> >> >>> >https://www.ovirt.org/privacy-policy.html >> >> >> >> >>>> >> >>> >> >>>> >>>> oVirt Code of Conduct: >> >> >> >> >>>> >> >>> >> >>>> >>>> >> >> >> >> >>>> >> >> >> >>https://www.ovirt.org/community/about/community-guidelines/ >> >> >> >> >>>> >> >>> >> >>>> >>>> List Archives: >> >> >> >> >>>> >> >>> >> >>>> >>>> >> >> >> >> >>>> >> >>> >> >>>> > >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> > >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> > >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> > >> >> >> >> >>>> >> >> >> >> >> >>>> > >> >> >> >> >>>> >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> >> >> > >> >> >> > >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/W7BP57OCIRS... >> >> >> >> >>>> >> >>> >> >>>> >>>> >> >> >> >> >>>> >> >>> >> >>>> >>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> This has to be resolved: >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> Engine status : >> >unknown >> >> >> >> >stale-data >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> Run again 'hosted-engine --vm-status'. If >> >it >> >> >> >remains >> >> >> >> >the >> >> >> >> >>>> >> >same, >> >> >> >> >>>> >> >>> >> >restart >> >> >> >> >>>> >> >>> >> >>>> ovirt-ha-broker.service & >> >> >ovirt-ha-agent.service >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> Verify that the engine's storage is >> >available. >> >> >> >Then >> >> >> >> >>>> >monitor >> >> >> >> >>>> >> >the >> >> >> >> >>>> >> >>> >> >broker >> >> >> >> >>>> >> >>> >> >>>> & agent logs in >> >> >/var/log/ovirt-hosted-engine-ha >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> Best Regards, >> >> >> >> >>>> >> >>> >> >>>> Strahil Nikolov >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >>>> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> Hi Shareef, >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> The flow of activation oVirt is more complex >> >than a >> >> >> >plain >> >> >> >> >KVM. >> >> >> >> >>>> >> >>> >> Mounting of the domains happen during the >> >> >activation >> >> >> >of >> >> >> >> >the >> >> >> >> >>>> >node >> >> >> >> >>>> >> >( >> >> >> >> >>>> >> >>> >the >> >> >> >> >>>> >> >>> >> HostedEngine is activating everything needed). >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> Focus on the HostedEngine VM. >> >> >> >> >>>> >> >>> >> Is it running properly ? >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> If not,try: >> >> >> >> >>>> >> >>> >> 1. Verify that the storage domain exists >> >> >> >> >>>> >> >>> >> 2. Check if it has 'ha_agents' directory >> >> >> >> >>>> >> >>> >> 3. Check if the links are OK, if not you can >> >> >safely >> >> >> >> >remove >> >> >> >> >>>> >the >> >> >> >> >>>> >> >links >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> 4. Next check the services are running: >> >> >> >> >>>> >> >>> >> A) sanlock >> >> >> >> >>>> >> >>> >> B) supervdsmd >> >> >> >> >>>> >> >>> >> C) vdsmd >> >> >> >> >>>> >> >>> >> D) libvirtd >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> 5. Increase the log level for broker and agent >> >> >> >services: >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> cd /etc/ovirt-hosted-engine-ha >> >> >> >> >>>> >> >>> >> vim *-log.conf >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> systemctl restart ovirt-ha-broker ovirt-ha-agent >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> 6. Check what they are complaining about >> >> >> >> >>>> >> >>> >> Keep in mind that agent will keep throwing >> >errors >> >> >> >untill >> >> >> >> >the >> >> >> >> >>>> >> >broker >> >> >> >> >>>> >> >>> >stops >> >> >> >> >>>> >> >>> >> doing it (agent depends on broker), so broker >> >> >must >> >> >> >be >> >> >> >> >OK >> >> >> >> >>>> >before >> >> >> >> >>>> >> >>> >> peoceeding with the agent log. >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> About the manual VM start, you need 2 things: >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> 1. Define the VM network >> >> >> >> >>>> >> >>> >> # cat vdsm-ovirtmgmt.xml <network> >> >> >> >> >>>> >> >>> >> <name>vdsm-ovirtmgmt</name> >> >> >> >> >>>> >> >>> >> >> ><uuid>8ded486e-e681-4754-af4b-5737c2b05405</uuid> >> >> >> >> >>>> >> >>> >> <forward mode='bridge'/> >> >> >> >> >>>> >> >>> >> <bridge name='ovirtmgmt'/> >> >> >> >> >>>> >> >>> >> </network> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> [root@ovirt1 HostedEngine-RECOVERY]# virsh >> >define >> >> >> >> >>>> >> >vdsm-ovirtmgmt.xml >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> 2. Get an xml definition which can be found in >> >the >> >> >> >vdsm >> >> >> >> >log. >> >> >> >> >>>> >> >Every VM >> >> >> >> >>>> >> >>> >at >> >> >> >> >>>> >> >>> >> start up has it's configuration printed out in >> >> >vdsm >> >> >> >log >> >> >> >> >on >> >> >> >> >>>> >the >> >> >> >> >>>> >> >host >> >> >> >> >>>> >> >>> >it >> >> >> >> >>>> >> >>> >> starts. >> >> >> >> >>>> >> >>> >> Save to file and then: >> >> >> >> >>>> >> >>> >> A) virsh define myvm.xml >> >> >> >> >>>> >> >>> >> B) virsh start myvm >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> It seems there is/was a problem with your NFS >> >> >shares. >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> Best Regards, >> >> >> >> >>>> >> >>> >> Strahil Nikolov >> >> >> >> >>>> >> >>> >> >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> Hey Shareef, >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> Check if there are any files or folders not owned >> >by >> >> >> >> >vdsm:kvm . >> >> >> >> >>>> >> >Something >> >> >> >> >>>> >> >>> like this: >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> find . -not -user 36 -not -group 36 -print >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> Also check if vdsm can access the images in the >> >> >> >> >>>> >> >>> '<vol-mount-point>/images' directories. >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> Best Regards, >> >> >> >> >>>> >> >>> Strahil Nikolov >> >> >> >> >>>> >> >>> >> >> >> >> >>>> >> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> And the IPv6 address '64:ff9b::c0a8:13d' ? >> >> >> >> >>>> >> >> >> >> >> >>>> >> I don't see in the log output. >> >> >> >> >>>> >> >> >> >> >> >>>> >> Best Regards, >> >> >> >> >>>> >> Strahil Nikolov >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >>>> Based on your output , you got a PTR record for IPv4 & >> >> >IPv6 >> >> >> >... >> >> >> >> >most >> >> >> >> >>>> probably it's the reason. >> >> >> >> >>>> >> >> >> >> >>>> Set the IPv6 on the interface and try again. >> >> >> >> >>>> >> >> >> >> >>>> Best Regards, >> >> >> >> >>>> Strahil Nikolov >> >> >> >> >>>> >> >> >> >> >>> >> >> >> >> >> >> >> >> Do you have firewalld up and running on the host ? >> >> >> >> >> >> >> >> Best Regards, >> >> >> >> Strahil Nikolov >> >> >> >> >> >> >> >> >> >> I am guessing, but your interface is not asaigned to any zone , >> >> >right? >> >> >> Just add the interface to the default zone (usually 'public'). >> >> >> >> >> >> Best Regards, >> >> >> Strahil Nikolov >> >> >> >> >> >> >> Keep in mind that there are a lot of playbooks that can be used >> >to >> >> deploy a HostedEngine Environment via ansible. >> >> >> >> Keep in mind that if you plan to use oVirt in Prod, you need to know >> >how >> >> to debug it (at least on basic level). >> >> >> >> Best Regards, >> >> Strahil Nikolov >> >> >> >> It's really interesting that you mention that topic. >> The only way I managed to break my engine was: >> A) bad SELINUX rpm which was solved via reinstall of the package and >> relabel >> B) Interrupted patch, as I forgot to use screen >> >> I think it is Prod ready, but it requires knowledge as it is not >> as dummy-proof like VMware. Yet, oVirt is way more flexible allowing you >> to run your own scripts before/during/after a certain event (vdsm hooks). >> >> Sadly Ansible (this is what is used for setup of gluster -> >> gdeploy, and for the engine) is quite dynamic and sometimes something >> might break. >> >> If you feel that oVirt breaks too often - just set your engine on a >> separate physical or virtual (non-hosted) machine, but do not complain that >> a free open-source product is not Production ready, just because you don't >> know how to debug it. >> >> >> You can trial the downstream solutions from Red Hat & Oracle and you >> will notice the difference. For me oVirt is like Fedora compared to >> RHEL/OEL/CentOS, but this is just a personal opinion. >> >> Best Regards, >> Strahil Nikolov >> >