[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

On April 15, 2020 5:59:46 PM GMT+03:00, Shareef Jalloq < shareef(a)jalloq.co.uk&gt; wrote: >Thanks for your help but I've decided to try and reinstall from >scratch. >This is taking too long. > >On Wed, Apr 15, 2020 at 3:25 PM Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; >wrote: > >> On April 15, 2020 2:40:52 PM GMT+03:00, Shareef Jalloq < >> shareef(a)jalloq.co.uk&gt; wrote: >> >Yes, but there are no zones set up, just ports 22, 6801 adn 6900. >> > >> >On Wed, Apr 15, 2020 at 12:37 PM Strahil Nikolov >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >wrote: >> > >> >> On April 15, 2020 2:28:05 PM GMT+03:00, Shareef Jalloq < >> >> shareef(a)jalloq.co.uk&gt; wrote: >> >> >Oh this is painful. It seems to progress if you have both >> >> >he_force_ipv4 >> >> >set and run the deployment with the '--4' switch. >> >> > >> >> >But then I get a failure when the ansible script checks for >> >> >firewalld-zones >> >> >and doesn't get anything back. Should the deployment flow not be >> >> >setting >> >> >any zones it needs? >> >> > >> >> >2020-04-15 10:57:25,439+0000 INFO >> >> >otopi.ovirt_hosted_engine_setup.ansible_utils >> >> >ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup >: >> >Get >> >> >active list of active firewalld zones] >> >> > >> >> >2020-04-15 10:57:26,641+0000 DEBUG >> >> >otopi.ovirt_hosted_engine_setup.ansible_utils >> >> >ansible_utils._process_output:103 {u'stderr_lines': [], >u'changed': >> >> >True, >> >> >u'end': u'2020-04-15 10:57:26.481202', u'_ansible_no_log': False, >> >> >u'stdout': u'', u'cmd': u'set -euo pipefail && firewall-cmd >> >> >--get-active-zones | grep -v "^\\s*interfaces"', u'start': >> >u'2020-04-15 >> >> >10:57:26.050203', u'delta': u'0:00:00.430999', u'stderr': u'', >> >u'rc': >> >> >1, >> >> >u'invocation': {u'module_args': {u'creates': None, u'executable': >> >None, >> >> >u'_uses_shell': True, u'strip_empty_ends': True, u'_raw_params': >> >u'set >> >> >-euo >> >> >pipefail && firewall-cmd --get-active-zones | grep -v >> >> >"^\\s*interfaces"', >> >> >u'removes': None, u'argv': None, u'warn': True, u'chdir': None, >> >> >u'stdin_add_newline': True, u'stdin': None}}, u'stdout_lines': >[], >> >> >u'msg': >> >> >u'non-zero return code'} >> >> > >> >> >2020-04-15 10:57:26,741+0000 ERROR >> >> >otopi.ovirt_hosted_engine_setup.ansible_utils >> >> >ansible_utils._process_output:107 fatal: [localhost]: FAILED! => >> >> >{"changed": true, "cmd": "set -euo pipefail && firewall-cmd >> >> >--get-active-zones | grep -v \"^\\s*interfaces\"", "delta": >> >> >"0:00:00.430999", "end": "2020-04-15 10:57:26.481202", "msg": >> >"non-zero >> >> >return code", "rc": 1, "start": "2020-04-15 10:57:26.050203", >> >"stderr": >> >> >"", >> >> >"stderr_lines": [], "stdout": "", "stdout_lines": []} >> >> > >> >> >On Wed, Apr 15, 2020 at 10:23 AM Shareef Jalloq >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >wrote: >> >> > >> >> >> Ha, spoke too soon. It's now stuck in a loop and a google >points >> >me >> >> >at >> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1746585 >> >> >> >> >> >> However, forcing ipv4 doesn't seem to have fixed the loop. >> >> >> >> >> >> On Wed, Apr 15, 2020 at 9:59 AM Shareef Jalloq >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >> wrote: >> >> >> >> >> >>> OK, that seems to have fixed it, thanks. Is this a side >effect >> >of >> >> >>> redeploying the HE over a first time install? Nothing has >changed >> >in >> >> >our >> >> >>> setup and I didn't need to do this when I initially set up our >> >> >nodes. >> >> >>> >> >> >>> >> >> >>> >> >> >>> On Tue, Apr 14, 2020 at 6:55 PM Strahil Nikolov >> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >>> wrote: >> >> >>> >> >> >>>> On April 14, 2020 6:17:17 PM GMT+03:00, Shareef Jalloq < >> >> >>>> shareef(a)jalloq.co.uk&gt; wrote: >> >> >>>> >Hmmm, we're not using ipv6. Is that the issue? >> >> >>>> > >> >> >>>> >On Tue, Apr 14, 2020 at 3:56 PM Strahil Nikolov >> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >>>> >wrote: >> >> >>>> > >> >> >>>> >> On April 14, 2020 1:27:24 PM GMT+03:00, Shareef Jalloq < >> >> >>>> >> shareef(a)jalloq.co.uk&gt; wrote: >> >> >>>> >> >Right, I've given up on recovering the HE so want to try >and >> >> >>>> >redeploy >> >> >>>> >> >it. >> >> >>>> >> >There doesn't seem to be enough information to debug why >the >> >> >>>> >> >broker/agent >> >> >>>> >> >won't start cleanly. >> >> >>>> >> > >> >> >>>> >> >In running 'hosted-engine --deploy', I'm seeing the >> >following >> >> >error >> >> >>>> >in >> >> >>>> >> >the >> >> >>>> >> >setup validation phase: >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:08,922+0000 DEBUG >> >> >otopi.plugins.otopi.dialog.human >> >> >>>> >> >dialog.__logString:204 DIALOG:SEND Please >> >> >provide >> >> >>>> >the >> >> >>>> >> >hostname of this host on the management network >> >> >>>> >> >[ovirt-node-00.phoelex.com]: >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,831+0000 DEBUG >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >>>> >> >hostname.getResolvedAddresses:432 >> >> >>>> >> >getResolvedAddresses: set(['64:ff9b::c0a8:13d', >> >> >'192.168.1.61']) >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,832+0000 DEBUG >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >>>> >> >hostname._validateFQDNresolvability:289 >> >> >ovirt-node-00.phoelex.com >> >> >>>> >> >resolves >> >> >>>> >> >to: set(['64:ff9b::c0a8:13d', '192.168.1.61']) >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,832+0000 DEBUG >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >plugin.executeRaw:813 >> >> >>>> >> >execute: >> >> >>>> >> >['/usr/bin/dig', '+noall', '+answer', >> >> >'ovirt-node-00.phoelex.com', >> >> >>>> >> >'ANY'], >> >> >>>> >> >executable='None', cwd='None', env=None >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,871+0000 DEBUG >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >plugin.executeRaw:863 >> >> >>>> >> >execute-result: ['/usr/bin/dig', '+noall', '+answer', ' >> >> >>>> >> >ovirt-node-00.phoelex.com', 'ANY'], rc=0 >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,872+0000 DEBUG >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >plugin.execute:921 >> >> >>>> >> >execute-output: ['/usr/bin/dig', '+noall', '+answer', ' >> >> >>>> >> >ovirt-node-00.phoelex.com', 'ANY'] stdout: >> >> >>>> >> > >> >> >>>> >> >ovirt-node-00.phoelex.com. 86400 IN A >192.168.1.61 >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,872+0000 DEBUG >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >plugin.execute:926 >> >> >>>> >> >execute-output: ['/usr/bin/dig', '+noall', '+answer', ' >> >> >>>> >> >ovirt-node-00.phoelex.com', 'ANY'] stderr: >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,872+0000 DEBUG >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >plugin.executeRaw:813 >> >> >>>> >> >execute: >> >> >>>> >> >('/usr/sbin/ip', 'addr'), executable='None', cwd='None', >> >> >env=None >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,876+0000 DEBUG >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >plugin.executeRaw:863 >> >> >>>> >> >execute-result: ('/usr/sbin/ip', 'addr'), rc=0 >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,876+0000 DEBUG >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >plugin.execute:921 >> >> >>>> >> >execute-output: ('/usr/sbin/ip', 'addr') stdout: >> >> >>>> >> > >> >> >>>> >> >1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue >state >> >> >UNKNOWN >> >> >>>> >> >group >> >> >>>> >> >default qlen 1000 >> >> >>>> >> > >> >> >>>> >> > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >> >> >>>> >> > >> >> >>>> >> > inet 127.0.0.1/8 scope host lo >> >> >>>> >> > >> >> >>>> >> > valid_lft forever preferred_lft forever >> >> >>>> >> > >> >> >>>> >> > inet6 ::1/128 scope host >> >> >>>> >> > >> >> >>>> >> > valid_lft forever preferred_lft forever >> >> >>>> >> > >> >> >>>> >> >2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >mq >> >> >master >> >> >>>> >> >ovirtmgmt state UP group default qlen 1000 >> >> >>>> >> > >> >> >>>> >> > link/ether ac:1f:6b:bc:32:6a brd ff:ff:ff:ff:ff:ff >> >> >>>> >> > >> >> >>>> >> >3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 >qdisc >> >mq >> >> >state >> >> >>>> >> >DOWN >> >> >>>> >> >group default qlen 1000 >> >> >>>> >> > >> >> >>>> >> > link/ether ac:1f:6b:bc:32:6b brd ff:ff:ff:ff:ff:ff >> >> >>>> >> > >> >> >>>> >> >4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop >> >state >> >> >DOWN >> >> >>>> >> >group >> >> >>>> >> >default qlen 1000 >> >> >>>> >> > >> >> >>>> >> > link/ether 02:e6:e2:80:93:8d brd ff:ff:ff:ff:ff:ff >> >> >>>> >> > >> >> >>>> >> >5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop >state >> >DOWN >> >> >>>> >group >> >> >>>> >> >default qlen 1000 >> >> >>>> >> > >> >> >>>> >> > link/ether 8a:26:44:50:ee:4a brd ff:ff:ff:ff:ff:ff >> >> >>>> >> > >> >> >>>> >> >21: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 >> >qdisc >> >> >>>> >noqueue >> >> >>>> >> >state UP group default qlen 1000 >> >> >>>> >> > >> >> >>>> >> > link/ether ac:1f:6b:bc:32:6a brd ff:ff:ff:ff:ff:ff >> >> >>>> >> > >> >> >>>> >> > inet 192.168.1.61/24 brd 192.168.1.255 scope global >> >> >ovirtmgmt >> >> >>>> >> > >> >> >>>> >> > valid_lft forever preferred_lft forever >> >> >>>> >> > >> >> >>>> >> > inet6 fe80::ae1f:6bff:febc:326a/64 scope link >> >> >>>> >> > >> >> >>>> >> > valid_lft forever preferred_lft forever >> >> >>>> >> > >> >> >>>> >> >22: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc >noop >> >> >state >> >> >>>> >DOWN >> >> >>>> >> >group >> >> >>>> >> >default qlen 1000 >> >> >>>> >> > >> >> >>>> >> > link/ether 3a:02:7b:7d:b3:2a brd ff:ff:ff:ff:ff:ff >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,876+0000 DEBUG >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >plugin.execute:926 >> >> >>>> >> >execute-output: ('/usr/sbin/ip', 'addr') stderr: >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,877+0000 DEBUG >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >>>> >> >hostname.getLocalAddresses:251 >> >> >>>> >> >addresses: [u'192.168.1.61', >u'fe80::ae1f:6bff:febc:326a'] >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,877+0000 DEBUG >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >hostname.test_hostname:464 >> >> >>>> >> >test_hostname exception >> >> >>>> >> > >> >> >>>> >> >Traceback (most recent call last): >> >> >>>> >> > >> >> >>>> >> >File >> >> >"/usr/lib/python2.7/site-packages/ovirt_setup_lib/hostname.py", >> >> >>>> >> >line >> >> >>>> >> >460, in test_hostname >> >> >>>> >> > >> >> >>>> >> > not_local_text, >> >> >>>> >> > >> >> >>>> >> >File >> >> >"/usr/lib/python2.7/site-packages/ovirt_setup_lib/hostname.py", >> >> >>>> >> >line >> >> >>>> >> >342, in _validateFQDNresolvability >> >> >>>> >> > >> >> >>>> >> > addresses=resolvedAddressesAsString >> >> >>>> >> > >> >> >>>> >> >RuntimeError: ovirt-node-00.phoelex.com resolves to >> >> >>>> >64:ff9b::c0a8:13d >> >> >>>> >> >192.168.1.61 and not all of them can be mapped to non >> >loopback >> >> >>>> >devices >> >> >>>> >> >on >> >> >>>> >> >this host >> >> >>>> >> > >> >> >>>> >> >2020-04-14 09:46:12,884+0000 ERROR >> >> >>>> >> >otopi.plugins.gr_he_common.network.bridge >> >> >dialog.queryEnvKey:120 >> >> >>>> >Host >> >> >>>> >> >name >> >> >>>> >> >is not valid: ovirt-node-00.phoelex.com resolves to >> >> >>>> >64:ff9b::c0a8:13d >> >> >>>> >> >192.168.1.61 and not all of them can be mapped to non >> >loopback >> >> >>>> >devices >> >> >>>> >> >on >> >> >>>> >> >this host >> >> >>>> >> > >> >> >>>> >> >The node I'm running on has an IP address of .61 and >> >resolves >> >> >>>> >> >correctly. >> >> >>>> >> > >> >> >>>> >> >On Fri, Apr 10, 2020 at 12:55 PM Shareef Jalloq >> >> >>>> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >>>> >> >wrote: >> >> >>>> >> > >> >> >>>> >> >> Where should I be checking if there are any >files/folder >> >not >> >> >owned >> >> >>>> >by >> >> >>>> >> >> vdsm:kvm? I checked on the mount the HA sits on and >it's >> >> >fine. >> >> >>>> >> >> >> >> >>>> >> >> How would I go about checking vdsm can access those >> >images? >> >> >If I >> >> >>>> >run >> >> >>>> >> >> virsh, it lists them and they were running yesterday >even >> >> >though >> >> >>>> >the >> >> >>>> >> >HA was >> >> >>>> >> >> down. I've since restarted both hosts but the broker >is >> >> >still >> >> >>>> >> >spitting out >> >> >>>> >> >> the same error (copied below). How do I find the >reason >> >the >> >> >>>> >broker >> >> >>>> >> >can't >> >> >>>> >> >> connect to the storage? The conf file is already at >DEBUG >> >> >>>> >verbosity: >> >> >>>> >> >> >> >> >>>> >> >> [handler_logfile] >> >> >>>> >> >> >> >> >>>> >> >> class=logging.handlers.TimedRotatingFileHandler >> >> >>>> >> >> >> >> >>>> >> >> args=('/var/log/ovirt-hosted-engine-ha/broker.log', >'d', >> >1, >> >> >7) >> >> >>>> >> >> >> >> >>>> >> >> level=DEBUG >> >> >>>> >> >> >> >> >>>> >> >> formatter=long >> >> >>>> >> >> >> >> >>>> >> >> And what are all these .prob-<num> files that are being >> >> >created? >> >> >>>> >> >There >> >> >>>> >> >> are over 250K of them now on the mount I'm using for >the >> >Data >> >> >>>> >domain. >> >> >>>> >> >> They're all of 0 size and of the form, >> >> >>>> >> >> /rhev/data-center/mnt/nas-01.phoelex.com: >> >> >>>> >> >> >> >_volume2_vmstore/.prob-ffa867da-93db-4211-82df-b1b04a625ab9 >> >> >>>> >> >> >> >> >>>> >> >> @eevans: The volume I have the Data Domain on has TB's >> >free. >> >> > The >> >> >>>> >HA >> >> >>>> >> >is >> >> >>>> >> >> dead so I can't ssh in. No idea what started these >errors >> >> >and the >> >> >>>> >> >other >> >> >>>> >> >> VMs were still running happily although they're on a >> >> >different >> >> >>>> >Data >> >> >>>> >> >Domain. >> >> >>>> >> >> >> >> >>>> >> >> Shareef. >> >> >>>> >> >> >> >> >>>> >> >> MainThread::INFO::2020-04-10 >> >> >>>> >> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>07:45:00,408::storage_backends::373::ovirt_hosted_engine_ha.lib.storage_backends::(connect) >> >> >>>> >> >> Connecting the storage >> >> >>>> >> >> >> >> >>>> >> >> MainThread::INFO::2020-04-10 >> >> >>>> >> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>07:45:00,408::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> >> >> Connecting storage server >> >> >>>> >> >> >> >> >>>> >> >> MainThread::INFO::2020-04-10 >> >> >>>> >> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>07:45:01,577::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> >> >> Connecting storage server >> >> >>>> >> >> >> >> >>>> >> >> MainThread::INFO::2020-04-10 >> >> >>>> >> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>07:45:02,692::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> >> >> Refreshing the storage domain >> >> >>>> >> >> >> >> >>>> >> >> MainThread::WARNING::2020-04-10 >> >> >>>> >> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>07:45:05,175::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) >> >> >>>> >> >> Can't connect vdsm storage: Command >StorageDomain.getInfo >> >> >with >> >> >>>> >args >> >> >>>> >> >> {'storagedomainID': >> >'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} >> >> >>>> >failed: >> >> >>>> >> >> >> >> >>>> >> >> (code=350, message=Error in storage domain action: >> >> >>>> >> >> (u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> >> >>>> >> >> >> >> >>>> >> >> On Thu, Apr 9, 2020 at 5:58 PM Strahil Nikolov >> >> >>>> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >>>> >> >> wrote: >> >> >>>> >> >> >> >> >>>> >> >>> On April 9, 2020 11:12:30 AM GMT+03:00, Shareef Jalloq >< >> >> >>>> >> >>> shareef(a)jalloq.co.uk&gt; wrote: >> >> >>>> >> >>> >OK, let's go through this. I'm looking at the node >that >> >at >> >> >>>> >least >> >> >>>> >> >still >> >> >>>> >> >>> >has >> >> >>>> >> >>> >some VMs running. virsh also tells me that the >> >> >HostedEngine VM >> >> >>>> >is >> >> >>>> >> >>> >running >> >> >>>> >> >>> >but it's unresponsive and I can't shut it down. >> >> >>>> >> >>> > >> >> >>>> >> >>> >1. All storage domains exist and are mounted. >> >> >>>> >> >>> >2. The ha_agent exists: >> >> >>>> >> >>> > >> >> >>>> >> >>> >[root@ovirt-node-01 ovirt-hosted-engine-ha]# ls >> >> >>>> >> >/rhev/data-center/mnt/ >> >> >>>> >> >>> >nas-01.phoelex.com >> >> >>>> >> >>> >\:_volume2_vmstore/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/ >> >> >>>> >> >>> > >> >> >>>> >> >>> >dom_md ha_agent images master >> >> >>>> >> >>> > >> >> >>>> >> >>> >3. There are two links >> >> >>>> >> >>> > >> >> >>>> >> >>> >[root@ovirt-node-01 ovirt-hosted-engine-ha]# ll >> >> >>>> >> >/rhev/data-center/mnt/ >> >> >>>> >> >>> >nas-01.phoelex.com >> >> >>>> >> >>> >> >> >>>> >> >>>\:_volume2_vmstore/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/ha_agent/ >> >> >>>> >> >>> > >> >> >>>> >> >>> >total 8 >> >> >>>> >> >>> > >> >> >>>> >> >>> >lrwxrwxrwx. 1 vdsm kvm 132 Apr 2 14:50 >> >> >hosted-engine.lockspace >> >> >>>> >-> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>/var/run/vdsm/storage/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/ffb90b82-42fe-4253-85d5-aaec8c280aaf/90e68791-0c6f-406a-89ac-e0d86c631604 >> >> >>>> >> >>> > >> >> >>>> >> >>> >lrwxrwxrwx. 1 vdsm kvm 132 Apr 2 14:50 >> >> >hosted-engine.metadata >> >> >>>> >-> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>/var/run/vdsm/storage/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/2161aed0-7250-4c1d-b667-ac94f60af17e/6b818e33-f80a-48cc-a59c-bba641e027d4 >> >> >>>> >> >>> > >> >> >>>> >> >>> >4. The services exist but all seem to have some sort >of >> >> >warning: >> >> >>>> >> >>> > >> >> >>>> >> >>> >a) Apr 08 18:10:55 ovirt-node-01.phoelex.com >> >sanlock[1728]: >> >> >>>> >> >*2020-04-08 >> >> >>>> >> >>> >18:10:55 1744152 [36796]: s16 delta_renew long write >> >time >> >> >10 >> >> >>>> >sec* >> >> >>>> >> >>> > >> >> >>>> >> >>> >b) Mar 23 18:02:59 ovirt-node-01.phoelex.com >> >> >supervdsmd[29409]: >> >> >>>> >> >*failed >> >> >>>> >> >>> >to >> >> >>>> >> >>> >load module nvdimm: libbd_nvdimm.so.2: cannot open >> >shared >> >> >object >> >> >>>> >> >file: >> >> >>>> >> >>> >No >> >> >>>> >> >>> >such file or directory* >> >> >>>> >> >>> > >> >> >>>> >> >>> >c) Apr 09 08:05:13 ovirt-node-01.phoelex.com >vdsm[4801]: >> >> >*ERROR >> >> >>>> >> >failed >> >> >>>> >> >>> >to >> >> >>>> >> >>> >retrieve Hosted Engine HA score '[Errno 2] No such >file >> >or >> >> >>>> >> >directory'Is >> >> >>>> >> >>> >the >> >> >>>> >> >>> >Hosted Engine setup finished?* >> >> >>>> >> >>> > >> >> >>>> >> >>> >d)Apr 08 22:48:27 ovirt-node-01.phoelex.com >> >> >libvirtd[29307]: >> >> >>>> >> >2020-04-08 >> >> >>>> >> >>> >22:48:27.134+0000: 29309: warning : >> >qemuGetProcessInfo:1404 >> >> >: >> >> >>>> >> >cannot >> >> >>>> >> >>> >parse >> >> >>>> >> >>> >process status data >> >> >>>> >> >>> > >> >> >>>> >> >>> >Apr 08 22:48:27 ovirt-node-01.phoelex.com >> >libvirtd[29307]: >> >> >>>> >> >2020-04-08 >> >> >>>> >> >>> >22:48:27.134+0000: 29309: error : >> >> >virNetDevTapInterfaceStats:764 >> >> >>>> >: >> >> >>>> >> >>> >internal >> >> >>>> >> >>> >error: /proc/net/dev: Interface not found >> >> >>>> >> >>> > >> >> >>>> >> >>> >Apr 08 23:09:39 ovirt-node-01.phoelex.com >> >libvirtd[29307]: >> >> >>>> >> >2020-04-08 >> >> >>>> >> >>> >23:09:39.844+0000: 29307: error : >> >virNetSocketReadWire:1806 >> >> >: >> >> >>>> >End >> >> >>>> >> >of >> >> >>>> >> >>> >file >> >> >>>> >> >>> >while reading data: Input/output error >> >> >>>> >> >>> > >> >> >>>> >> >>> >Apr 09 01:05:26 ovirt-node-01.phoelex.com >> >libvirtd[29307]: >> >> >>>> >> >2020-04-09 >> >> >>>> >> >>> >01:05:26.660+0000: 29307: error : >> >virNetSocketReadWire:1806 >> >> >: >> >> >>>> >End >> >> >>>> >> >of >> >> >>>> >> >>> >file >> >> >>>> >> >>> >while reading data: Input/output error >> >> >>>> >> >>> > >> >> >>>> >> >>> >5 & 6. The broker log is continually printing this >> >error: >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,438::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >> >>>> >> >>> >ovirt-hosted-engine-ha broker 2.3.6 started >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,438::broker::55::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >> >>>> >> >>> >Running broker >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,438::broker::120::ovirt_hosted_engine_ha.broker.broker.Broker::(_get_monitor) >> >> >>>> >> >>> >Starting monitor >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,438::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Searching for submonitors in >> >> >>>> >> >>> >> >> >>/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker >> >> >>>> >> >>> > >> >> >>>> >> >>> >/submonitors >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,439::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor network >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,440::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor cpu-load-no-engine >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor mgmt-bridge >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor network >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor cpu-load >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor engine-health >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,442::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor mgmt-bridge >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,442::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor cpu-load-no-engine >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor cpu-load >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor mem-free >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor storage-domain >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor storage-domain >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor mem-free >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,444::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Loaded submonitor engine-health >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,444::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >Finished loading submonitors >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,444::broker::128::ovirt_hosted_engine_ha.broker.broker.Broker::(_get_storage_broker) >> >> >>>> >> >>> >Starting storage broker >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,444::storage_backends::369::ovirt_hosted_engine_ha.lib.storage_backends::(connect) >> >> >>>> >> >>> >Connecting to VDSM >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,444::util::384::ovirt_hosted_engine_ha.lib.storage_backends::(__log_debug) >> >> >>>> >> >>> >Creating a new json-rpc connection to VDSM >> >> >>>> >> >>> > >> >> >>>> >> >>> >Client localhost:54321::DEBUG::2020-04-09 >> >> >>>> >> >>> >08:07:31,453::concurrent::258::root::(run) START >thread >> >> >>>> >> ><Thread(Client >> >> >>>> >> >>> >localhost:54321, started daemon 139992488138496)> >> >> >(func=<bound >> >> >>>> >> >method >> >> >>>> >> >>> >Reactor.process_requests of >> >> ><yajsonrpc.betterAsyncore.Reactor >> >> >>>> >> >object at >> >> >>>> >> >>> >0x7f528acabc90>>, args=(), kwargs={}) >> >> >>>> >> >>> > >> >> >>>> >> >>> >Client localhost:54321::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,459::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) >> >> >>>> >> >>> >Stomp connection established >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>08:07:31,467::stompclient::294::jsonrpc.AsyncoreClient::(send) >> >> >>>> >> >Sending >> >> >>>> >> >>> >response >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,530::storage_backends::373::ovirt_hosted_engine_ha.lib.storage_backends::(connect) >> >> >>>> >> >>> >Connecting the storage >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,531::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> >> >>> >Connecting storage server >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>08:07:31,531::stompclient::294::jsonrpc.AsyncoreClient::(send) >> >> >>>> >> >Sending >> >> >>>> >> >>> >response >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>08:07:31,534::stompclient::294::jsonrpc.AsyncoreClient::(send) >> >> >>>> >> >Sending >> >> >>>> >> >>> >response >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:32,199::storage_server::158::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(_validate_pre_connected_path) >> >> >>>> >> >>> >Storage domain a6cea67d-dbfb-45cf-a775-b4d0d47b26f2 >is >> >not >> >> >>>> >> >available >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:32,199::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> >> >>> >Connecting storage server >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>08:07:32,199::stompclient::294::jsonrpc.AsyncoreClient::(send) >> >> >>>> >> >Sending >> >> >>>> >> >>> >response >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:32,814::storage_server::363::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> >> >>> >[{u'status': 0, u'id': >> >> >u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}] >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::INFO::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:32,814::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> >> >>> >Refreshing the storage domain >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>08:07:32,815::stompclient::294::jsonrpc.AsyncoreClient::(send) >> >> >>>> >> >Sending >> >> >>>> >> >>> >response >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:33,129::storage_server::420::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> >> >>> >Error refreshing storage domain: Command >> >> >StorageDomain.getStats >> >> >>>> >> >with >> >> >>>> >> >>> >args >> >> >>>> >> >>> >{'storagedomainID': >> >'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} >> >> >>>> >failed: >> >> >>>> >> >>> > >> >> >>>> >> >>> >(code=350, message=Error in storage domain action: >> >> >>>> >> >>> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>08:07:33,130::stompclient::294::jsonrpc.AsyncoreClient::(send) >> >> >>>> >> >Sending >> >> >>>> >> >>> >response >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::DEBUG::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:33,795::storage_backends::208::ovirt_hosted_engine_ha.lib.storage_backends::(_get_sector_size) >> >> >>>> >> >>> >Command StorageDomain.getInfo with args >> >{'storagedomainID': >> >> >>>> >> >>> >'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} failed: >> >> >>>> >> >>> > >> >> >>>> >> >>> >(code=350, message=Error in storage domain action: >> >> >>>> >> >>> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> >> >>>> >> >>> > >> >> >>>> >> >>> >MainThread::WARNING::2020-04-09 >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:33,795::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) >> >> >>>> >> >>> >Can't connect vdsm storage: Command >> >StorageDomain.getInfo >> >> >with >> >> >>>> >args >> >> >>>> >> >>> >{'storagedomainID': >> >'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} >> >> >>>> >failed: >> >> >>>> >> >>> > >> >> >>>> >> >>> >(code=350, message=Error in storage domain action: >> >> >>>> >> >>> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> >> >>>> >> >>> > >> >> >>>> >> >>> > >> >> >>>> >> >>> >The UUID it is moaning about is indeed the one that >the >> >HA >> >> >sits >> >> >>>> >on >> >> >>>> >> >and >> >> >>>> >> >>> >is >> >> >>>> >> >>> >the one I listed the contents of in step 2 above. >> >> >>>> >> >>> > >> >> >>>> >> >>> > >> >> >>>> >> >>> >So why can't it see this domain? >> >> >>>> >> >>> > >> >> >>>> >> >>> > >> >> >>>> >> >>> >Thanks, Shareef. >> >> >>>> >> >>> > >> >> >>>> >> >>> >On Thu, Apr 9, 2020 at 6:12 AM Strahil Nikolov >> >> >>>> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >>>> >> >>> >wrote: >> >> >>>> >> >>> > >> >> >>>> >> >>> >> On April 9, 2020 1:51:05 AM GMT+03:00, Shareef >Jalloq >> >< >> >> >>>> >> >>> >> shareef(a)jalloq.co.uk&gt; wrote: >> >> >>>> >> >>> >> >Don't know if this is useful or not, but I just >tried >> >to >> >> >>>> >> >shutdown >> >> >>>> >> >>> >and >> >> >>>> >> >>> >> >start >> >> >>>> >> >>> >> >another VM on one of the hosts and get the >following >> >> >error: >> >> >>>> >> >>> >> > >> >> >>>> >> >>> >> >virsh # start scratch >> >> >>>> >> >>> >> > >> >> >>>> >> >>> >> >error: Failed to start domain scratch >> >> >>>> >> >>> >> > >> >> >>>> >> >>> >> >error: Network not found: no network with matching >> >name >> >> >>>> >> >>> >> >'vdsm-ovirtmgmt' >> >> >>>> >> >>> >> > >> >> >>>> >> >>> >> >Is this not referring to the interface name as the >> >> >network is >> >> >>>> >> >called >> >> >>>> >> >>> >> >'ovirtmgnt'. >> >> >>>> >> >>> >> > >> >> >>>> >> >>> >> >On Wed, Apr 8, 2020 at 11:35 PM Shareef Jalloq >> >> >>>> >> >>> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >>>> >> >>> >> >wrote: >> >> >>>> >> >>> >> > >> >> >>>> >> >>> >> >> Hmmm, virsh tells me the HE is running but it >> >hasn't >> >> >come >> >> >>>> >up >> >> >>>> >> >and >> >> >>>> >> >>> >the >> >> >>>> >> >>> >> >> agent.log is full of the same errors. >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >> On Wed, Apr 8, 2020 at 11:31 PM Shareef Jalloq >> >> >>>> >> >>> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >>>> >> >>> >> >> wrote: >> >> >>>> >> >>> >> >> >> >> >>>> >> >>> >> >>> Ah hah! Ok, so I've managed to start it using >> >virsh >> >> >on >> >> >>>> >the >> >> >>>> >> >>> >second >> >> >>>> >> >>> >> >host >> >> >>>> >> >>> >> >>> but my first host is still dead. >> >> >>>> >> >>> >> >>> >> >> >>>> >> >>> >> >>> First of all, what are these 56,317 .prob- >files >> >that >> >> >get >> >> >>>> >> >dumped >> >> >>>> >> >>> >to >> >> >>>> >> >>> >> >the >> >> >>>> >> >>> >> >>> NFS mounts? >> >> >>>> >> >>> >> >>> >> >> >>>> >> >>> >> >>> Secondly, why doesn't the node mount the NFS >> >> >directories >> >> >>>> >at >> >> >>>> >> >boot? >> >> >>>> >> >>> >> >Is >> >> >>>> >> >>> >> >>> that the issue with this particular node? >> >> >>>> >> >>> >> >>> >> >> >>>> >> >>> >> >>> On Wed, Apr 8, 2020 at 11:12 PM >> >> >>>> &gt;&lt;eevans(a)digitaldatatechs.com&gt; >> >> >>>> >> >>> >wrote: >> >> >>>> >> >>> >> >>> >> >> >>>> >> >>> >> >>>> Did you try virsh list --inactive >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> Eric Evans >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> Digital Data Services LLC. >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> 304.660.9080 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> *From:* Shareef Jalloq <shareef(a)jalloq.co.uk&gt; >> >> >>>> >> >>> >> >>>> *Sent:* Wednesday, April 8, 2020 5:58 PM >> >> >>>> >> >>> >> >>>> *To:* Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; >> >> >>>> >> >>> >> >>>> *Cc:* Ovirt Users <users(a)ovirt.org&gt; >> >> >>>> >> >>> >> >>>> *Subject:* [ovirt-users] Re: ovirt-engine >> >> >unresponsive - >> >> >>>> >how >> >> >>>> >> >to >> >> >>>> >> >>> >> >rescue? >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> I've now shut down the VMs on one host and >> >rebooted >> >> >it >> >> >>>> >but >> >> >>>> >> >the >> >> >>>> >> >>> >> >agent >> >> >>>> >> >>> >> >>>> service doesn't start. If I run >'hosted-engine >> >> >>>> >--vm-status' >> >> >>>> >> >I >> >> >>>> >> >>> >get: >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> The hosted engine configuration has not been >> >> >retrieved >> >> >>>> >from >> >> >>>> >> >>> >shared >> >> >>>> >> >>> >> >>>> storage. Please ensure that ovirt-ha-agent is >> >> >running and >> >> >>>> >> >the >> >> >>>> >> >>> >> >storage >> >> >>>> >> >>> >> >>>> server is reachable. >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> and indeed if I list the mounts under >> >> >>>> >/rhev/data-center/mnt, >> >> >>>> >> >>> >only >> >> >>>> >> >>> >> >one of >> >> >>>> >> >>> >> >>>> the directories is mounted. I have 3 NFS >mounts, >> >> >one ISO >> >> >>>> >> >Domain >> >> >>>> >> >>> >> >and two >> >> >>>> >> >>> >> >>>> Data Domains. Only one Data Domain has >mounted >> >and >> >> >this >> >> >>>> >has >> >> >>>> >> >>> >lots >> >> >>>> >> >>> >> >of .prob >> >> >>>> >> >>> >> >>>> files in. So why haven't the other NFS >exports >> >been >> >> >>>> >> >mounted? >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> Manually mounting them doesn't seem to have >> >helped >> >> >much >> >> >>>> >> >either. >> >> >>>> >> >>> >I >> >> >>>> >> >>> >> >can >> >> >>>> >> >>> >> >>>> start the broker service but the agent service >> >says >> >> >no. >> >> >>>> >> >Same >> >> >>>> >> >>> >error >> >> >>>> >> >>> >> >as the >> >> >>>> >> >>> >> >>>> one in my last email. >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> Shareef. >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> On Wed, Apr 8, 2020 at 9:57 PM Shareef Jalloq >> >> >>>> >> >>> >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >>>> >> >>> >> >>>> wrote: >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> Right, still down. I've run virsh and it >doesn't >> >> >know >> >> >>>> >> >anything >> >> >>>> >> >>> >> >about >> >> >>>> >> >>> >> >>>> the engine vm. >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> I've restarted the broker and agent services >and >> >I >> >> >still >> >> >>>> >get >> >> >>>> >> >>> >> >nothing in >> >> >>>> >> >>> >> >>>> virsh->list. >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> In the logs under >/var/log/ovirt-hosted-engine-ha >> >I >> >> >see >> >> >>>> >lots >> >> >>>> >> >of >> >> >>>> >> >>> >> >errors: >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> broker.log: >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,138::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >> >>>> >> >>> >> >>>> ovirt-hosted-engine-ha broker 2.3.6 started >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,138::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Searching for submonitors in >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,138::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor network >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,140::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor cpu-load-no-engine >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,140::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor mgmt-bridge >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor network >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor cpu-load >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor engine-health >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor mgmt-bridge >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,142::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor cpu-load-no-engine >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,142::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor cpu-load >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,142::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor mem-free >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor storage-domain >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor storage-domain >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor mem-free >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Loaded submonitor engine-health >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,143::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Finished loading submonitors >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,197::storage_backends::373::ovirt_hosted_engine_ha.lib.storage_backends::(connect) >> >> >>>> >> >>> >> >>>> Connecting the storage >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,197::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> >> >>> >> >>>> Connecting storage server >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,414::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> >> >>> >> >>>> Connecting storage server >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:20,628::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> >> >>> >> >>>> Refreshing the storage domain >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::WARNING::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:21,057::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) >> >> >>>> >> >>> >> >>>> Can't connect vdsm storage: Command >> >> >StorageDomain.getInfo >> >> >>>> >> >with >> >> >>>> >> >>> >args >> >> >>>> >> >>> >> >>>> {'storagedomainID': >> >> >>>> >'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} >> >> >>>> >> >>> >failed: >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> (code=350, message=Error in storage domain >> >action: >> >> >>>> >> >>> >> >>>> >> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:21,901::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >> >>>> >> >>> >> >>>> ovirt-hosted-engine-ha broker 2.3.6 started >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:56:21,901::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> >> >>> >> >>>> Searching for submonitors in >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> agent.log: >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::ERROR::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:57:00,799::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) >> >> >>>> >> >>> >> >>>> Trying to restart agent >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:57:00,799::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) >> >> >>>> >> >>> >> >>>> Agent shutting down >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:57:11,144::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) >> >> >>>> >> >>> >> >>>> ovirt-hosted-engine-ha agent 2.3.6 started >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:57:11,182::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) >> >> >>>> >> >>> >> >>>> Found certificate common name: >> >> >ovirt-node-01.phoelex.com >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:57:11,294::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) >> >> >>>> >> >>> >> >>>> Initializing ha-broker connection >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:57:11,296::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) >> >> >>>> >> >>> >> >>>> Starting monitor network, options >> >{'tcp_t_address': >> >> >'', >> >> >>>> >> >>> >> >'network_test': >> >> >>>> >> >>> >> >>>> 'dns', 'tcp_t_port': '', 'addr': >'192.168.1.99'} >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::ERROR::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:57:11,296::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) >> >> >>>> >> >>> >> >>>> Failed to start necessary monitors >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::ERROR::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:57:11,297::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) >> >> >>>> >> >>> >> >>>> Traceback (most recent call last): >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> File >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", >> >> >>>> >> >>> >> >>>> line 131, in _run_agent >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> return action(he) >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> File >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", >> >> >>>> >> >>> >> >>>> line 55, in action_proper >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> return he.start_monitoring() >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> File >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >> >> >>>> >> >>> >> >>>> line 432, in start_monitoring >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> self._initialize_broker() >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> File >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >> >> >>>> >> >>> >> >>>> line 556, in _initialize_broker >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> m.get('options', {})) >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> File >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", >> >> >>>> >> >>> >> >>>> line 89, in start_monitor >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> ).format(t=type, o=options, e=e) >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> RequestError: brokerlink - failed to start >> >monitor >> >> >via >> >> >>>> >> >>> >> >ovirt-ha-broker: >> >> >>>> >> >>> >> >>>> [Errno 2] No such file or directory, [monitor: >> >> >'network', >> >> >>>> >> >>> >options: >> >> >>>> >> >>> >> >>>> {'tcp_t_address': '', 'network_test': 'dns', >> >> >>>> >'tcp_t_port': >> >> >>>> >> >'', >> >> >>>> >> >>> >> >'addr': >> >> >>>> >> >>> >> >>>> '192.168.1.99'}] >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::ERROR::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:57:11,297::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) >> >> >>>> >> >>> >> >>>> Trying to restart agent >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>20:57:11,297::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) >> >> >>>> >> >>> >> >>>> Agent shutting down >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> On Wed, Apr 8, 2020 at 6:10 PM Strahil Nikolov >> >> >>>> >> >>> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >>>> >> >>> >> >>>> wrote: >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> On April 8, 2020 7:47:20 PM GMT+03:00, "Maton, >> >> >Brett" < >> >> >>>> >> >>> >> >>>> matonb(a)ltresources.co.uk&gt; wrote: >> >> >>>> >> >>> >> >>>> >On the host you tried to restart the engine >on: >> >> >>>> >> >>> >> >>>> > >> >> >>>> >> >>> >> >>>> >Add an alias to virsh (authenticates with >> >> >>>> >virsh_auth.conf) >> >> >>>> >> >>> >> >>>> > >> >> >>>> >> >>> >> >>>> >alias virsh='virsh -c >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >>>> >> >> >>>>>qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf' >> >> >>>> >> >>> >> >>>> > >> >> >>>> >> >>> >> >>>> >Then run virsh: >> >> >>>> >> >>> >> >>>> > >> >> >>>> >> >>> >> >>>> >virsh >> >> >>>> >> >>> >> >>>> > >> >> >>>> >> >>> >> >>>> >virsh # list >> >> >>>> >> >>> >> >>>> > Id Name State >> >> >>>> >> >>> >> >>>> >> >> >>---------------------------------------------------- >> >> >>>> >> >>> >> >>>> > xx HostedEngine Paused >> >> >>>> >> >>> >> >>>> > xx ********** running >> >> >>>> >> >>> >> >>>> > ... >> >> >>>> >> >>> >> >>>> > xx ********** >running >> >> >>>> >> >>> >> >>>> > >> >> >>>> >> >>> >> >>>> >HostedEngine should be in the list, try and >> >resume >> >> >the >> >> >>>> >> >engine: >> >> >>>> >> >>> >> >>>> > >> >> >>>> >> >>> >> >>>> >virsh # resume HostedEngine >> >> >>>> >> >>> >> >>>> > >> >> >>>> >> >>> >> >>>> >On Wed, 8 Apr 2020 at 17:28, Shareef Jalloq >> >> >>>> >> >>> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >>>> >> >>> >> >>>> >wrote: >> >> >>>> >> >>> >> >>>> > >> >> >>>> >> >>> >> >>>> >> Thanks! >> >> >>>> >> >>> >> >>>> >> >> >> >>>> >> >>> >> >>>> >> The status hangs due to, I guess, the VM >being >> >> >>>> >down.... >> >> >>>> >> >>> >> >>>> >> >> >> >>>> >> >>> >> >>>> >> [root@ovirt-node-01 ~]# hosted-engine >> >--vm-start >> >> >>>> >> >>> >> >>>> >> VM exists and is down, cleaning up and >> >restarting >> >> >>>> >> >>> >> >>>> >> VM in WaitForLaunch >> >> >>>> >> >>> >> >>>> >> >> >> >>>> >> >>> >> >>>> >> but this doesn't seem to do anything. OK, >> >after >> >> >a >> >> >>>> >while >> >> >>>> >> >I >> >> >>>> >> >>> >get a >> >> >>>> >> >>> >> >>>> >status of >> >> >>>> >> >>> >> >>>> >> it being barfed... >> >> >>>> >> >>> >> >>>> >> >> >> >>>> >> >>> >> >>>> >> --== Host ovirt-node-00.phoelex.com (id: 1) >> >> >status >> >> >>>> >==-- >> >> >>>> >> >>> >> >>>> >> >> >> >>>> >> >>> >> >>>> >> conf_on_shared_storage : True >> >> >>>> >> >>> >> >>>> >> Status up-to-date : False >> >> >>>> >> >>> >> >>>> >> Hostname : >> >> >>>> >> >>> >ovirt-node-00.phoelex.com >> >> >>>> >> >>> >> >>>> >> Host ID : 1 >> >> >>>> >> >>> >> >>>> >> Engine status : >unknown >> >> >>>> >stale-data >> >> >>>> >> >>> >> >>>> >> Score : 3400 >> >> >>>> >> >>> >> >>>> >> stopped : False >> >> >>>> >> >>> >> >>>> >> Local maintenance : False >> >> >>>> >> >>> >> >>>> >> crc32 : >9c4a034b >> >> >>>> >> >>> >> >>>> >> local_conf_timestamp : 523362 >> >> >>>> >> >>> >> >>>> >> Host timestamp : 523608 >> >> >>>> >> >>> >> >>>> >> Extra metadata (valid at timestamp): >> >> >>>> >> >>> >> >>>> >> metadata_parse_version=1 >> >> >>>> >> >>> >> >>>> >> metadata_feature_version=1 >> >> >>>> >> >>> >> >>>> >> timestamp=523608 (Wed Apr 8 16:17:11 2020) >> >> >>>> >> >>> >> >>>> >> host-id=1 >> >> >>>> >> >>> >> >>>> >> score=3400 >> >> >>>> >> >>> >> >>>> >> vm_conf_refresh_time=523362 (Wed Apr 8 >> >16:13:06 >> >> >2020) >> >> >>>> >> >>> >> >>>> >> conf_on_shared_storage=True >> >> >>>> >> >>> >> >>>> >> maintenance=False >> >> >>>> >> >>> >> >>>> >> state=EngineDown >> >> >>>> >> >>> >> >>>> >> stopped=False >> >> >>>> >> >>> >> >>>> >> >> >> >>>> >> >>> >> >>>> >> >> >> >>>> >> >>> >> >>>> >> --== Host ovirt-node-01.phoelex.com (id: 2) >> >> >status >> >> >>>> >==-- >> >> >>>> >> >>> >> >>>> >> >> >> >>>> >> >>> >> >>>> >> conf_on_shared_storage : True >> >> >>>> >> >>> >> >>>> >> Status up-to-date : True >> >> >>>> >> >>> >> >>>> >> Hostname : >> >> >>>> >> >>> >ovirt-node-01.phoelex.com >> >> >>>> >> >>> >> >>>> >> Host ID : 2 >> >> >>>> >> >>> >> >>>> >> Engine status : >> >{"reason": >> >> >"bad >> >> >>>> >vm >> >> >>>> >> >>> >status", >> >> >>>> >> >>> >> >>>> >"health": >> >> >>>> >> >>> >> >>>> >> "bad", "vm": "down_unexpected", "detail": >> >"Down"} >> >> >>>> >> >>> >> >>>> >> Score : 0 >> >> >>>> >> >>> >> >>>> >> stopped : False >> >> >>>> >> >>> >> >>>> >> Local maintenance : False >> >> >>>> >> >>> >> >>>> >> crc32 : >5045f2eb >> >> >>>> >> >>> >> >>>> >> local_conf_timestamp : >1737037 >> >> >>>> >> >>> >> >>>> >> Host timestamp : >1737283 >> >> >>>> >> >>> >> >>>> >> Extra metadata (valid at timestamp): >> >> >>>> >> >>> >> >>>> >> metadata_parse_version=1 >> >> >>>> >> >>> >> >>>> >> metadata_feature_version=1 >> >> >>>> >> >>> >> >>>> >> timestamp=1737283 (Wed Apr 8 16:16:17 >2020) >> >> >>>> >> >>> >> >>>> >> host-id=2 >> >> >>>> >> >>> >> >>>> >> score=0 >> >> >>>> >> >>> >> >>>> >> vm_conf_refresh_time=1737037 (Wed Apr 8 >> >16:12:11 >> >> >>>> >2020) >> >> >>>> >> >>> >> >>>> >> conf_on_shared_storage=True >> >> >>>> >> >>> >> >>>> >> maintenance=False >> >> >>>> >> >>> >> >>>> >> state=EngineUnexpectedlyDown >> >> >>>> >> >>> >> >>>> >> stopped=False >> >> >>>> >> >>> >> >>>> >> >> >> >>>> >> >>> >> >>>> >> On Wed, Apr 8, 2020 at 5:09 PM Maton, Brett >> >> >>>> >> >>> >> >>>> &gt;&lt;matonb(a)ltresources.co.uk&gt; >> >> >>>> >> >>> >> >>>> >> wrote: >> >> >>>> >> >>> >> >>>> >> >> >> >>>> >> >>> >> >>>> >>> First steps, on one of your hosts as root: >> >> >>>> >> >>> >> >>>> >>> >> >> >>>> >> >>> >> >>>> >>> To get information: >> >> >>>> >> >>> >> >>>> >>> hosted-engine --vm-status >> >> >>>> >> >>> >> >>>> >>> >> >> >>>> >> >>> >> >>>> >>> To start the engine: >> >> >>>> >> >>> >> >>>> >>> hosted-engine --vm-start >> >> >>>> >> >>> >> >>>> >>> >> >> >>>> >> >>> >> >>>> >>> >> >> >>>> >> >>> >> >>>> >>> On Wed, 8 Apr 2020 at 17:00, Shareef >Jalloq >> >> >>>> >> >>> >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >>>> >> >>> >> >>>> >wrote: >> >> >>>> >> >>> >> >>>> >>> >> >> >>>> >> >>> >> >>>> >>>> So my engine has gone down and I can't >ssh >> >into >> >> >it >> >> >>>> >> >either. >> >> >>>> >> >>> >If >> >> >>>> >> >>> >> >I >> >> >>>> >> >>> >> >>>> >try to >> >> >>>> >> >>> >> >>>> >>>> log into the web-ui of the node it is >> >running >> >> >on, I >> >> >>>> >get >> >> >>>> >> >>> >> >redirected >> >> >>>> >> >>> >> >>>> >because >> >> >>>> >> >>> >> >>>> >>>> the node can't reach the engine. >> >> >>>> >> >>> >> >>>> >>>> >> >> >>>> >> >>> >> >>>> >>>> What are my next steps? >> >> >>>> >> >>> >> >>>> >>>> >> >> >>>> >> >>> >> >>>> >>>> Shareef. >> >> >>>> >> >>> >> >>>> >>>> >> >_______________________________________________ >> >> >>>> >> >>> >> >>>> >>>> Users mailing list -- users(a)ovirt.org >> >> >>>> >> >>> >> >>>> >>>> To unsubscribe send an email to >> >> >>>> &gt;users-leave(a)ovirt.org >> >> >>>> >> >>> >> >>>> >>>> Privacy Statement: >> >> >>>> >> >>> >https://www.ovirt.org/privacy-policy.html >> >> >>>> >> >>> >> >>>> >>>> oVirt Code of Conduct: >> >> >>>> >> >>> >> >>>> >>>> >> >> >>>> >> >>https://www.ovirt.org/community/about/community-guidelines/ >> >> >>>> >> >>> >> >>>> >>>> List Archives: >> >> >>>> >> >>> >> >>>> >>>> >> >> >>>> >> >>> >> >>>> > >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> > >> >> >>>> >> >>> >> >> >> >>>> >> >>> > >> >> >>>> >> >>> >> >> >>>> >> > >> >> >>>> >> >> >> >>>> > >> >> >>>> >> >> > >> >> >> > >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/W7BP57OCIRS... >> >> >>>> >> >>> >> >>>> >>>> >> >> >>>> >> >>> >> >>>> >>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> This has to be resolved: >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> Engine status : unknown >> >> >stale-data >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> Run again 'hosted-engine --vm-status'. If it >> >remains >> >> >the >> >> >>>> >> >same, >> >> >>>> >> >>> >> >restart >> >> >>>> >> >>> >> >>>> ovirt-ha-broker.service & >ovirt-ha-agent.service >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> Verify that the engine's storage is available. >> >Then >> >> >>>> >monitor >> >> >>>> >> >the >> >> >>>> >> >>> >> >broker >> >> >>>> >> >>> >> >>>> & agent logs in >/var/log/ovirt-hosted-engine-ha >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> Best Regards, >> >> >>>> >> >>> >> >>>> Strahil Nikolov >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> Hi Shareef, >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> The flow of activation oVirt is more complex than a >> >plain >> >> >KVM. >> >> >>>> >> >>> >> Mounting of the domains happen during the >activation >> >of >> >> >the >> >> >>>> >node >> >> >>>> >> >( >> >> >>>> >> >>> >the >> >> >>>> >> >>> >> HostedEngine is activating everything needed). >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> Focus on the HostedEngine VM. >> >> >>>> >> >>> >> Is it running properly ? >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> If not,try: >> >> >>>> >> >>> >> 1. Verify that the storage domain exists >> >> >>>> >> >>> >> 2. Check if it has 'ha_agents' directory >> >> >>>> >> >>> >> 3. Check if the links are OK, if not you can >safely >> >> >remove >> >> >>>> >the >> >> >>>> >> >links >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> 4. Next check the services are running: >> >> >>>> >> >>> >> A) sanlock >> >> >>>> >> >>> >> B) supervdsmd >> >> >>>> >> >>> >> C) vdsmd >> >> >>>> >> >>> >> D) libvirtd >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> 5. Increase the log level for broker and agent >> >services: >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> cd /etc/ovirt-hosted-engine-ha >> >> >>>> >> >>> >> vim *-log.conf >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> systemctl restart ovirt-ha-broker ovirt-ha-agent >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> 6. Check what they are complaining about >> >> >>>> >> >>> >> Keep in mind that agent will keep throwing errors >> >untill >> >> >the >> >> >>>> >> >broker >> >> >>>> >> >>> >stops >> >> >>>> >> >>> >> doing it (agent depends on broker), so broker >must >> >be >> >> >OK >> >> >>>> >before >> >> >>>> >> >>> >> peoceeding with the agent log. >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> About the manual VM start, you need 2 things: >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> 1. Define the VM network >> >> >>>> >> >>> >> # cat vdsm-ovirtmgmt.xml <network> >> >> >>>> >> >>> >> <name>vdsm-ovirtmgmt</name> >> >> >>>> >> >>> >> <uuid>8ded486e-e681-4754-af4b-5737c2b05405</uuid> >> >> >>>> >> >>> >> <forward mode='bridge'/> >> >> >>>> >> >>> >> <bridge name='ovirtmgmt'/> >> >> >>>> >> >>> >> </network> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> [root@ovirt1 HostedEngine-RECOVERY]# virsh define >> >> >>>> >> >vdsm-ovirtmgmt.xml >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> 2. Get an xml definition which can be found in the >> >vdsm >> >> >log. >> >> >>>> >> >Every VM >> >> >>>> >> >>> >at >> >> >>>> >> >>> >> start up has it's configuration printed out in >vdsm >> >log >> >> >on >> >> >>>> >the >> >> >>>> >> >host >> >> >>>> >> >>> >it >> >> >>>> >> >>> >> starts. >> >> >>>> >> >>> >> Save to file and then: >> >> >>>> >> >>> >> A) virsh define myvm.xml >> >> >>>> >> >>> >> B) virsh start myvm >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> It seems there is/was a problem with your NFS >shares. >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> Best Regards, >> >> >>>> >> >>> >> Strahil Nikolov >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> Hey Shareef, >> >> >>>> >> >>> >> >> >>>> >> >>> Check if there are any files or folders not owned by >> >> >vdsm:kvm . >> >> >>>> >> >Something >> >> >>>> >> >>> like this: >> >> >>>> >> >>> >> >> >>>> >> >>> find . -not -user 36 -not -group 36 -print >> >> >>>> >> >>> >> >> >>>> >> >>> Also check if vdsm can access the images in the >> >> >>>> >> >>> '<vol-mount-point>/images' directories. >> >> >>>> >> >>> >> >> >>>> >> >>> Best Regards, >> >> >>>> >> >>> Strahil Nikolov >> >> >>>> >> >>> >> >> >>>> >> >> >> >> >>>> >> >> >> >>>> >> And the IPv6 address '64:ff9b::c0a8:13d' ? >> >> >>>> >> >> >> >>>> >> I don't see in the log output. >> >> >>>> >> >> >> >>>> >> Best Regards, >> >> >>>> >> Strahil Nikolov >> >> >>>> >> >> >> >>>> >> >> >>>> Based on your output , you got a PTR record for IPv4 & >IPv6 >> >... >> >> >most >> >> >>>> probably it's the reason. >> >> >>>> >> >> >>>> Set the IPv6 on the interface and try again. >> >> >>>> >> >> >>>> Best Regards, >> >> >>>> Strahil Nikolov >> >> >>>> >> >> >>> >> >> >> >> Do you have firewalld up and running on the host ? >> >> >> >> Best Regards, >> >> Strahil Nikolov >> >> >> >> I am guessing, but your interface is not asaigned to any zone , >right? >> Just add the interface to the default zone (usually 'public'). >> >> Best Regards, >> Strahil Nikolov >> Keep in mind that there are a lot of playbooks that can be used to deploy a HostedEngine Environment via ansible. Keep in mind that if you plan to use oVirt in Prod, you need to know how to debug it (at least on basic level). Best Regards, Strahil Nikolov