[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

Hmmm, we're not using ipv6. Is that the issue? On Tue, Apr 14, 2020 at 3:56 PM Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; wrote: > On April 14, 2020 1:27:24 PM GMT+03:00, Shareef Jalloq < > shareef(a)jalloq.co.uk&gt; wrote: > >Right, I've given up on recovering the HE so want to try and redeploy > >it. > >There doesn't seem to be enough information to debug why the > >broker/agent > >won't start cleanly. > > > >In running 'hosted-engine --deploy', I'm seeing the following error in > >the > >setup validation phase: > > > >2020-04-14 09:46:08,922+0000 DEBUG otopi.plugins.otopi.dialog.human > >dialog.__logString:204 DIALOG:SEND Please provide the > >hostname of this host on the management network > >[ovirt-node-00.phoelex.com]: > > > > > >2020-04-14 09:46:12,831+0000 DEBUG > >otopi.plugins.gr_he_common.network.bridge > >hostname.getResolvedAddresses:432 > >getResolvedAddresses: set(['64:ff9b::c0a8:13d', '192.168.1.61']) > > > >2020-04-14 09:46:12,832+0000 DEBUG > >otopi.plugins.gr_he_common.network.bridge > >hostname._validateFQDNresolvability:289 ovirt-node-00.phoelex.com > >resolves > >to: set(['64:ff9b::c0a8:13d', '192.168.1.61']) > > > >2020-04-14 09:46:12,832+0000 DEBUG > >otopi.plugins.gr_he_common.network.bridge plugin.executeRaw:813 > >execute: > >['/usr/bin/dig', '+noall', '+answer', 'ovirt-node-00.phoelex.com', > >'ANY'], > >executable='None', cwd='None', env=None > > > >2020-04-14 09:46:12,871+0000 DEBUG > >otopi.plugins.gr_he_common.network.bridge plugin.executeRaw:863 > >execute-result: ['/usr/bin/dig', '+noall', '+answer', ' > >ovirt-node-00.phoelex.com', 'ANY'], rc=0 > > > >2020-04-14 09:46:12,872+0000 DEBUG > >otopi.plugins.gr_he_common.network.bridge plugin.execute:921 > >execute-output: ['/usr/bin/dig', '+noall', '+answer', ' > >ovirt-node-00.phoelex.com', 'ANY'] stdout: > > > >ovirt-node-00.phoelex.com. 86400 IN A 192.168.1.61 > > > > > >2020-04-14 09:46:12,872+0000 DEBUG > >otopi.plugins.gr_he_common.network.bridge plugin.execute:926 > >execute-output: ['/usr/bin/dig', '+noall', '+answer', ' > >ovirt-node-00.phoelex.com', 'ANY'] stderr: > > > > > > > >2020-04-14 09:46:12,872+0000 DEBUG > >otopi.plugins.gr_he_common.network.bridge plugin.executeRaw:813 > >execute: > >('/usr/sbin/ip', 'addr'), executable='None', cwd='None', env=None > > > >2020-04-14 09:46:12,876+0000 DEBUG > >otopi.plugins.gr_he_common.network.bridge plugin.executeRaw:863 > >execute-result: ('/usr/sbin/ip', 'addr'), rc=0 > > > >2020-04-14 09:46:12,876+0000 DEBUG > >otopi.plugins.gr_he_common.network.bridge plugin.execute:921 > >execute-output: ('/usr/sbin/ip', 'addr') stdout: > > > >1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN > >group > >default qlen 1000 > > > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > > > inet 127.0.0.1/8 scope host lo > > > > valid_lft forever preferred_lft forever > > > > inet6 ::1/128 scope host > > > > valid_lft forever preferred_lft forever > > > >2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master > >ovirtmgmt state UP group default qlen 1000 > > > > link/ether ac:1f:6b:bc:32:6a brd ff:ff:ff:ff:ff:ff > > > >3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state > >DOWN > >group default qlen 1000 > > > > link/ether ac:1f:6b:bc:32:6b brd ff:ff:ff:ff:ff:ff > > > >4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN > >group > >default qlen 1000 > > > > link/ether 02:e6:e2:80:93:8d brd ff:ff:ff:ff:ff:ff > > > >5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group > >default qlen 1000 > > > > link/ether 8a:26:44:50:ee:4a brd ff:ff:ff:ff:ff:ff > > > >21: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue > >state UP group default qlen 1000 > > > > link/ether ac:1f:6b:bc:32:6a brd ff:ff:ff:ff:ff:ff > > > > inet 192.168.1.61/24 brd 192.168.1.255 scope global ovirtmgmt > > > > valid_lft forever preferred_lft forever > > > > inet6 fe80::ae1f:6bff:febc:326a/64 scope link > > > > valid_lft forever preferred_lft forever > > > >22: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN > >group > >default qlen 1000 > > > > link/ether 3a:02:7b:7d:b3:2a brd ff:ff:ff:ff:ff:ff > > > > > >2020-04-14 09:46:12,876+0000 DEBUG > >otopi.plugins.gr_he_common.network.bridge plugin.execute:926 > >execute-output: ('/usr/sbin/ip', 'addr') stderr: > > > > > > > >2020-04-14 09:46:12,877+0000 DEBUG > >otopi.plugins.gr_he_common.network.bridge > >hostname.getLocalAddresses:251 > >addresses: [u'192.168.1.61', u'fe80::ae1f:6bff:febc:326a'] > > > >2020-04-14 09:46:12,877+0000 DEBUG > >otopi.plugins.gr_he_common.network.bridge hostname.test_hostname:464 > >test_hostname exception > > > >Traceback (most recent call last): > > > >File "/usr/lib/python2.7/site-packages/ovirt_setup_lib/hostname.py", > >line > >460, in test_hostname > > > > not_local_text, > > > >File "/usr/lib/python2.7/site-packages/ovirt_setup_lib/hostname.py", > >line > >342, in _validateFQDNresolvability > > > > addresses=resolvedAddressesAsString > > > >RuntimeError: ovirt-node-00.phoelex.com resolves to 64:ff9b::c0a8:13d > >192.168.1.61 and not all of them can be mapped to non loopback devices > >on > >this host > > > >2020-04-14 09:46:12,884+0000 ERROR > >otopi.plugins.gr_he_common.network.bridge dialog.queryEnvKey:120 Host > >name > >is not valid: ovirt-node-00.phoelex.com resolves to 64:ff9b::c0a8:13d > >192.168.1.61 and not all of them can be mapped to non loopback devices > >on > >this host > > > >The node I'm running on has an IP address of .61 and resolves > >correctly. > > > >On Fri, Apr 10, 2020 at 12:55 PM Shareef Jalloq <shareef(a)jalloq.co.uk&gt; > >wrote: > > > >> Where should I be checking if there are any files/folder not owned by > >> vdsm:kvm? I checked on the mount the HA sits on and it's fine. > >> > >> How would I go about checking vdsm can access those images? If I run > >> virsh, it lists them and they were running yesterday even though the > >HA was > >> down. I've since restarted both hosts but the broker is still > >spitting out > >> the same error (copied below). How do I find the reason the broker > >can't > >> connect to the storage? The conf file is already at DEBUG verbosity: > >> > >> [handler_logfile] > >> > >> class=logging.handlers.TimedRotatingFileHandler > >> > >> args=('/var/log/ovirt-hosted-engine-ha/broker.log', 'd', 1, 7) > >> > >> level=DEBUG > >> > >> formatter=long > >> > >> And what are all these .prob-<num> files that are being created? > >There > >> are over 250K of them now on the mount I'm using for the Data domain. > >> They're all of 0 size and of the form, > >> /rhev/data-center/mnt/nas-01.phoelex.com: > >> _volume2_vmstore/.prob-ffa867da-93db-4211-82df-b1b04a625ab9 > >> > >> @eevans: The volume I have the Data Domain on has TB's free. The HA > >is > >> dead so I can't ssh in. No idea what started these errors and the > >other > >> VMs were still running happily although they're on a different Data > >Domain. > >> > >> Shareef. > >> > >> MainThread::INFO::2020-04-10 > >> > > >07:45:00,408::storage_backends::373::ovirt_hosted_engine_ha.lib.storage_backends::(connect) > >> Connecting the storage > >> > >> MainThread::INFO::2020-04-10 > >> > > >07:45:00,408::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > >> Connecting storage server > >> > >> MainThread::INFO::2020-04-10 > >> > > >07:45:01,577::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > >> Connecting storage server > >> > >> MainThread::INFO::2020-04-10 > >> > > >07:45:02,692::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > >> Refreshing the storage domain > >> > >> MainThread::WARNING::2020-04-10 > >> > > >07:45:05,175::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) > >> Can't connect vdsm storage: Command StorageDomain.getInfo with args > >> {'storagedomainID': 'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} failed: > >> > >> (code=350, message=Error in storage domain action: > >> (u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) > >> > >> On Thu, Apr 9, 2020 at 5:58 PM Strahil Nikolov > &gt;&lt;hunter86_bg(a)yahoo.com&gt; > >> wrote: > >> > >>> On April 9, 2020 11:12:30 AM GMT+03:00, Shareef Jalloq < > >>> shareef(a)jalloq.co.uk&gt; wrote: > >>> >OK, let's go through this. I'm looking at the node that at least > >still > >>> >has > >>> >some VMs running. virsh also tells me that the HostedEngine VM is > >>> >running > >>> >but it's unresponsive and I can't shut it down. > >>> > > >>> >1. All storage domains exist and are mounted. > >>> >2. The ha_agent exists: > >>> > > >>> >[root@ovirt-node-01 ovirt-hosted-engine-ha]# ls > >/rhev/data-center/mnt/ > >>> >nas-01.phoelex.com > >>> \:_volume2_vmstore/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/ > >>> > > >>> >dom_md ha_agent images master > >>> > > >>> >3. There are two links > >>> > > >>> >[root@ovirt-node-01 ovirt-hosted-engine-ha]# ll > >/rhev/data-center/mnt/ > >>> >nas-01.phoelex.com > >>> >\:_volume2_vmstore/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/ha_agent/ > >>> > > >>> >total 8 > >>> > > >>> >lrwxrwxrwx. 1 vdsm kvm 132 Apr 2 14:50 hosted-engine.lockspace -> > >>> > >>> > > >>/var/run/vdsm/storage/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/ffb90b82-42fe-4253-85d5-aaec8c280aaf/90e68791-0c6f-406a-89ac-e0d86c631604 > >>> > > >>> >lrwxrwxrwx. 1 vdsm kvm 132 Apr 2 14:50 hosted-engine.metadata -> > >>> > >>> > > >>/var/run/vdsm/storage/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/2161aed0-7250-4c1d-b667-ac94f60af17e/6b818e33-f80a-48cc-a59c-bba641e027d4 > >>> > > >>> >4. The services exist but all seem to have some sort of warning: > >>> > > >>> >a) Apr 08 18:10:55 ovirt-node-01.phoelex.com sanlock[1728]: > >*2020-04-08 > >>> >18:10:55 1744152 [36796]: s16 delta_renew long write time 10 sec* > >>> > > >>> >b) Mar 23 18:02:59 ovirt-node-01.phoelex.com supervdsmd[29409]: > >*failed > >>> >to > >>> >load module nvdimm: libbd_nvdimm.so.2: cannot open shared object > >file: > >>> >No > >>> >such file or directory* > >>> > > >>> >c) Apr 09 08:05:13 ovirt-node-01.phoelex.com vdsm[4801]: *ERROR > >failed > >>> >to > >>> >retrieve Hosted Engine HA score '[Errno 2] No such file or > >directory'Is > >>> >the > >>> >Hosted Engine setup finished?* > >>> > > >>> >d)Apr 08 22:48:27 ovirt-node-01.phoelex.com libvirtd[29307]: > >2020-04-08 > >>> >22:48:27.134+0000: 29309: warning : qemuGetProcessInfo:1404 : > >cannot > >>> >parse > >>> >process status data > >>> > > >>> >Apr 08 22:48:27 ovirt-node-01.phoelex.com libvirtd[29307]: > >2020-04-08 > >>> >22:48:27.134+0000: 29309: error : virNetDevTapInterfaceStats:764 : > >>> >internal > >>> >error: /proc/net/dev: Interface not found > >>> > > >>> >Apr 08 23:09:39 ovirt-node-01.phoelex.com libvirtd[29307]: > >2020-04-08 > >>> >23:09:39.844+0000: 29307: error : virNetSocketReadWire:1806 : End > >of > >>> >file > >>> >while reading data: Input/output error > >>> > > >>> >Apr 09 01:05:26 ovirt-node-01.phoelex.com libvirtd[29307]: > >2020-04-09 > >>> >01:05:26.660+0000: 29307: error : virNetSocketReadWire:1806 : End > >of > >>> >file > >>> >while reading data: Input/output error > >>> > > >>> >5 & 6. The broker log is continually printing this error: > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,438::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) > >>> >ovirt-hosted-engine-ha broker 2.3.6 started > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> > >>> > > >>08:07:31,438::broker::55::ovirt_hosted_engine_ha.broker.broker.Broker::(run) > >>> >Running broker > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> > >>> > > >>08:07:31,438::broker::120::ovirt_hosted_engine_ha.broker.broker.Broker::(_get_monitor) > >>> >Starting monitor > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,438::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Searching for submonitors in > >>> >/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker > >>> > > >>> >/submonitors > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,439::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor network > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,440::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor cpu-load-no-engine > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor mgmt-bridge > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor network > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor cpu-load > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor engine-health > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,442::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor mgmt-bridge > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,442::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor cpu-load-no-engine > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor cpu-load > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor mem-free > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor storage-domain > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor storage-domain > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor mem-free > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,444::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Loaded submonitor engine-health > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,444::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >Finished loading submonitors > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> > >>> > > >>08:07:31,444::broker::128::ovirt_hosted_engine_ha.broker.broker.Broker::(_get_storage_broker) > >>> >Starting storage broker > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> > >>> > > >>08:07:31,444::storage_backends::369::ovirt_hosted_engine_ha.lib.storage_backends::(connect) > >>> >Connecting to VDSM > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> > >>> > > >>08:07:31,444::util::384::ovirt_hosted_engine_ha.lib.storage_backends::(__log_debug) > >>> >Creating a new json-rpc connection to VDSM > >>> > > >>> >Client localhost:54321::DEBUG::2020-04-09 > >>> >08:07:31,453::concurrent::258::root::(run) START thread > ><Thread(Client > >>> >localhost:54321, started daemon 139992488138496)> (func=<bound > >method > >>> >Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor > >object at > >>> >0x7f528acabc90>>, args=(), kwargs={}) > >>> > > >>> >Client localhost:54321::DEBUG::2020-04-09 > >>> > >>> > > >>08:07:31,459::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) > >>> >Stomp connection established > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> >08:07:31,467::stompclient::294::jsonrpc.AsyncoreClient::(send) > >Sending > >>> >response > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,530::storage_backends::373::ovirt_hosted_engine_ha.lib.storage_backends::(connect) > >>> >Connecting the storage > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:31,531::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > >>> >Connecting storage server > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> >08:07:31,531::stompclient::294::jsonrpc.AsyncoreClient::(send) > >Sending > >>> >response > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> >08:07:31,534::stompclient::294::jsonrpc.AsyncoreClient::(send) > >Sending > >>> >response > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> > >>> > > >>08:07:32,199::storage_server::158::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(_validate_pre_connected_path) > >>> >Storage domain a6cea67d-dbfb-45cf-a775-b4d0d47b26f2 is not > >available > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:32,199::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > >>> >Connecting storage server > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> >08:07:32,199::stompclient::294::jsonrpc.AsyncoreClient::(send) > >Sending > >>> >response > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> > >>> > > >>08:07:32,814::storage_server::363::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > >>> >[{u'status': 0, u'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}] > >>> > > >>> >MainThread::INFO::2020-04-09 > >>> > >>> > > >>08:07:32,814::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > >>> >Refreshing the storage domain > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> >08:07:32,815::stompclient::294::jsonrpc.AsyncoreClient::(send) > >Sending > >>> >response > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> > >>> > > >>08:07:33,129::storage_server::420::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > >>> >Error refreshing storage domain: Command StorageDomain.getStats > >with > >>> >args > >>> >{'storagedomainID': 'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} failed: > >>> > > >>> >(code=350, message=Error in storage domain action: > >>> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> >08:07:33,130::stompclient::294::jsonrpc.AsyncoreClient::(send) > >Sending > >>> >response > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> > >>> > > >>08:07:33,795::storage_backends::208::ovirt_hosted_engine_ha.lib.storage_backends::(_get_sector_size) > >>> >Command StorageDomain.getInfo with args {'storagedomainID': > >>> >'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} failed: > >>> > > >>> >(code=350, message=Error in storage domain action: > >>> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) > >>> > > >>> >MainThread::WARNING::2020-04-09 > >>> > >>> > > >>08:07:33,795::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) > >>> >Can't connect vdsm storage: Command StorageDomain.getInfo with args > >>> >{'storagedomainID': 'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} failed: > >>> > > >>> >(code=350, message=Error in storage domain action: > >>> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) > >>> > > >>> > > >>> >The UUID it is moaning about is indeed the one that the HA sits on > >and > >>> >is > >>> >the one I listed the contents of in step 2 above. > >>> > > >>> > > >>> >So why can't it see this domain? > >>> > > >>> > > >>> >Thanks, Shareef. > >>> > > >>> >On Thu, Apr 9, 2020 at 6:12 AM Strahil Nikolov > &gt;&lt;hunter86_bg(a)yahoo.com&gt; > >>> >wrote: > >>> > > >>> >> On April 9, 2020 1:51:05 AM GMT+03:00, Shareef Jalloq < > >>> >> shareef(a)jalloq.co.uk&gt; wrote: > >>> >> >Don't know if this is useful or not, but I just tried to > >shutdown > >>> >and > >>> >> >start > >>> >> >another VM on one of the hosts and get the following error: > >>> >> > > >>> >> >virsh # start scratch > >>> >> > > >>> >> >error: Failed to start domain scratch > >>> >> > > >>> >> >error: Network not found: no network with matching name > >>> >> >'vdsm-ovirtmgmt' > >>> >> > > >>> >> >Is this not referring to the interface name as the network is > >called > >>> >> >'ovirtmgnt'. > >>> >> > > >>> >> >On Wed, Apr 8, 2020 at 11:35 PM Shareef Jalloq > >>> &gt;&lt;shareef(a)jalloq.co.uk&gt; > >>> >> >wrote: > >>> >> > > >>> >> >> Hmmm, virsh tells me the HE is running but it hasn't come up > >and > >>> >the > >>> >> >> agent.log is full of the same errors. > >>> >> >> > >>> >> >> On Wed, Apr 8, 2020 at 11:31 PM Shareef Jalloq > >>> &gt;&lt;shareef(a)jalloq.co.uk&gt; > >>> >> >> wrote: > >>> >> >> > >>> >> >>> Ah hah! Ok, so I've managed to start it using virsh on the > >>> >second > >>> >> >host > >>> >> >>> but my first host is still dead. > >>> >> >>> > >>> >> >>> First of all, what are these 56,317 .prob- files that get > >dumped > >>> >to > >>> >> >the > >>> >> >>> NFS mounts? > >>> >> >>> > >>> >> >>> Secondly, why doesn't the node mount the NFS directories at > >boot? > >>> >> >Is > >>> >> >>> that the issue with this particular node? > >>> >> >>> > >>> >> >>> On Wed, Apr 8, 2020 at 11:12 PM <eevans(a)digitaldatatechs.com&gt; > >>> >wrote: > >>> >> >>> > >>> >> >>>> Did you try virsh list --inactive > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> Eric Evans > >>> >> >>>> > >>> >> >>>> Digital Data Services LLC. > >>> >> >>>> > >>> >> >>>> 304.660.9080 > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> *From:* Shareef Jalloq <shareef(a)jalloq.co.uk&gt; > >>> >> >>>> *Sent:* Wednesday, April 8, 2020 5:58 PM > >>> >> >>>> *To:* Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; > >>> >> >>>> *Cc:* Ovirt Users <users(a)ovirt.org&gt; > >>> >> >>>> *Subject:* [ovirt-users] Re: ovirt-engine unresponsive - how > >to > >>> >> >rescue? > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> I've now shut down the VMs on one host and rebooted it but > >the > >>> >> >agent > >>> >> >>>> service doesn't start. If I run 'hosted-engine --vm-status' > >I > >>> >get: > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> The hosted engine configuration has not been retrieved from > >>> >shared > >>> >> >>>> storage. Please ensure that ovirt-ha-agent is running and > >the > >>> >> >storage > >>> >> >>>> server is reachable. > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> and indeed if I list the mounts under /rhev/data-center/mnt, > >>> >only > >>> >> >one of > >>> >> >>>> the directories is mounted. I have 3 NFS mounts, one ISO > >Domain > >>> >> >and two > >>> >> >>>> Data Domains. Only one Data Domain has mounted and this has > >>> >lots > >>> >> >of .prob > >>> >> >>>> files in. So why haven't the other NFS exports been > >mounted? > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> Manually mounting them doesn't seem to have helped much > >either. > >>> >I > >>> >> >can > >>> >> >>>> start the broker service but the agent service says no. > >Same > >>> >error > >>> >> >as the > >>> >> >>>> one in my last email. > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> Shareef. > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> On Wed, Apr 8, 2020 at 9:57 PM Shareef Jalloq > >>> >> &gt;&lt;shareef(a)jalloq.co.uk&gt; > >>> >> >>>> wrote: > >>> >> >>>> > >>> >> >>>> Right, still down. I've run virsh and it doesn't know > >anything > >>> >> >about > >>> >> >>>> the engine vm. > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> I've restarted the broker and agent services and I still get > >>> >> >nothing in > >>> >> >>>> virsh->list. > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> In the logs under /var/log/ovirt-hosted-engine-ha I see lots > >of > >>> >> >errors: > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> broker.log: > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,138::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) > >>> >> >>>> ovirt-hosted-engine-ha broker 2.3.6 started > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,138::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Searching for submonitors in > >>> >> >>>> > >>> >> > >>> > >>> > > >>>/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,138::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor network > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,140::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor cpu-load-no-engine > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,140::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor mgmt-bridge > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor network > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor cpu-load > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor engine-health > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor mgmt-bridge > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,142::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor cpu-load-no-engine > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,142::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor cpu-load > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,142::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor mem-free > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor storage-domain > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor storage-domain > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor mem-free > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Loaded submonitor engine-health > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,143::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Finished loading submonitors > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,197::storage_backends::373::ovirt_hosted_engine_ha.lib.storage_backends::(connect) > >>> >> >>>> Connecting the storage > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,197::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > >>> >> >>>> Connecting storage server > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,414::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > >>> >> >>>> Connecting storage server > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:20,628::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > >>> >> >>>> Refreshing the storage domain > >>> >> >>>> > >>> >> >>>> MainThread::WARNING::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:21,057::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) > >>> >> >>>> Can't connect vdsm storage: Command StorageDomain.getInfo > >with > >>> >args > >>> >> >>>> {'storagedomainID': 'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} > >>> >failed: > >>> >> >>>> > >>> >> >>>> (code=350, message=Error in storage domain action: > >>> >> >>>> (u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:21,901::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) > >>> >> >>>> ovirt-hosted-engine-ha broker 2.3.6 started > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:56:21,901::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) > >>> >> >>>> Searching for submonitors in > >>> >> >>>> > >>> >> > >>> > >>> > > >>>/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> agent.log: > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> MainThread::ERROR::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:57:00,799::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) > >>> >> >>>> Trying to restart agent > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> > >>>20:57:00,799::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) > >>> >> >>>> Agent shutting down > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> > >>>20:57:11,144::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) > >>> >> >>>> ovirt-hosted-engine-ha agent 2.3.6 started > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:57:11,182::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) > >>> >> >>>> Found certificate common name: ovirt-node-01.phoelex.com > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:57:11,294::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) > >>> >> >>>> Initializing ha-broker connection > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:57:11,296::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > >>> >> >>>> Starting monitor network, options {'tcp_t_address': '', > >>> >> >'network_test': > >>> >> >>>> 'dns', 'tcp_t_port': '', 'addr': '192.168.1.99'} > >>> >> >>>> > >>> >> >>>> MainThread::ERROR::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:57:11,296::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) > >>> >> >>>> Failed to start necessary monitors > >>> >> >>>> > >>> >> >>>> MainThread::ERROR::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:57:11,297::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) > >>> >> >>>> Traceback (most recent call last): > >>> >> >>>> > >>> >> >>>> File > >>> >> >>>> > >>> >> > >>> > >>> > > >>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > >>> >> >>>> line 131, in _run_agent > >>> >> >>>> > >>> >> >>>> return action(he) > >>> >> >>>> > >>> >> >>>> File > >>> >> >>>> > >>> >> > >>> > >>> > > >>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > >>> >> >>>> line 55, in action_proper > >>> >> >>>> > >>> >> >>>> return he.start_monitoring() > >>> >> >>>> > >>> >> >>>> File > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > >>> >> >>>> line 432, in start_monitoring > >>> >> >>>> > >>> >> >>>> self._initialize_broker() > >>> >> >>>> > >>> >> >>>> File > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > >>> >> >>>> line 556, in _initialize_broker > >>> >> >>>> > >>> >> >>>> m.get('options', {})) > >>> >> >>>> > >>> >> >>>> File > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > >>> >> >>>> line 89, in start_monitor > >>> >> >>>> > >>> >> >>>> ).format(t=type, o=options, e=e) > >>> >> >>>> > >>> >> >>>> RequestError: brokerlink - failed to start monitor via > >>> >> >ovirt-ha-broker: > >>> >> >>>> [Errno 2] No such file or directory, [monitor: 'network', > >>> >options: > >>> >> >>>> {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': > >'', > >>> >> >'addr': > >>> >> >>>> '192.168.1.99'}] > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> MainThread::ERROR::2020-04-08 > >>> >> >>>> > >>> >> > >>> >> > >>> > >>> > > >>>20:57:11,297::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) > >>> >> >>>> Trying to restart agent > >>> >> >>>> > >>> >> >>>> MainThread::INFO::2020-04-08 > >>> >> >>>> > >>> >> > >>> > >>>20:57:11,297::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) > >>> >> >>>> Agent shutting down > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> On Wed, Apr 8, 2020 at 6:10 PM Strahil Nikolov > >>> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; > >>> >> >>>> wrote: > >>> >> >>>> > >>> >> >>>> On April 8, 2020 7:47:20 PM GMT+03:00, "Maton, Brett" < > >>> >> >>>> matonb(a)ltresources.co.uk&gt; wrote: > >>> >> >>>> >On the host you tried to restart the engine on: > >>> >> >>>> > > >>> >> >>>> >Add an alias to virsh (authenticates with virsh_auth.conf) > >>> >> >>>> > > >>> >> >>>> >alias virsh='virsh -c > >>> >> >>>> > >>> >>qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf' > >>> >> >>>> > > >>> >> >>>> >Then run virsh: > >>> >> >>>> > > >>> >> >>>> >virsh > >>> >> >>>> > > >>> >> >>>> >virsh # list > >>> >> >>>> > Id Name State > >>> >> >>>> >---------------------------------------------------- > >>> >> >>>> > xx HostedEngine Paused > >>> >> >>>> > xx ********** running > >>> >> >>>> > ... > >>> >> >>>> > xx ********** running > >>> >> >>>> > > >>> >> >>>> >HostedEngine should be in the list, try and resume the > >engine: > >>> >> >>>> > > >>> >> >>>> >virsh # resume HostedEngine > >>> >> >>>> > > >>> >> >>>> >On Wed, 8 Apr 2020 at 17:28, Shareef Jalloq > >>> &gt;&lt;shareef(a)jalloq.co.uk&gt; > >>> >> >>>> >wrote: > >>> >> >>>> > > >>> >> >>>> >> Thanks! > >>> >> >>>> >> > >>> >> >>>> >> The status hangs due to, I guess, the VM being down.... > >>> >> >>>> >> > >>> >> >>>> >> [root@ovirt-node-01 ~]# hosted-engine --vm-start > >>> >> >>>> >> VM exists and is down, cleaning up and restarting > >>> >> >>>> >> VM in WaitForLaunch > >>> >> >>>> >> > >>> >> >>>> >> but this doesn't seem to do anything. OK, after a while > >I > >>> >get a > >>> >> >>>> >status of > >>> >> >>>> >> it being barfed... > >>> >> >>>> >> > >>> >> >>>> >> --== Host ovirt-node-00.phoelex.com (id: 1) status ==-- > >>> >> >>>> >> > >>> >> >>>> >> conf_on_shared_storage : True > >>> >> >>>> >> Status up-to-date : False > >>> >> >>>> >> Hostname : > >>> >ovirt-node-00.phoelex.com > >>> >> >>>> >> Host ID : 1 > >>> >> >>>> >> Engine status : unknown stale-data > >>> >> >>>> >> Score : 3400 > >>> >> >>>> >> stopped : False > >>> >> >>>> >> Local maintenance : False > >>> >> >>>> >> crc32 : 9c4a034b > >>> >> >>>> >> local_conf_timestamp : 523362 > >>> >> >>>> >> Host timestamp : 523608 > >>> >> >>>> >> Extra metadata (valid at timestamp): > >>> >> >>>> >> metadata_parse_version=1 > >>> >> >>>> >> metadata_feature_version=1 > >>> >> >>>> >> timestamp=523608 (Wed Apr 8 16:17:11 2020) > >>> >> >>>> >> host-id=1 > >>> >> >>>> >> score=3400 > >>> >> >>>> >> vm_conf_refresh_time=523362 (Wed Apr 8 16:13:06 2020) > >>> >> >>>> >> conf_on_shared_storage=True > >>> >> >>>> >> maintenance=False > >>> >> >>>> >> state=EngineDown > >>> >> >>>> >> stopped=False > >>> >> >>>> >> > >>> >> >>>> >> > >>> >> >>>> >> --== Host ovirt-node-01.phoelex.com (id: 2) status ==-- > >>> >> >>>> >> > >>> >> >>>> >> conf_on_shared_storage : True > >>> >> >>>> >> Status up-to-date : True > >>> >> >>>> >> Hostname : > >>> >ovirt-node-01.phoelex.com > >>> >> >>>> >> Host ID : 2 > >>> >> >>>> >> Engine status : {"reason": "bad vm > >>> >status", > >>> >> >>>> >"health": > >>> >> >>>> >> "bad", "vm": "down_unexpected", "detail": "Down"} > >>> >> >>>> >> Score : 0 > >>> >> >>>> >> stopped : False > >>> >> >>>> >> Local maintenance : False > >>> >> >>>> >> crc32 : 5045f2eb > >>> >> >>>> >> local_conf_timestamp : 1737037 > >>> >> >>>> >> Host timestamp : 1737283 > >>> >> >>>> >> Extra metadata (valid at timestamp): > >>> >> >>>> >> metadata_parse_version=1 > >>> >> >>>> >> metadata_feature_version=1 > >>> >> >>>> >> timestamp=1737283 (Wed Apr 8 16:16:17 2020) > >>> >> >>>> >> host-id=2 > >>> >> >>>> >> score=0 > >>> >> >>>> >> vm_conf_refresh_time=1737037 (Wed Apr 8 16:12:11 2020) > >>> >> >>>> >> conf_on_shared_storage=True > >>> >> >>>> >> maintenance=False > >>> >> >>>> >> state=EngineUnexpectedlyDown > >>> >> >>>> >> stopped=False > >>> >> >>>> >> > >>> >> >>>> >> On Wed, Apr 8, 2020 at 5:09 PM Maton, Brett > >>> >> >>>> &gt;&lt;matonb(a)ltresources.co.uk&gt; > >>> >> >>>> >> wrote: > >>> >> >>>> >> > >>> >> >>>> >>> First steps, on one of your hosts as root: > >>> >> >>>> >>> > >>> >> >>>> >>> To get information: > >>> >> >>>> >>> hosted-engine --vm-status > >>> >> >>>> >>> > >>> >> >>>> >>> To start the engine: > >>> >> >>>> >>> hosted-engine --vm-start > >>> >> >>>> >>> > >>> >> >>>> >>> > >>> >> >>>> >>> On Wed, 8 Apr 2020 at 17:00, Shareef Jalloq > >>> >> &gt;&lt;shareef(a)jalloq.co.uk&gt; > >>> >> >>>> >wrote: > >>> >> >>>> >>> > >>> >> >>>> >>>> So my engine has gone down and I can't ssh into it > >either. > >>> >If > >>> >> >I > >>> >> >>>> >try to > >>> >> >>>> >>>> log into the web-ui of the node it is running on, I get > >>> >> >redirected > >>> >> >>>> >because > >>> >> >>>> >>>> the node can't reach the engine. > >>> >> >>>> >>>> > >>> >> >>>> >>>> What are my next steps? > >>> >> >>>> >>>> > >>> >> >>>> >>>> Shareef. > >>> >> >>>> >>>> _______________________________________________ > >>> >> >>>> >>>> Users mailing list -- users(a)ovirt.org > >>> >> >>>> >>>> To unsubscribe send an email to users-leave(a)ovirt.org > >>> >> >>>> >>>> Privacy Statement: > >>> >https://www.ovirt.org/privacy-policy.html > >>> >> >>>> >>>> oVirt Code of Conduct: > >>> >> >>>> >>>> > >https://www.ovirt.org/community/about/community-guidelines/ > >>> >> >>>> >>>> List Archives: > >>> >> >>>> >>>> > >>> >> >>>> > > >>> >> >>>> > >>> >> > > >>> >> > >>> > > >>> > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/W7BP57OCIRS... > >>> >> >>>> >>>> > >>> >> >>>> >>> > >>> >> >>>> > >>> >> >>>> This has to be resolved: > >>> >> >>>> > >>> >> >>>> Engine status : unknown stale-data > >>> >> >>>> > >>> >> >>>> Run again 'hosted-engine --vm-status'. If it remains the > >same, > >>> >> >restart > >>> >> >>>> ovirt-ha-broker.service & ovirt-ha-agent.service > >>> >> >>>> > >>> >> >>>> Verify that the engine's storage is available. Then monitor > >the > >>> >> >broker > >>> >> >>>> & agent logs in /var/log/ovirt-hosted-engine-ha > >>> >> >>>> > >>> >> >>>> Best Regards, > >>> >> >>>> Strahil Nikolov > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> >>>> > >>> >> > >>> >> Hi Shareef, > >>> >> > >>> >> The flow of activation oVirt is more complex than a plain KVM. > >>> >> Mounting of the domains happen during the activation of the node > >( > >>> >the > >>> >> HostedEngine is activating everything needed). > >>> >> > >>> >> Focus on the HostedEngine VM. > >>> >> Is it running properly ? > >>> >> > >>> >> If not,try: > >>> >> 1. Verify that the storage domain exists > >>> >> 2. Check if it has 'ha_agents' directory > >>> >> 3. Check if the links are OK, if not you can safely remove the > >links > >>> >> > >>> >> 4. Next check the services are running: > >>> >> A) sanlock > >>> >> B) supervdsmd > >>> >> C) vdsmd > >>> >> D) libvirtd > >>> >> > >>> >> 5. Increase the log level for broker and agent services: > >>> >> > >>> >> cd /etc/ovirt-hosted-engine-ha > >>> >> vim *-log.conf > >>> >> > >>> >> systemctl restart ovirt-ha-broker ovirt-ha-agent > >>> >> > >>> >> 6. Check what they are complaining about > >>> >> Keep in mind that agent will keep throwing errors untill the > >broker > >>> >stops > >>> >> doing it (agent depends on broker), so broker must be OK before > >>> >> peoceeding with the agent log. > >>> >> > >>> >> About the manual VM start, you need 2 things: > >>> >> > >>> >> 1. Define the VM network > >>> >> # cat vdsm-ovirtmgmt.xml <network> > >>> >> <name>vdsm-ovirtmgmt</name> > >>> >> <uuid>8ded486e-e681-4754-af4b-5737c2b05405</uuid> > >>> >> <forward mode='bridge'/> > >>> >> <bridge name='ovirtmgmt'/> > >>> >> </network> > >>> >> > >>> >> [root@ovirt1 HostedEngine-RECOVERY]# virsh define > >vdsm-ovirtmgmt.xml > >>> >> > >>> >> 2. Get an xml definition which can be found in the vdsm log. > >Every VM > >>> >at > >>> >> start up has it's configuration printed out in vdsm log on the > >host > >>> >it > >>> >> starts. > >>> >> Save to file and then: > >>> >> A) virsh define myvm.xml > >>> >> B) virsh start myvm > >>> >> > >>> >> It seems there is/was a problem with your NFS shares. > >>> >> > >>> >> > >>> >> Best Regards, > >>> >> Strahil Nikolov > >>> >> > >>> > >>> Hey Shareef, > >>> > >>> Check if there are any files or folders not owned by vdsm:kvm . > >Something > >>> like this: > >>> > >>> find . -not -user 36 -not -group 36 -print > >>> > >>> Also check if vdsm can access the images in the > >>> '<vol-mount-point>/images' directories. > >>> > >>> Best Regards, > >>> Strahil Nikolov > >>> > >> > > And the IPv6 address '64:ff9b::c0a8:13d' ? > > I don't see in the log output. > > Best Regards, > Strahil Nikolov >