[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

Right, I've given up on recovering the HE so want to try and redeploy it. There doesn't seem to be enough information to debug why the broker/agent won't start cleanly. In running 'hosted-engine --deploy', I'm seeing the following error in the setup validation phase: 2020-04-14 09:46:08,922+0000 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND Please provide the hostname of this host on the management network [ovirt-node-00.phoelex.com]: 2020-04-14 09:46:12,831+0000 DEBUG otopi.plugins.gr_he_common.network.bridge hostname.getResolvedAddresses:432 getResolvedAddresses: set(['64:ff9b::c0a8:13d', '192.168.1.61']) 2020-04-14 09:46:12,832+0000 DEBUG otopi.plugins.gr_he_common.network.bridge hostname._validateFQDNresolvability:289 ovirt-node-00.phoelex.com resolves to: set(['64:ff9b::c0a8:13d', '192.168.1.61']) 2020-04-14 09:46:12,832+0000 DEBUG otopi.plugins.gr_he_common.network.bridge plugin.executeRaw:813 execute: ['/usr/bin/dig', '+noall', '+answer', 'ovirt-node-00.phoelex.com', 'ANY'], executable='None', cwd='None', env=None 2020-04-14 09:46:12,871+0000 DEBUG otopi.plugins.gr_he_common.network.bridge plugin.executeRaw:863 execute-result: ['/usr/bin/dig', '+noall', '+answer', ' ovirt-node-00.phoelex.com', 'ANY'], rc=0 2020-04-14 09:46:12,872+0000 DEBUG otopi.plugins.gr_he_common.network.bridge plugin.execute:921 execute-output: ['/usr/bin/dig', '+noall', '+answer', ' ovirt-node-00.phoelex.com', 'ANY'] stdout: ovirt-node-00.phoelex.com. 86400 IN A 192.168.1.61 2020-04-14 09:46:12,872+0000 DEBUG otopi.plugins.gr_he_common.network.bridge plugin.execute:926 execute-output: ['/usr/bin/dig', '+noall', '+answer', ' ovirt-node-00.phoelex.com', 'ANY'] stderr: 2020-04-14 09:46:12,872+0000 DEBUG otopi.plugins.gr_he_common.network.bridge plugin.executeRaw:813 execute: ('/usr/sbin/ip', 'addr'), executable='None', cwd='None', env=None 2020-04-14 09:46:12,876+0000 DEBUG otopi.plugins.gr_he_common.network.bridge plugin.executeRaw:863 execute-result: ('/usr/sbin/ip', 'addr'), rc=0 2020-04-14 09:46:12,876+0000 DEBUG otopi.plugins.gr_he_common.network.bridge plugin.execute:921 execute-output: ('/usr/sbin/ip', 'addr') stdout: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovirtmgmt state UP group default qlen 1000 link/ether ac:1f:6b:bc:32:6a brd ff:ff:ff:ff:ff:ff 3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether ac:1f:6b:bc:32:6b brd ff:ff:ff:ff:ff:ff 4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 02:e6:e2:80:93:8d brd ff:ff:ff:ff:ff:ff 5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 8a:26:44:50:ee:4a brd ff:ff:ff:ff:ff:ff 21: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether ac:1f:6b:bc:32:6a brd ff:ff:ff:ff:ff:ff inet 192.168.1.61/24 brd 192.168.1.255 scope global ovirtmgmt valid_lft forever preferred_lft forever inet6 fe80::ae1f:6bff:febc:326a/64 scope link valid_lft forever preferred_lft forever 22: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 3a:02:7b:7d:b3:2a brd ff:ff:ff:ff:ff:ff 2020-04-14 09:46:12,876+0000 DEBUG otopi.plugins.gr_he_common.network.bridge plugin.execute:926 execute-output: ('/usr/sbin/ip', 'addr') stderr: 2020-04-14 09:46:12,877+0000 DEBUG otopi.plugins.gr_he_common.network.bridge hostname.getLocalAddresses:251 addresses: [u'192.168.1.61', u'fe80::ae1f:6bff:febc:326a'] 2020-04-14 09:46:12,877+0000 DEBUG otopi.plugins.gr_he_common.network.bridge hostname.test_hostname:464 test_hostname exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_setup_lib/hostname.py", line 460, in test_hostname not_local_text, File "/usr/lib/python2.7/site-packages/ovirt_setup_lib/hostname.py", line 342, in _validateFQDNresolvability addresses=resolvedAddressesAsString RuntimeError: ovirt-node-00.phoelex.com resolves to 64:ff9b::c0a8:13d 192.168.1.61 and not all of them can be mapped to non loopback devices on this host 2020-04-14 09:46:12,884+0000 ERROR otopi.plugins.gr_he_common.network.bridge dialog.queryEnvKey:120 Host name is not valid: ovirt-node-00.phoelex.com resolves to 64:ff9b::c0a8:13d 192.168.1.61 and not all of them can be mapped to non loopback devices on this host The node I'm running on has an IP address of .61 and resolves correctly. On Fri, Apr 10, 2020 at 12:55 PM Shareef Jalloq <shareef(a)jalloq.co.uk&gt; wrote: > Where should I be checking if there are any files/folder not owned by > vdsm:kvm? I checked on the mount the HA sits on and it's fine. > > How would I go about checking vdsm can access those images? If I run > virsh, it lists them and they were running yesterday even though the HA was > down. I've since restarted both hosts but the broker is still spitting out > the same error (copied below). How do I find the reason the broker can't > connect to the storage? The conf file is already at DEBUG verbosity: > > [handler_logfile] > > class=logging.handlers.TimedRotatingFileHandler > > args=('/var/log/ovirt-hosted-engine-ha/broker.log', 'd', 1, 7) > > level=DEBUG > > formatter=long > > And what are all these .prob-<num> files that are being created? There > are over 250K of them now on the mount I'm using for the Data domain. > They're all of 0 size and of the form, > /rhev/data-center/mnt/nas-01.phoelex.com: > _volume2_vmstore/.prob-ffa867da-93db-4211-82df-b1b04a625ab9 > > @eevans: The volume I have the Data Domain on has TB's free. The HA is > dead so I can't ssh in. No idea what started these errors and the other > VMs were still running happily although they're on a different Data Domain. > > Shareef. > > MainThread::INFO::2020-04-10 > 07:45:00,408::storage_backends::373::ovirt_hosted_engine_ha.lib.storage_backends::(connect) > Connecting the storage > > MainThread::INFO::2020-04-10 > 07:45:00,408::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > Connecting storage server > > MainThread::INFO::2020-04-10 > 07:45:01,577::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > Connecting storage server > > MainThread::INFO::2020-04-10 > 07:45:02,692::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > Refreshing the storage domain > > MainThread::WARNING::2020-04-10 > 07:45:05,175::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) > Can't connect vdsm storage: Command StorageDomain.getInfo with args > {'storagedomainID': 'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} failed: > > (code=350, message=Error in storage domain action: > (u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) > > On Thu, Apr 9, 2020 at 5:58 PM Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; > wrote: > >> On April 9, 2020 11:12:30 AM GMT+03:00, Shareef Jalloq < >> shareef(a)jalloq.co.uk&gt; wrote: >> >OK, let's go through this. I'm looking at the node that at least still >> >has >> >some VMs running. virsh also tells me that the HostedEngine VM is >> >running >> >but it's unresponsive and I can't shut it down. >> > >> >1. All storage domains exist and are mounted. >> >2. The ha_agent exists: >> > >> >[root@ovirt-node-01 ovirt-hosted-engine-ha]# ls /rhev/data-center/mnt/ >> >nas-01.phoelex.com >> \:_volume2_vmstore/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/ >> > >> >dom_md ha_agent images master >> > >> >3. There are two links >> > >> >[root@ovirt-node-01 ovirt-hosted-engine-ha]# ll /rhev/data-center/mnt/ >> >nas-01.phoelex.com >> >\:_volume2_vmstore/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/ha_agent/ >> > >> >total 8 >> > >> >lrwxrwxrwx. 1 vdsm kvm 132 Apr 2 14:50 hosted-engine.lockspace -> >> >> >/var/run/vdsm/storage/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/ffb90b82-42fe-4253-85d5-aaec8c280aaf/90e68791-0c6f-406a-89ac-e0d86c631604 >> > >> >lrwxrwxrwx. 1 vdsm kvm 132 Apr 2 14:50 hosted-engine.metadata -> >> >> >/var/run/vdsm/storage/a6cea67d-dbfb-45cf-a775-b4d0d47b26f2/2161aed0-7250-4c1d-b667-ac94f60af17e/6b818e33-f80a-48cc-a59c-bba641e027d4 >> > >> >4. The services exist but all seem to have some sort of warning: >> > >> >a) Apr 08 18:10:55 ovirt-node-01.phoelex.com sanlock[1728]: *2020-04-08 >> >18:10:55 1744152 [36796]: s16 delta_renew long write time 10 sec* >> > >> >b) Mar 23 18:02:59 ovirt-node-01.phoelex.com supervdsmd[29409]: *failed >> >to >> >load module nvdimm: libbd_nvdimm.so.2: cannot open shared object file: >> >No >> >such file or directory* >> > >> >c) Apr 09 08:05:13 ovirt-node-01.phoelex.com vdsm[4801]: *ERROR failed >> >to >> >retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is >> >the >> >Hosted Engine setup finished?* >> > >> >d)Apr 08 22:48:27 ovirt-node-01.phoelex.com libvirtd[29307]: 2020-04-08 >> >22:48:27.134+0000: 29309: warning : qemuGetProcessInfo:1404 : cannot >> >parse >> >process status data >> > >> >Apr 08 22:48:27 ovirt-node-01.phoelex.com libvirtd[29307]: 2020-04-08 >> >22:48:27.134+0000: 29309: error : virNetDevTapInterfaceStats:764 : >> >internal >> >error: /proc/net/dev: Interface not found >> > >> >Apr 08 23:09:39 ovirt-node-01.phoelex.com libvirtd[29307]: 2020-04-08 >> >23:09:39.844+0000: 29307: error : virNetSocketReadWire:1806 : End of >> >file >> >while reading data: Input/output error >> > >> >Apr 09 01:05:26 ovirt-node-01.phoelex.com libvirtd[29307]: 2020-04-09 >> >01:05:26.660+0000: 29307: error : virNetSocketReadWire:1806 : End of >> >file >> >while reading data: Input/output error >> > >> >5 & 6. The broker log is continually printing this error: >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,438::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >ovirt-hosted-engine-ha broker 2.3.6 started >> > >> >MainThread::DEBUG::2020-04-09 >> >> >08:07:31,438::broker::55::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >Running broker >> > >> >MainThread::DEBUG::2020-04-09 >> >> >08:07:31,438::broker::120::ovirt_hosted_engine_ha.broker.broker.Broker::(_get_monitor) >> >Starting monitor >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,438::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Searching for submonitors in >> >/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker >> > >> >/submonitors >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,439::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor network >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,440::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor cpu-load-no-engine >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor mgmt-bridge >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor network >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor cpu-load >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor engine-health >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,442::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor mgmt-bridge >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,442::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor cpu-load-no-engine >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor cpu-load >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor mem-free >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor storage-domain >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor storage-domain >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,443::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor mem-free >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,444::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Loaded submonitor engine-health >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,444::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >Finished loading submonitors >> > >> >MainThread::DEBUG::2020-04-09 >> >> >08:07:31,444::broker::128::ovirt_hosted_engine_ha.broker.broker.Broker::(_get_storage_broker) >> >Starting storage broker >> > >> >MainThread::DEBUG::2020-04-09 >> >> >08:07:31,444::storage_backends::369::ovirt_hosted_engine_ha.lib.storage_backends::(connect) >> >Connecting to VDSM >> > >> >MainThread::DEBUG::2020-04-09 >> >> >08:07:31,444::util::384::ovirt_hosted_engine_ha.lib.storage_backends::(__log_debug) >> >Creating a new json-rpc connection to VDSM >> > >> >Client localhost:54321::DEBUG::2020-04-09 >> >08:07:31,453::concurrent::258::root::(run) START thread <Thread(Client >> >localhost:54321, started daemon 139992488138496)> (func=<bound method >> >Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at >> >0x7f528acabc90>>, args=(), kwargs={}) >> > >> >Client localhost:54321::DEBUG::2020-04-09 >> >> >08:07:31,459::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) >> >Stomp connection established >> > >> >MainThread::DEBUG::2020-04-09 >> >08:07:31,467::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending >> >response >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,530::storage_backends::373::ovirt_hosted_engine_ha.lib.storage_backends::(connect) >> >Connecting the storage >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:31,531::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >Connecting storage server >> > >> >MainThread::DEBUG::2020-04-09 >> >08:07:31,531::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending >> >response >> > >> >MainThread::DEBUG::2020-04-09 >> >08:07:31,534::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending >> >response >> > >> >MainThread::DEBUG::2020-04-09 >> >> >08:07:32,199::storage_server::158::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(_validate_pre_connected_path) >> >Storage domain a6cea67d-dbfb-45cf-a775-b4d0d47b26f2 is not available >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:32,199::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >Connecting storage server >> > >> >MainThread::DEBUG::2020-04-09 >> >08:07:32,199::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending >> >response >> > >> >MainThread::DEBUG::2020-04-09 >> >> >08:07:32,814::storage_server::363::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >[{u'status': 0, u'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}] >> > >> >MainThread::INFO::2020-04-09 >> >> >08:07:32,814::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >Refreshing the storage domain >> > >> >MainThread::DEBUG::2020-04-09 >> >08:07:32,815::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending >> >response >> > >> >MainThread::DEBUG::2020-04-09 >> >> >08:07:33,129::storage_server::420::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >Error refreshing storage domain: Command StorageDomain.getStats with >> >args >> >{'storagedomainID': 'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} failed: >> > >> >(code=350, message=Error in storage domain action: >> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> > >> >MainThread::DEBUG::2020-04-09 >> >08:07:33,130::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending >> >response >> > >> >MainThread::DEBUG::2020-04-09 >> >> >08:07:33,795::storage_backends::208::ovirt_hosted_engine_ha.lib.storage_backends::(_get_sector_size) >> >Command StorageDomain.getInfo with args {'storagedomainID': >> >'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} failed: >> > >> >(code=350, message=Error in storage domain action: >> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> > >> >MainThread::WARNING::2020-04-09 >> >> >08:07:33,795::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) >> >Can't connect vdsm storage: Command StorageDomain.getInfo with args >> >{'storagedomainID': 'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} failed: >> > >> >(code=350, message=Error in storage domain action: >> >(u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> > >> > >> >The UUID it is moaning about is indeed the one that the HA sits on and >> >is >> >the one I listed the contents of in step 2 above. >> > >> > >> >So why can't it see this domain? >> > >> > >> >Thanks, Shareef. >> > >> >On Thu, Apr 9, 2020 at 6:12 AM Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; >> >wrote: >> > >> >> On April 9, 2020 1:51:05 AM GMT+03:00, Shareef Jalloq < >> >> shareef(a)jalloq.co.uk&gt; wrote: >> >> >Don't know if this is useful or not, but I just tried to shutdown >> >and >> >> >start >> >> >another VM on one of the hosts and get the following error: >> >> > >> >> >virsh # start scratch >> >> > >> >> >error: Failed to start domain scratch >> >> > >> >> >error: Network not found: no network with matching name >> >> >'vdsm-ovirtmgmt' >> >> > >> >> >Is this not referring to the interface name as the network is called >> >> >'ovirtmgnt'. >> >> > >> >> >On Wed, Apr 8, 2020 at 11:35 PM Shareef Jalloq >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >wrote: >> >> > >> >> >> Hmmm, virsh tells me the HE is running but it hasn't come up and >> >the >> >> >> agent.log is full of the same errors. >> >> >> >> >> >> On Wed, Apr 8, 2020 at 11:31 PM Shareef Jalloq >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >> wrote: >> >> >> >> >> >>> Ah hah! Ok, so I've managed to start it using virsh on the >> >second >> >> >host >> >> >>> but my first host is still dead. >> >> >>> >> >> >>> First of all, what are these 56,317 .prob- files that get dumped >> >to >> >> >the >> >> >>> NFS mounts? >> >> >>> >> >> >>> Secondly, why doesn't the node mount the NFS directories at boot? >> >> >Is >> >> >>> that the issue with this particular node? >> >> >>> >> >> >>> On Wed, Apr 8, 2020 at 11:12 PM <eevans(a)digitaldatatechs.com&gt; >> >wrote: >> >> >>> >> >> >>>> Did you try virsh list --inactive >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> Eric Evans >> >> >>>> >> >> >>>> Digital Data Services LLC. >> >> >>>> >> >> >>>> 304.660.9080 >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> *From:* Shareef Jalloq <shareef(a)jalloq.co.uk&gt; >> >> >>>> *Sent:* Wednesday, April 8, 2020 5:58 PM >> >> >>>> *To:* Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; >> >> >>>> *Cc:* Ovirt Users <users(a)ovirt.org&gt; >> >> >>>> *Subject:* [ovirt-users] Re: ovirt-engine unresponsive - how to >> >> >rescue? >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> I've now shut down the VMs on one host and rebooted it but the >> >> >agent >> >> >>>> service doesn't start. If I run 'hosted-engine --vm-status' I >> >get: >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> The hosted engine configuration has not been retrieved from >> >shared >> >> >>>> storage. Please ensure that ovirt-ha-agent is running and the >> >> >storage >> >> >>>> server is reachable. >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> and indeed if I list the mounts under /rhev/data-center/mnt, >> >only >> >> >one of >> >> >>>> the directories is mounted. I have 3 NFS mounts, one ISO Domain >> >> >and two >> >> >>>> Data Domains. Only one Data Domain has mounted and this has >> >lots >> >> >of .prob >> >> >>>> files in. So why haven't the other NFS exports been mounted? >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> Manually mounting them doesn't seem to have helped much either. >> >I >> >> >can >> >> >>>> start the broker service but the agent service says no. Same >> >error >> >> >as the >> >> >>>> one in my last email. >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> Shareef. >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> On Wed, Apr 8, 2020 at 9:57 PM Shareef Jalloq >> >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >>>> wrote: >> >> >>>> >> >> >>>> Right, still down. I've run virsh and it doesn't know anything >> >> >about >> >> >>>> the engine vm. >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> I've restarted the broker and agent services and I still get >> >> >nothing in >> >> >>>> virsh->list. >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> In the logs under /var/log/ovirt-hosted-engine-ha I see lots of >> >> >errors: >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> broker.log: >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,138::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >> >>>> ovirt-hosted-engine-ha broker 2.3.6 started >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,138::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Searching for submonitors in >> >> >>>> >> >> >> >> >>/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,138::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor network >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,140::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor cpu-load-no-engine >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,140::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor mgmt-bridge >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor network >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor cpu-load >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor engine-health >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,141::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor mgmt-bridge >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,142::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor cpu-load-no-engine >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,142::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor cpu-load >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,142::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor mem-free >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor storage-domain >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor storage-domain >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor mem-free >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,143::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Loaded submonitor engine-health >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,143::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Finished loading submonitors >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,197::storage_backends::373::ovirt_hosted_engine_ha.lib.storage_backends::(connect) >> >> >>>> Connecting the storage >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,197::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> Connecting storage server >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,414::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> Connecting storage server >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:20,628::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >> >> >>>> Refreshing the storage domain >> >> >>>> >> >> >>>> MainThread::WARNING::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:21,057::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) >> >> >>>> Can't connect vdsm storage: Command StorageDomain.getInfo with >> >args >> >> >>>> {'storagedomainID': 'a6cea67d-dbfb-45cf-a775-b4d0d47b26f2'} >> >failed: >> >> >>>> >> >> >>>> (code=350, message=Error in storage domain action: >> >> >>>> (u'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:21,901::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >> >>>> ovirt-hosted-engine-ha broker 2.3.6 started >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:56:21,901::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>>> Searching for submonitors in >> >> >>>> >> >> >> >> >>/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> agent.log: >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> MainThread::ERROR::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:57:00,799::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) >> >> >>>> Trying to restart agent >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >>20:57:00,799::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) >> >> >>>> Agent shutting down >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >>20:57:11,144::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) >> >> >>>> ovirt-hosted-engine-ha agent 2.3.6 started >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:57:11,182::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) >> >> >>>> Found certificate common name: ovirt-node-01.phoelex.com >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:57:11,294::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) >> >> >>>> Initializing ha-broker connection >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:57:11,296::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) >> >> >>>> Starting monitor network, options {'tcp_t_address': '', >> >> >'network_test': >> >> >>>> 'dns', 'tcp_t_port': '', 'addr': '192.168.1.99'} >> >> >>>> >> >> >>>> MainThread::ERROR::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:57:11,296::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) >> >> >>>> Failed to start necessary monitors >> >> >>>> >> >> >>>> MainThread::ERROR::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:57:11,297::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) >> >> >>>> Traceback (most recent call last): >> >> >>>> >> >> >>>> File >> >> >>>> >> >> >> >> >>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", >> >> >>>> line 131, in _run_agent >> >> >>>> >> >> >>>> return action(he) >> >> >>>> >> >> >>>> File >> >> >>>> >> >> >> >> >>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", >> >> >>>> line 55, in action_proper >> >> >>>> >> >> >>>> return he.start_monitoring() >> >> >>>> >> >> >>>> File >> >> >>>> >> >> >> >> >> >> >>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >> >> >>>> line 432, in start_monitoring >> >> >>>> >> >> >>>> self._initialize_broker() >> >> >>>> >> >> >>>> File >> >> >>>> >> >> >> >> >> >> >>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >> >> >>>> line 556, in _initialize_broker >> >> >>>> >> >> >>>> m.get('options', {})) >> >> >>>> >> >> >>>> File >> >> >>>> >> >> >> >> >> >> >>"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", >> >> >>>> line 89, in start_monitor >> >> >>>> >> >> >>>> ).format(t=type, o=options, e=e) >> >> >>>> >> >> >>>> RequestError: brokerlink - failed to start monitor via >> >> >ovirt-ha-broker: >> >> >>>> [Errno 2] No such file or directory, [monitor: 'network', >> >options: >> >> >>>> {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', >> >> >'addr': >> >> >>>> '192.168.1.99'}] >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> MainThread::ERROR::2020-04-08 >> >> >>>> >> >> >> >> >> >> >>20:57:11,297::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) >> >> >>>> Trying to restart agent >> >> >>>> >> >> >>>> MainThread::INFO::2020-04-08 >> >> >>>> >> >> >> >>20:57:11,297::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) >> >> >>>> Agent shutting down >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> On Wed, Apr 8, 2020 at 6:10 PM Strahil Nikolov >> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >>>> wrote: >> >> >>>> >> >> >>>> On April 8, 2020 7:47:20 PM GMT+03:00, "Maton, Brett" < >> >> >>>> matonb(a)ltresources.co.uk&gt; wrote: >> >> >>>> >On the host you tried to restart the engine on: >> >> >>>> > >> >> >>>> >Add an alias to virsh (authenticates with virsh_auth.conf) >> >> >>>> > >> >> >>>> >alias virsh='virsh -c >> >> >>>> >> >>qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf' >> >> >>>> > >> >> >>>> >Then run virsh: >> >> >>>> > >> >> >>>> >virsh >> >> >>>> > >> >> >>>> >virsh # list >> >> >>>> > Id Name State >> >> >>>> >---------------------------------------------------- >> >> >>>> > xx HostedEngine Paused >> >> >>>> > xx ********** running >> >> >>>> > ... >> >> >>>> > xx ********** running >> >> >>>> > >> >> >>>> >HostedEngine should be in the list, try and resume the engine: >> >> >>>> > >> >> >>>> >virsh # resume HostedEngine >> >> >>>> > >> >> >>>> >On Wed, 8 Apr 2020 at 17:28, Shareef Jalloq >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >>>> >wrote: >> >> >>>> > >> >> >>>> >> Thanks! >> >> >>>> >> >> >> >>>> >> The status hangs due to, I guess, the VM being down.... >> >> >>>> >> >> >> >>>> >> [root@ovirt-node-01 ~]# hosted-engine --vm-start >> >> >>>> >> VM exists and is down, cleaning up and restarting >> >> >>>> >> VM in WaitForLaunch >> >> >>>> >> >> >> >>>> >> but this doesn't seem to do anything. OK, after a while I >> >get a >> >> >>>> >status of >> >> >>>> >> it being barfed... >> >> >>>> >> >> >> >>>> >> --== Host ovirt-node-00.phoelex.com (id: 1) status ==-- >> >> >>>> >> >> >> >>>> >> conf_on_shared_storage : True >> >> >>>> >> Status up-to-date : False >> >> >>>> >> Hostname : >> >ovirt-node-00.phoelex.com >> >> >>>> >> Host ID : 1 >> >> >>>> >> Engine status : unknown stale-data >> >> >>>> >> Score : 3400 >> >> >>>> >> stopped : False >> >> >>>> >> Local maintenance : False >> >> >>>> >> crc32 : 9c4a034b >> >> >>>> >> local_conf_timestamp : 523362 >> >> >>>> >> Host timestamp : 523608 >> >> >>>> >> Extra metadata (valid at timestamp): >> >> >>>> >> metadata_parse_version=1 >> >> >>>> >> metadata_feature_version=1 >> >> >>>> >> timestamp=523608 (Wed Apr 8 16:17:11 2020) >> >> >>>> >> host-id=1 >> >> >>>> >> score=3400 >> >> >>>> >> vm_conf_refresh_time=523362 (Wed Apr 8 16:13:06 2020) >> >> >>>> >> conf_on_shared_storage=True >> >> >>>> >> maintenance=False >> >> >>>> >> state=EngineDown >> >> >>>> >> stopped=False >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> --== Host ovirt-node-01.phoelex.com (id: 2) status ==-- >> >> >>>> >> >> >> >>>> >> conf_on_shared_storage : True >> >> >>>> >> Status up-to-date : True >> >> >>>> >> Hostname : >> >ovirt-node-01.phoelex.com >> >> >>>> >> Host ID : 2 >> >> >>>> >> Engine status : {"reason": "bad vm >> >status", >> >> >>>> >"health": >> >> >>>> >> "bad", "vm": "down_unexpected", "detail": "Down"} >> >> >>>> >> Score : 0 >> >> >>>> >> stopped : False >> >> >>>> >> Local maintenance : False >> >> >>>> >> crc32 : 5045f2eb >> >> >>>> >> local_conf_timestamp : 1737037 >> >> >>>> >> Host timestamp : 1737283 >> >> >>>> >> Extra metadata (valid at timestamp): >> >> >>>> >> metadata_parse_version=1 >> >> >>>> >> metadata_feature_version=1 >> >> >>>> >> timestamp=1737283 (Wed Apr 8 16:16:17 2020) >> >> >>>> >> host-id=2 >> >> >>>> >> score=0 >> >> >>>> >> vm_conf_refresh_time=1737037 (Wed Apr 8 16:12:11 2020) >> >> >>>> >> conf_on_shared_storage=True >> >> >>>> >> maintenance=False >> >> >>>> >> state=EngineUnexpectedlyDown >> >> >>>> >> stopped=False >> >> >>>> >> >> >> >>>> >> On Wed, Apr 8, 2020 at 5:09 PM Maton, Brett >> >> >>>> &gt;&lt;matonb(a)ltresources.co.uk&gt; >> >> >>>> >> wrote: >> >> >>>> >> >> >> >>>> >>> First steps, on one of your hosts as root: >> >> >>>> >>> >> >> >>>> >>> To get information: >> >> >>>> >>> hosted-engine --vm-status >> >> >>>> >>> >> >> >>>> >>> To start the engine: >> >> >>>> >>> hosted-engine --vm-start >> >> >>>> >>> >> >> >>>> >>> >> >> >>>> >>> On Wed, 8 Apr 2020 at 17:00, Shareef Jalloq >> >> &gt;&lt;shareef(a)jalloq.co.uk&gt; >> >> >>>> >wrote: >> >> >>>> >>> >> >> >>>> >>>> So my engine has gone down and I can't ssh into it either. >> >If >> >> >I >> >> >>>> >try to >> >> >>>> >>>> log into the web-ui of the node it is running on, I get >> >> >redirected >> >> >>>> >because >> >> >>>> >>>> the node can't reach the engine. >> >> >>>> >>>> >> >> >>>> >>>> What are my next steps? >> >> >>>> >>>> >> >> >>>> >>>> Shareef. >> >> >>>> >>>> _______________________________________________ >> >> >>>> >>>> Users mailing list -- users(a)ovirt.org >> >> >>>> >>>> To unsubscribe send an email to users-leave(a)ovirt.org >> >> >>>> >>>> Privacy Statement: >> >https://www.ovirt.org/privacy-policy.html >> >> >>>> >>>> oVirt Code of Conduct: >> >> >>>> >>>> https://www.ovirt.org/community/about/community-guidelines/ >> >> >>>> >>>> List Archives: >> >> >>>> >>>> >> >> >>>> > >> >> >>>> >> >> > >> >> >> > >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/W7BP57OCIRS... >> >> >>>> >>>> >> >> >>>> >>> >> >> >>>> >> >> >>>> This has to be resolved: >> >> >>>> >> >> >>>> Engine status : unknown stale-data >> >> >>>> >> >> >>>> Run again 'hosted-engine --vm-status'. If it remains the same, >> >> >restart >> >> >>>> ovirt-ha-broker.service & ovirt-ha-agent.service >> >> >>>> >> >> >>>> Verify that the engine's storage is available. Then monitor the >> >> >broker >> >> >>>> & agent logs in /var/log/ovirt-hosted-engine-ha >> >> >>>> >> >> >>>> Best Regards, >> >> >>>> Strahil Nikolov >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >> >> Hi Shareef, >> >> >> >> The flow of activation oVirt is more complex than a plain KVM. >> >> Mounting of the domains happen during the activation of the node ( >> >the >> >> HostedEngine is activating everything needed). >> >> >> >> Focus on the HostedEngine VM. >> >> Is it running properly ? >> >> >> >> If not,try: >> >> 1. Verify that the storage domain exists >> >> 2. Check if it has 'ha_agents' directory >> >> 3. Check if the links are OK, if not you can safely remove the links >> >> >> >> 4. Next check the services are running: >> >> A) sanlock >> >> B) supervdsmd >> >> C) vdsmd >> >> D) libvirtd >> >> >> >> 5. Increase the log level for broker and agent services: >> >> >> >> cd /etc/ovirt-hosted-engine-ha >> >> vim *-log.conf >> >> >> >> systemctl restart ovirt-ha-broker ovirt-ha-agent >> >> >> >> 6. Check what they are complaining about >> >> Keep in mind that agent will keep throwing errors untill the broker >> >stops >> >> doing it (agent depends on broker), so broker must be OK before >> >> peoceeding with the agent log. >> >> >> >> About the manual VM start, you need 2 things: >> >> >> >> 1. Define the VM network >> >> # cat vdsm-ovirtmgmt.xml <network> >> >> <name>vdsm-ovirtmgmt</name> >> >> <uuid>8ded486e-e681-4754-af4b-5737c2b05405</uuid> >> >> <forward mode='bridge'/> >> >> <bridge name='ovirtmgmt'/> >> >> </network> >> >> >> >> [root@ovirt1 HostedEngine-RECOVERY]# virsh define vdsm-ovirtmgmt.xml >> >> >> >> 2. Get an xml definition which can be found in the vdsm log. Every VM >> >at >> >> start up has it's configuration printed out in vdsm log on the host >> >it >> >> starts. >> >> Save to file and then: >> >> A) virsh define myvm.xml >> >> B) virsh start myvm >> >> >> >> It seems there is/was a problem with your NFS shares. >> >> >> >> >> >> Best Regards, >> >> Strahil Nikolov >> >> >> >> Hey Shareef, >> >> Check if there are any files or folders not owned by vdsm:kvm . Something >> like this: >> >> find . -not -user 36 -not -group 36 -print >> >> Also check if vdsm can access the images in the >> '<vol-mount-point>/images' directories. >> >> Best Regards, >> Strahil Nikolov >> >