<div dir="ltr">Sure not a problem. For the first issue regarding agent and broker crashing. Again the hosted engine VM is up and running at this time, I have no idea why the logs are saying volume doesn't exist and why file /var/run/vdsm/storage/<wbr>248f46f0-d793-4581-9810-<wbr>c9d965e2f286/14a20941-1b84-<wbr>4b82-be8f-ace38d7c037a/<wbr>8582bdfc-ef54-47af-9f1e-<wbr>f5b7ec1f1cf8 does not exist when the file actually does exist in that path. <div><br></div><div>I assume this problem is most likely also related or causing my other problems when accessing hosted vm snapshot section of web gui as well. <br><div><br></div><div>vdsm log:</div><div><br></div><div><div>jsonrpc/0::ERROR::2018-01-14 09:48:09,302::task::875::stora<wbr>ge.TaskManager.Task::(_setErro<wbr>r) (Task='37eba553-9c13-4e69-90f7<wbr>-d0c987cc694c') Unexpected error</div><div>Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/task.py", line 882, in _run</div><div> return fn(*args, **kargs)</div><div> File "<string>", line 2, in prepareImage</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/common/api.py", line 48, in method</div><div> ret = func(*args, **kwargs)</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/hsm.py", line 3162, in prepareImage</div><div> raise se.VolumeDoesNotExist(leafUUID<wbr>)</div><div>VolumeDoesNotExist: Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b<wbr>7ec1f1cf8',)</div><div>jsonrpc/0::ERROR::2018-01-14 09:48:09,303::dispatcher::82::<wbr>storage.Dispatcher::(wrapper) FINISH prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b<wbr>7ec1f1cf8',)</div></div><div><br></div><div>agent log:</div><div><br></div><div><div>MainThread::ERROR::2018-01-14 09:49:26,546::agent::145::ovir<wbr>t_hosted_engine_ha.agent.agent<wbr>.Agent::(_run_agent) Trying to restart agent</div><div>MainThread::ERROR::2018-01-14 09:49:37,782::hosted_engine::5<wbr>38::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_broker) Failed to start necessary monitors</div><div>MainThread::ERROR::2018-01-14 09:49:37,783::agent::144::ovir<wbr>t_hosted_engine_ha.agent.agent<wbr>.Agent::(_run_agent) Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/<wbr>agent/agent.py", line 131, in _run_agent</div><div> return action(he)</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/<wbr>agent/agent.py", line 55, in action_proper</div><div> return he.start_monitoring()</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/<wbr>agent/hosted_engine.py", line 416, in start_monitoring</div><div> self._initialize_broker()</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/<wbr>agent/hosted_engine.py", line 535, in _initialize_broker</div><div> m.get('options', {}))</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/<wbr>lib/brokerlink.py", line 83, in start_monitor</div><div> .format(type, options, e))</div><div>RequestError: Failed to start monitor ping, options {'addr': '192.168.0.1'}: [Errno 2] No such file or directory</div><div><br></div><div>MainThread::ERROR::2018-01-14 09:49:37,783::agent::145::ovir<wbr>t_hosted_engine_ha.agent.agent<wbr>.Agent::(_run_agent) Trying to restart agent</div></div><div><br></div><div><br></div><div>broker log:</div><div><br></div><div><div>StatusStorageThread::ERROR::<wbr>2018-01-12 14:03:57,629::status_broker::<wbr>85::ovirt_hosted_engine_ha.<wbr>broker.status_broker.<wbr>StatusBroker.Update::(run) Failed to update state.</div><div>Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/broker/status_broker.py", line 81, in run</div><div> entry.data</div><div> File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/broker/storage_broker.py", line 212, in put_stats</div><div> .format(str(e)))</div><div>RequestError: failed to write metadata: [Errno 2] No such file or directory: '/var/run/vdsm/storage/<wbr>248f46f0-d793-4581-9810-<wbr>c9d965e2f286/14a20941-1b84-<wbr>4b82-be8f-ace38d7c037a/<wbr>8582bdfc-ef54-47af-9f1e-<wbr>f5b7ec1f1cf8'</div><div>StatusStorageThread::ERROR::<wbr>2018-01-12 14:03:57,629::storage_broker::<wbr>160::ovirt_hosted_engine_ha.<wbr>broker.storage_broker.<wbr>StorageBroker::(get_raw_stats) Failed to read metadata from /var/run/vdsm/storage/<wbr>248f46f0-d793-4581-9810-<wbr>c9d965e2f286/14a20941-1b84-<wbr>4b82-be8f-ace38d7c037a/<wbr>8582bdfc-ef54-47af-9f1e-<wbr>f5b7ec1f1cf8</div><div>Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/broker/storage_broker.py", line 151, in get_raw_stats</div><div> f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC)</div><div>OSError: [Errno 2] No such file or directory: '/var/run/vdsm/storage/<wbr>248f46f0-d793-4581-9810-<wbr>c9d965e2f286/14a20941-1b84-<wbr>4b82-be8f-ace38d7c037a/<wbr>8582bdfc-ef54-47af-9f1e-<wbr>f5b7ec1f1cf8'</div><div>StatusStorageThread::ERROR::<wbr>2018-01-12 14:03:57,630::status_broker::<wbr>92::ovirt_hosted_engine_ha.<wbr>broker.status_broker.<wbr>StatusBroker.Update::(run) Failed to read state.</div><div>Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/broker/status_broker.py", line 88, in run</div><div> self._storage_broker.get_raw_<wbr>stats()</div><div> File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/broker/storage_broker.py", line 162, in get_raw_stats</div><div> .format(str(e)))</div><div>RequestError: failed to read metadata: [Errno 2] No such file or directory: '/var/run/vdsm/storage/<wbr>248f46f0-d793-4581-9810-<wbr>c9d965e2f286/14a20941-1b84-<wbr>4b82-be8f-ace38d7c037a/<wbr>8582bdfc-ef54-47af-9f1e-<wbr>f5b7ec1f1cf8'</div></div><div><br></div><div>Syslog:</div><div><br></div><div><div style="font-size:16px">Jan 12 16:52:34 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b<wbr>7ec1f1cf8',)</div><div style="font-size:16px">Jan 12 16:52:34 cultivar0 python: detected unhandled Python exception in '/usr/share/ovirt-hosted-engin<wbr>e-ha/ovirt-ha-broker'</div><div style="font-size:16px">Jan 12 16:52:34 cultivar0 abrt-server: Not saving repeating crash in '/usr/share/ovirt-hosted-engin<wbr>e-ha/ovirt-ha-broker'</div><div style="font-size:16px">Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service: main process exited, code=exited, status=1/FAILURE</div><div style="font-size:16px">Jan 12 16:52:34 cultivar0 systemd: Unit ovirt-ha-broker.service entered failed state.</div><div style="font-size:16px">Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service failed.</div><div style="font-size:16px">Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service holdoff time over, scheduling restart.</div><div style="font-size:16px">Jan 12 16:52:34 cultivar0 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked.</div><div style="font-size:16px">Jan 12 16:52:34 cultivar0 systemd: Started oVirt Hosted Engine High Availability Communications Broker.</div><div style="font-size:16px">Jan 12 16:52:34 cultivar0 systemd: Starting oVirt Hosted Engine High Availability Communications Broker...</div><div style="font-size:16px">Jan 12 16:52:36 cultivar0 journal: vdsm storage.TaskManager.Task ERROR (Task='73141dec-9d8f-4164-9c4e<wbr>-67c43a102eff') Unexpected error#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012 File "<string>", line 2, in prepareImage#012 File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/common/api.py", line 48, in method#012 ret = func(*args, **kwargs)#012 File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/hsm.py", line 3162, in prepareImage#012 raise se.VolumeDoesNotExist(leafUUID<wbr>)#012VolumeDoesNotExist: Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b<wbr>7ec1f1cf8',)</div><div style="font-size:16px">Jan 12 16:52:36 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b<wbr>7ec1f1cf8',)</div><div style="font-size:16px">Jan 12 16:52:36 cultivar0 python: detected unhandled Python exception in '/usr/share/ovirt-hosted-engin<wbr>e-ha/ovirt-ha-broker'</div><div style="font-size:16px">Jan 12 16:52:36 cultivar0 abrt-server: Not saving repeating crash in '/usr/share/ovirt-hosted-engin<wbr>e-ha/ovirt-ha-broker'</div><div style="font-size:16px">Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service: main process exited, code=exited, status=1/FAILURE</div><div style="font-size:16px">Jan 12 16:52:36 cultivar0 systemd: Unit ovirt-ha-broker.service entered failed state.</div><div style="font-size:16px">Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service failed.</div><div style="font-size:16px"><br></div><div style="font-size:16px">Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service holdoff time over, scheduling restart.</div><div style="font-size:16px">Jan 12 16:52:36 cultivar0 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked.</div><div style="font-size:16px">Jan 12 16:52:36 cultivar0 systemd: Started oVirt Hosted Engine High Availability Communications Broker.</div><div style="font-size:16px">Jan 12 16:52:36 cultivar0 systemd: Starting oVirt Hosted Engine High Availability Communications Broker...</div><div style="font-size:16px">Jan 12 16:52:37 cultivar0 journal: vdsm storage.TaskManager.Task ERROR (Task='bc7af1e2-0ab2-4164-ae88<wbr>-d2bee03500f9') Unexpected error#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012 File "<string>", line 2, in prepareImage#012 File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/common/api.py", line 48, in method#012 ret = func(*args, **kwargs)#012 File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/hsm.py", line 3162, in prepareImage#012 raise se.VolumeDoesNotExist(leafUUID<wbr>)#012VolumeDoesNotExist: Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b<wbr>7ec1f1cf8',)</div><div style="font-size:16px">Jan 12 16:52:37 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b<wbr>7ec1f1cf8',)</div><div style="font-size:16px">Jan 12 16:52:37 cultivar0 python: detected unhandled Python exception in '/usr/share/ovirt-hosted-engin<wbr>e-ha/ovirt-ha-broker'</div><div style="font-size:16px">Jan 12 16:52:38 cultivar0 abrt-server: Not saving repeating crash in '/usr/share/ovirt-hosted-engin<wbr>e-ha/ovirt-ha-broker'</div><div style="font-size:16px">Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service: main process exited, code=exited, status=1/FAILURE</div><div style="font-size:16px">Jan 12 16:52:38 cultivar0 systemd: Unit ovirt-ha-broker.service entered failed state.</div><div style="font-size:16px">Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service failed.</div><div style="font-size:16px">Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service holdoff time over, scheduling restart.</div><div style="font-size:16px">Jan 12 16:52:38 cultivar0 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked.</div><div style="font-size:16px">Jan 12 16:52:38 cultivar0 systemd: start request repeated too quickly for ovirt-ha-broker.service</div><div style="font-size:16px">Jan 12 16:52:38 cultivar0 systemd: Failed to start oVirt Hosted Engine High Availability Communications Broker.</div><div style="font-size:16px">Jan 12 16:52:38 cultivar0 systemd: Unit ovirt-ha-broker.service entered failed state.</div><div style="font-size:16px">Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service failed.</div><div style="font-size:16px">Jan 12 16:52:40 cultivar0 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart.</div><div style="font-size:16px">Jan 12 16:52:40 cultivar0 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked.</div><div style="font-size:16px">Jan 12 16:52:40 cultivar0 systemd: Started oVirt Hosted Engine High Availability Communications Broker.</div><div style="font-size:16px">Jan 12 16:52:40 cultivar0 systemd: Starting oVirt Hosted Engine High Availability Communications Broker...</div><div style="font-size:16px">Jan 12 16:52:40 cultivar0 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent.</div><div style="font-size:16px">Jan 12 16:52:40 cultivar0 systemd: Starting oVirt Hosted Engine High Availability Monitoring Agent...</div><div style="font-size:16px">Jan 12 16:52:41 cultivar0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.h<wbr>osted_engine.HostedEngine ERROR Failed to start necessary monitors</div><div style="font-size:16px">Jan 12 16:52:41 cultivar0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.a<wbr>gent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/<wbr>agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/<wbr>agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/<wbr>agent/hosted_engine.py", line 416, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/<wbr>agent/hosted_engine.py", line 535, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/<wbr>lib/brokerlink.py", line 83, in start_monitor#012 .format(type, options, e))#012RequestError: Failed to start monitor ping, options {'addr': '192.168.0.1'}: [Errno 2] No such file or directory</div><div style="font-size:16px">Jan 12 16:52:41 cultivar0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.a<wbr>gent.Agent ERROR Trying to restart agent</div><div style="font-size:16px">Jan 12 16:52:42 cultivar0 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a</div><div style="font-size:16px">Jan 12 16:52:42 cultivar0 systemd: Unit ovirt-ha-agent.service entered failed state.</div><div style="font-size:16px">Jan 12 16:52:42 cultivar0 systemd: ovirt-ha-agent.service failed.</div></div><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jan 14, 2018 at 9:46 AM, Yedidyah Bar David <span dir="ltr"><<a href="mailto:didi@redhat.com" target="_blank">didi@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Sun, Jan 14, 2018 at 3:37 PM, Jayme <<a href="mailto:jaymef@gmail.com">jaymef@gmail.com</a>> wrote:<br>
> First, apologies for all the posts to this list lately, I've been having a<br>
> heck of a time after 4.2 upgrade and you've been helpful, I appreciate that.<br>
><br>
> Since 4.2 upgrade I'm experiencing a few problems that I'm trying to debug.<br>
><br>
> Current status is engine and all hosts are upgraded to 4.2, and cluster and<br>
> domain set to 4.2 compatibility. Hosted Engine VM is running and ui<br>
> accessible etc, all VMs on hosts are running but no HA service. Web UI is<br>
> giving a few errors when checking network and snapshots on the hosted engine<br>
> VM only, it doesn't give errors on any of the others VMs that I spot<br>
> checked.<br>
><br>
> 1. HA-agent and HA-broker are continually crashing on all three hosts over<br>
> and over every few seconds. I sent an email to users list with more details<br>
> on this problem but unfortunately haven't heard anything back yet. The<br>
> general error in the logs seems to be:<br>
> VolumeDoesNotExist(leafUUID)#<wbr>012VolumeDoesNotExist: Volume does not exist:<br>
> (u'8582bdfc-ef54-47af-9f1e-<wbr>f5b7ec1f1cf8',) -- What? Volume doesn't exist,<br>
> why not?<br>
<br>
</span>If agent/broker logs do not reveal this, the next step is usually checking<br>
vdsm logs and/or system logs. Can you please check/share these? Thanks.<br>
<span class=""><br>
><br>
> 2. Error when clicking "network interfaces" in the web gui for the hosted VM<br>
> engine.<br>
><br>
> 3. Similar to #2 above an error is given when clicking "snapshots" in the<br>
> web gui for the hosted engine VM.<br>
><br>
> The errors for #2 and #3 are generic "cannot read property 'a' of null".<br>
> I've read previous postings on ovirt-mailing list that suggest you can<br>
> install debug-info package to get a human readable error.. but this package<br>
> does not seem to be compatible with 4.2, it expects 4.1: Requires:<br>
> "ovirt-engine-webadmin-portal = 4.1.2.2-1.el7.centos" -- Perhaps this<br>
> package is no longer required? I do see some additional details in the<br>
> ui.log that I can post if helpful.<br>
><br>
> There is obviously something odd going on here with the hosted engine VM.<br>
> All three errors appear to related to a problem with it, although it is<br>
> indeed up and running. I'd really like to get HA broker and agent back up<br>
> and running, and fix these GUI errors related to hosted engine VM. All<br>
> three problems may be connected to one common issue?<br>
><br>
> Thanks in advance!<br>
><br>
><br>
><br>
</span>> ______________________________<wbr>_________________<br>
> Users mailing list<br>
> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
><br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
<br>
--<br>
Didi<br>
</font></span></blockquote></div><br></div>