<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">2017-12-21 5:13 GMT+01:00 Andy <span dir="ltr"><<a href="mailto:farkey_2000@yahoo.com" target="_blank">farkey_2000@yahoo.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:10px"><div>Hello all,</div><div><br></div><div>I just upgraded my OVIRT instance to 4.2, the engine completed successfully, however after I upgraded the hosts the HA Broker will not start. The 2 hosts are running CentOS 7.4, running gluster and CTDB. The VIPS are up and can be reached from both hosts as well as I can mount the gluster storage. </div><div><br></div><div>The error from the agent.log: </div><div><br></div><div>MainThread::INFO::2017-12-20 21:02:19,219::agent::67::<wbr>ovirt_hosted_engine_ha.agent.<wbr>agent.Agent::(run) ovirt-hosted-engine-ha agent 2.2.2 started<br>MainThread::INFO::2017-12-20 21:02:19,346::hosted_engine::<wbr>243::ovirt_hosted_engine_ha.<wbr>agent.hosted_engine.<wbr>HostedEngine::(_get_hostname) Found certificate common name: hm3svr01.hm3.loc<br>MainThread::INFO::2017-12-20 21:02:20,478::hosted_engine::<wbr>525::ovirt_hosted_engine_ha.<wbr>agent.hosted_engine.<wbr>HostedEngine::(_initialize_<wbr>broker) Initializing ha-broker connection<br>MainThread::INFO::2017-12-20 21:02:20,482::brokerlink::77::<wbr>ovirt_hosted_engine_ha.lib.<wbr>brokerlink.BrokerLink::(start_<wbr>monitor) Starting monitor ping, options {'addr': '192.168.3.1'}<br>MainThread::ERROR::2017-12-20 21:02:20,483::hosted_engine::<wbr>538::ovirt_hosted_engine_ha.<wbr>agent.hosted_engine.<wbr>HostedEngine::(_initialize_<wbr>broker) Failed to start necessary monitors<br>MainThread::ERROR::2017-12-20 21:02:20,485::agent::144::<wbr>ovirt_hosted_engine_ha.agent.<wbr>agent.Agent::(_run_agent) Traceback (most recent call last):<br> File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/agent/agent.py", line 131, in _run_agent<br> return action(he)<br> File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/agent/agent.py", line 55, in action_proper<br> return he.start_monitoring()<br> File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/agent/hosted_engine.py", line 416, in start_monitoring<br> self._initialize_broker()<br> File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/agent/hosted_engine.py", line 535, in _initialize_broker<br> m.get('options', {}))<br> File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/lib/brokerlink.py", line 83, in start_monitor<br> .format(type, options, e))<br>RequestError: Failed to start monitor ping, options {'addr': '192.168.x.x'}: [Errno 2] No such file or directory<br><div><br></div><div><br></div><div>The broker.log:</div><div><br></div><div>MainThread::INFO::2017-12-20 23:06:19,405::monitor::50::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Finished loading submonitors<br>MainThread::INFO::2017-12-20 23:06:20,324::storage_<wbr>backends::346::ovirt_hosted_<wbr>engine_ha.lib.storage_<wbr>backends::(connect) Connecting the storage<br>MainThread::INFO::2017-12-20 23:06:20,325::storage_server::<wbr>252::ovirt_hosted_engine_ha.<wbr>lib.storage_server.<wbr>StorageServer::(connect_<wbr>storage_server) Connecting storage server<br>MainThread::INFO::2017-12-20 23:06:20,849::storage_server::<wbr>259::ovirt_hosted_engine_ha.<wbr>lib.storage_server.<wbr>StorageServer::(connect_<wbr>storage_server) Connecting storage server<br>MainThread::WARNING::2017-12-<wbr>20 23:06:20,913::storage_broker::<wbr>96::ovirt_hosted_engine_ha.<wbr>broker.storage_broker.<wbr>StorageBroker::(__init__) Can't connect vdsm storage: Connection to storage server failed <br>MainThread::INFO::2017-12-20 23:06:22,087::broker::45::<wbr>ovirt_hosted_engine_ha.broker.<wbr>broker.Broker::(run) ovirt-hosted-engine-ha broker 2.2.2 started<br>MainThread::INFO::2017-12-20 23:06:22,088::monitor::40::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Searching for submonitors in /usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/broker/s<br>ubmonitors<br>MainThread::INFO::2017-12-20 23:06:22,089::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor cpu-load<br>MainThread::INFO::2017-12-20 23:06:22,093::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor cpu-load-no-engine<br>MainThread::INFO::2017-12-20 23:06:22,146::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor engine-health<br>MainThread::INFO::2017-12-20 23:06:22,147::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor mem-free<br>MainThread::INFO::2017-12-20 23:06:22,147::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor mem-load<br>MainThread::INFO::2017-12-20 23:06:22,148::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor mgmt-bridge<br>MainThread::INFO::2017-12-20 23:06:22,149::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor ping<br>MainThread::INFO::2017-12-20 23:06:22,149::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor storage-domain<br>MainThread::INFO::2017-12-20 23:06:22,150::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor cpu-load<br>MainThread::INFO::2017-12-20 23:06:22,151::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor cpu-load-no-engine<br>MainThread::INFO::2017-12-20 23:06:22,152::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor engine-health<br>MainThread::INFO::2017-12-20 23:06:22,153::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor mem-free<br>MainThread::INFO::2017-12-20 23:06:22,153::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor mem-load<br>MainThread::INFO::2017-12-20 23:06:22,154::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor mgmt-bridge<br>MainThread::INFO::2017-12-20 23:06:22,154::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor ping<br>MainThread::INFO::2017-12-20 23:06:22,155::monitor::49::<wbr>ovirt_hosted_engine_ha.broker.<wbr>monitor.Monitor::(_discover_<wbr>submonitors) Loaded submonitor storage-domain<br><br><div><br></div><div>The VDSM log has alot of JSON errors with the storage fai2017-12-20 23:13:00,311-0500 INFO (jsonrpc/6) [vdsm.api] FINISH getStorageDomainInfo error=Storage domain does not exist: (u'1cc6cc89-571e-4b6a-9d41-<wbr>c742d763e1cc',) from=::1,54630, task_id=ff009157-48f3-480c-<wbr>b8fe-b8d0a791c922 (api:50)<br>2017-12-20 23:13:00,312-0500 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='ff009157-48f3-480c-<wbr>b8fe-b8d0a791c922') Unexpected error (task:875)<br>2017-12-20 23:13:00,314-0500 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getStorageDomainInfo error=Storage domain does not exist: (u'1cc6cc89-571e-4b6a-9d41-<wbr>c742d763e1cc',) (dispatcher:82)<br>2017-12-20 23:13:00,314-0500 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getInfo failed (error 358) in 0.48 seconds (__init__:573)<br> raise convert_to_error(kind, result)<br>2017-12-20 23:13:03,092-0500 INFO (jsonrpc/3) [vdsm.api] FINISH getStorageDomainInfo error=Storage domain does not exist: (u'1cc6cc89-571e-4b6a-9d41-<wbr>c742d763e1cc',) from=::1,54632, task_id=39e022e5-db99-4bc4-<wbr>88e1-9a218104b3c7 (api:50)<br>2017-12-20 23:13:03,093-0500 ERROR (jsonrpc/3) [storage.TaskManager.Task] (Task='39e022e5-db99-4bc4-<wbr>88e1-9a218104b3c7') Unexpected error (task:875)<br>2017-12-20 23:13:03,095-0500 ERROR (jsonrpc/3) [storage.Dispatcher] FINISH getStorageDomainInfo error=Storage domain does not exist: (u'1cc6cc89-571e-4b6a-9d41-<wbr>c742d763e1cc',) (dispatcher:82)<br>2017-12-20 23:13:03,095-0500 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getInfo failed (error 358) in 0.49 seconds (__init__:573)<br> raise convert_to_error(kind, result)<br>2017-12-20 23:13:07,568-0500 INFO (jsonrpc/4) [vdsm.api] FINISH getStorageDomainInfo error=Storage domain does not exist: (u'1cc6cc89-571e-4b6a-9d41-<wbr>c742d763e1cc',) from=::1,54640, task_id=c1b1b1a1-a7e6-494a-<wbr>bda6-19c617820dec (api:50)<br>2017-12-20 23:13:07,569-0500 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='c1b1b1a1-a7e6-494a-<wbr>bda6-19c617820dec') Unexpected error (task:875)<br>2017-12-20 23:13:07,571-0500 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH getStorageDomainInfo error=Storage domain does not exist: (u'1cc6cc89-571e-4b6a-9d41-<wbr>c742d763e1cc',) (dispatcher:82)<br>2017-12-20 23:13:07,571-0500 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getInfo failed (error 358) in 0.48 seconds (__init__:573)<br> raise convert_to_error(kind, result)<br>2017-12-20 23:13:10,323-0500 INFO (jsonrpc/0) [vdsm.api] FINISH getStorageDomainInfo error=Storage domain does not exist: (u'1cc6cc89-571e-4b6a-9d41-<wbr>c742d763e1cc',) from=::1,54642, task_id=6354fa3d-933c-4fd0-<wbr>9301-00f8abd29ec7 (api:50)<br>2017-12-20 23:13:10,323-0500 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='6354fa3d-933c-4fd0-<wbr>9301-00f8abd29ec7') Unexpected error (task:875)<br>2017-12-20 23:13:10,325-0500 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH getStorageDomainInfo error=Storage domain does not exist: (u'1cc6cc89-571e-4b6a-9d41-<wbr>c742d763e1cc',) (dispatcher:82)<br>2017-12-20 23:13:10,326-0500 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getInfo failed (error 358) in 0.48 seconds (__init__:573)<br><br><div>ling</div><div><br></div><div><br></div><div>Any help is appreciated. </div><div><br></div><div>thanks Andy<br></div></div><div><br></div></div></div></div></div></blockquote><div><br></div><div>Adding relevant developers.</div><div>Andy, do you mind open a bug on <a href="https://bugzilla.redhat.com/enter_bug.cgi?product=ovirt-hosted-engine-ha">https://bugzilla.redhat.com/enter_bug.cgi?product=ovirt-hosted-engine-ha</a> to track this?</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:10px"><div><div><div></div><div><br></div> </div><div><br></div><div><br></div><br></div></div></div><br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>SANDRO</span> <span>BONAZZOLA</span></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D</span></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <span>EMEA</span></a></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><div><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></div></td></tr></tbody></table><br></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div></div>