[ovirt-devel] Re: Recent he-basic-ansible-suite-4.2 failures

On Mon, Jul 2, 2018 at 9:47 AM, Eyal Edri <eedri(a)redhat.com&gt; wrote: OK. Dug a bit more into [6] below, and saw this: agent is stuck in the following loop: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4... MainThread::INFO::2018-06-28 20:06:37,547::brokerlink::77::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.200.1'} MainThread::ERROR::2018-06-28 20:06:37,548::hosted_engine::538::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2018-06-28 20:06:37,550::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 413, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 535, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 83, in start_monitor .format(type, options, e)) RequestError: Failed to start monitor ping, options {'addr': '192.168.200.1'}: [Errno 2] No such file or directory So it seems like it tries to connect to the ping monitor in the broker and fails. broker has: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4... MainThread::INFO::2018-06-28 20:06:28,027::storage_backends::345::ovirt_hosted_engine_ha.lib.storage_backends::(connect) Connecting the storage MainThread::INFO::2018-06-28 20:06:28,028::storage_server::255::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2018-06-28 20:06:28,093::storage_server::262::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server and that's it. That's the end of the log. So it seems like it's trying to connect to the storage before it initialized the ping monitor (and others), and waits "forever" (and OST timing out after 10 minutes). vdsm has: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4... 2018-06-28 20:06:36,793-0400 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList(options=None) from=internal, task_id=c8f60a38-d4c5-48ee-8806-a645bd2f4316 (api:46) 2018-06-28 20:06:36,793-0400 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=c8f60a38-d4c5-48ee-8806-a645bd2f4316 (api:52) 2018-06-28 20:06:36,793-0400 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:707) ... 2018-06-28 20:06:41,799-0400 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList(options=None) from=internal, task_id=23ab1933-e0eb-4dc2-8404-3cf016437ec0 (api:46) 2018-06-28 20:06:41,799-0400 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=23ab1933-e0eb-4dc2-8404-3cf016437ec0 (api:52) 2018-06-28 20:06:41,799-0400 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:707) etc. Could not find anything interesting in supervdsm.log. Adding Simone and Nir. Any idea? Whoever that has a clue, you might reply changing the subject, so it's easier to follow. Thanks. -- Didi