<div dir="ltr"><div><div><div><div>Looking at the logs I see that the issue occurred when we were in setupNetworks and in between we called getStats.<br><br></div>We called:<br><br>2017-02-26 05:43:44,655-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient] (org.ovirt.thread.pool-7-thread-4) [] Message sent: SEND<br>destination:jms.topic.vdsm_requests<br>content-length:370<br>reply-to:jms.topic.vdsm_responses<br><br><JsonRpcRequest id: "8eea12f1-f41a-4174-8032-03f34fa2e806", method: Host.setupNetworks, params: {networks={VLAN200_Network={vlan=200, netmask=255.255.255.0, ipv6autoconf=false, nic=eth0, bridged=false, ipaddr=192.0.3.1, dhcpv6=false, mtu=1500, switch=legacy}}, bondings={}, options={connectivityTimeout=120, connectivityCheck=true}}><br><br></div>in between we see:<br><br>2017-02-26 05:43:51,741-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (DefaultQuartzScheduler5) [63e30973] SEND<br>destination:jms.topic.vdsm_requests<br>reply-to:jms.topic.vdsm_responses<br>content-length:98<br><br>{"jsonrpc":"2.0","method":"Host.getStats","params":{},"id":"27ffe862-ddda-46a1-a59f-5c5e3478a436"}\00<br><br></div>and setupNetworks response arrived at:<br><br>2017-02-26 05:43:51,762-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [40fe71c] MESSAGE<br>content-length:106<br>destination:jms.topic.vdsm_responses<br>content-type:application/json<br>subscription:c1dfc84f-c309-4ea0-8f1c-92462aacbd8e<br><br>{"jsonrpc": "2.0", "id": "8eea12f1-f41a-4174-8032-03f34fa2e806", "result": {"message": "Done", "code": 0}}\00<br><br></div>Response to Host.getStats arrived only partially. I thought that we hold host level lock to stop host monitoring to run in parallel with setupNetworks.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Feb 26, 2017 at 2:23 PM, Nir Soffer <span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Sun, Feb 26, 2017 at 3:18 PM, Yaniv Kaul <span dir="ltr"><<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span><div dir="ltr" class="m_744294330789664470m_4658829584392345580gmail_msg"><div class="gmail_quote m_744294330789664470m_4658829584392345580gmail_msg"><div dir="ltr" class="m_744294330789664470m_4658829584392345580gmail_msg">On Sun, Feb 26, 2017 at 3:04 PM Shlomo Ben David <<a href="mailto:sbendavi@redhat.com" class="m_744294330789664470m_4658829584392345580gmail_msg" target="_blank">sbendavi@redhat.com</a>> wrote:<br class="m_744294330789664470m_4658829584392345580gmail_msg"></div><blockquote class="gmail_quote m_744294330789664470m_4658829584392345580gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_744294330789664470m_4658829584392345580gmail_msg"><div class="m_744294330789664470m_4658829584392345580gmail_msg"><span id="m_744294330789664470m_4658829584392345580m_3235290391022882350m_9180424802943089896gmail-docs-internal-guid-6358cb1a-7a7a-efd2-5bd8-cdaa85d9e907" class="m_744294330789664470m_4658829584392345580gmail_msg"><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" class="m_744294330789664470m_4658829584392345580gmail_msg"><span style="font-size:11pt;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="m_744294330789664470m_4658829584392345580gmail_msg">Hi,</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" class="m_744294330789664470m_4658829584392345580gmail_msg"><span style="font-size:11pt;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="m_744294330789664470m_4658829584392345580gmail_msg"><br class="m_744294330789664470m_4658829584392345580gmail_msg"></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" class="m_744294330789664470m_4658829584392345580gmail_msg"><span style="font-size:11pt;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="m_744294330789664470m_4658829584392345580gmail_msg">Test failed: [ test-repo_ovirt_experimental_m<wbr>aster ] </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" class="m_744294330789664470m_4658829584392345580gmail_msg"><span style="font-size:11pt;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="m_744294330789664470m_4658829584392345580gmail_msg">Link to Job: [1]</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" class="m_744294330789664470m_4658829584392345580gmail_msg"><span style="font-size:11pt;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="m_744294330789664470m_4658829584392345580gmail_msg">Link to all logs: [2]</span></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" class="m_744294330789664470m_4658829584392345580gmail_msg"><span style="font-size:11pt;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="m_744294330789664470m_4658829584392345580gmail_msg">Link to error log: [3]</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" class="m_744294330789664470m_4658829584392345580gmail_msg"><span style="font-size:11pt;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="m_744294330789664470m_4658829584392345580gmail_msg"><br class="m_744294330789664470m_4658829584392345580gmail_msg"></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" class="m_744294330789664470m_4658829584392345580gmail_msg"><span style="font-size:11pt;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="m_744294330789664470m_4658829584392345580gmail_msg">[1] </span><span style="color:rgb(0,0,0);font-family:arial;font-size:14.6667px;white-space:pre-wrap" class="m_744294330789664470m_4658829584392345580gmail_msg"><a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538" class="m_744294330789664470m_4658829584392345580gmail_msg" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_ma<wbr>ster/5538</a></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" class="m_744294330789664470m_4658829584392345580gmail_msg"><span style="font-size:11pt;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="m_744294330789664470m_4658829584392345580gmail_msg">[2] <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-006_migrations.py/" class="m_744294330789664470m_4658829584392345580gmail_msg" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_ma<wbr>ster/5538/artifact/exported-ar<wbr>tifacts/basic-suit-master-el7/<wbr>test_logs/basic-suite-master/<wbr>post-006_migrations.py/</a></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" class="m_744294330789664470m_4658829584392345580gmail_msg"><span style="font-size:11pt;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="m_744294330789664470m_4658829584392345580gmail_msg">[3] <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538/artifact/exported-artifacts/basic-suit-master-el7/nosetests-006_migrations.py.xml" class="m_744294330789664470m_4658829584392345580gmail_msg" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_ma<wbr>ster/5538/artifact/exported-ar<wbr>tifacts/basic-suit-master-el7/<wbr>nosetests-006_migrations.py.<wbr>xml</a></span></p><br class="m_744294330789664470m_4658829584392345580gmail_msg"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" class="m_744294330789664470m_4658829584392345580gmail_msg"><span style="font-size:11pt;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap" class="m_744294330789664470m_4658829584392345580gmail_msg">Error snippet from the log: </span></p></span></div></div></blockquote><div class="m_744294330789664470m_4658829584392345580gmail_msg"><br class="m_744294330789664470m_4658829584392345580gmail_msg"></div></div></div></span><div dir="ltr" class="m_744294330789664470m_4658829584392345580gmail_msg"><div class="gmail_quote m_744294330789664470m_4658829584392345580gmail_msg"><div class="m_744294330789664470m_4658829584392345580gmail_msg">The below is not the issue. The issue is (engine log):</div><div class="m_744294330789664470m_4658829584392345580gmail_msg">2017-02-26 05:43:52,178-05 ERROR [<a href="http://org.ovirt.engine.core.bll.net">org.ovirt.engine.core.bll.net</a><wbr>work.host.HostValidator] (default task-11) [d3b3a59d-6cc0-4896-b3f2-8483f<wbr>9b77fe2] Unable to setup network: operation can only be done when Host status is one of: Maintenance, Up, NonOperational; current status is Connecting</div><div class="m_744294330789664470m_4658829584392345580gmail_msg"><br class="m_744294330789664470m_4658829584392345580gmail_msg"></div><div class="m_744294330789664470m_4658829584392345580gmail_msg">And it comes from the host disconnecting a bit from engine, with various errors such as (again, engine log):</div><div class="m_744294330789664470m_4658829584392345580gmail_msg"><br class="m_744294330789664470m_4658829584392345580gmail_msg"></div><div class="m_744294330789664470m_4658829584392345580gmail_msg"><br class="m_744294330789664470m_4658829584392345580gmail_msg"></div><div class="m_744294330789664470m_4658829584392345580gmail_msg">2017-02-26 05:43:26,763-05 ERROR [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (DefaultQuartzScheduler5) [63e30973] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,<wbr>802), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM lago-basic-suite-master-host1 <b>command FullListVDS failed: Unrecognized message received</b> <br class="m_744294330789664470m_4658829584392345580gmail_msg"></div></div></div></div></blockquote><div><br></div></span><div>The error Shlomo posted may not be the error failing the test, but is is </div><div>very important error that must not be in vdsm logs.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Nir</div></font></span></div></div></div>
</blockquote></div><br></div>