<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 16 Mar 2017, at 15:18, Yaniv Kaul <<a href="mailto:ykaul@redhat.com" class="">ykaul@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><br class="Apple-interchange-newline"><br class=""><div class="gmail_quote">On Thu, Mar 16, 2017 at 10:46 AM, Nir Soffer<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:nsoffer@redhat.com" target="_blank" class="">nsoffer@redhat.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class="">If found this error in system tests - looks like wrong assert - code should check<div class="">if disk is not None before checking state.<br class=""><div class=""><br class=""></div><div class="">I'm not sure who is the owner of this test, so posting here.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Theoretically, perhaps. </div><div class="">Practically, it worked (until yesterday?) and now I'm also seeing this failure - it's not a coincidence.</div></div></div></div></div></blockquote><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">However, looking at my failure[1], I'm seeing other nasty stuff, which may explain the later on failures</div><div class=""><br class=""></div><div class="">For example, new NPE I have not seen in the past:</div><div class=""><div class="">2017-03-16 09:57:57,581-04 INFO [org.ovirt.engine.core.bll.AddVmTemplateCommand] (DefaultQuartzScheduler1) [5d94233] Ending command 'org.ovirt.engine.core.bll.AddVmTemplateCommand' successfully.</div></div></div></div></div></div></blockquote><div><br class=""></div>Is this IST or which TZ?</div><div>Likely <a href="https://gerrit.ovirt.org/#/c/69323" class="">https://gerrit.ovirt.org/#/c/69323</a> merged 10:33:20 CET today </div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><div class="">2017-03-16 09:57:57,591-04 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{runAsync='true', vmId='00000000-0000-0000-0000-000000000000', status='Down', exitStatus='Normal'}), log id: 30ee3299</div><div class="">2017-03-16 09:57:57,593-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (DefaultQuartzScheduler1) [5d94233] Compiled stored procedure. Call string is [{call getvmdynamicbyvmguid(?)}]</div><div class="">2017-03-16 09:57:57,594-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (DefaultQuartzScheduler1) [5d94233] SqlCall for procedure [GetVmDynamicByVmGuid] compiled</div><div class="">2017-03-16 09:57:57,595-04 ERROR [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] Command 'SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{runAsync='true', vmId='00000000-0000-0000-0000-000000000000', status='Down', exitStatus='Normal'})' execution failed: null</div><div class="">2017-03-16 09:57:57,595-04 DEBUG [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] Exception: java.lang.NullPointerException</div><div class=""><span class="gmail-Apple-tab-span" style="white-space: pre;"> </span>at org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand.executeVDSCommand(SetVmStatusVDSCommand.java:33) [vdsbroker.jar:]</div><div class=""><span class="gmail-Apple-tab-span" style="white-space: pre;"> </span>at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:]</div><div class=""><span class="gmail-Apple-tab-span" style="white-space: pre;"> </span>at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:]</div><div class=""><span class="gmail-Apple-tab-span" style="white-space: pre;"> </span>at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:]</div><div class=""><span class="gmail-Apple-tab-span" style="white-space: pre;"> </span>at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:404) [vdsbroker.jar:]</div><div class=""><span class="gmail-Apple-tab-span" style="white-space: pre;"> </span>at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]</div><div class=""><span class="gmail-Apple-tab-span" style="white-space: pre;"> </span>at org.ovirt.engine.core.bll.VmHandler.unLockVm(VmHandler.java:377) [bll.jar:]</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Y.</div><div class=""><br class=""></div><div class="">[1] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/326/artifact/exported-artifacts/basic_suite_master__logs/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log" class="">http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/326/artifact/exported-artifacts/basic_suite_master__logs/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log</a></div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div class=""><div class=""><br class=""></div><div class=""><pre class="gmail-m_-3711950224702553848inbox-inbox-console-output" style="box-sizing: border-box; white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px; color: rgb(51, 51, 51); font-size: 16px;"><span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:05</b> </span><span style="box-sizing: border-box; color: rgb(0, 205, 205);" class=""> # snapshots_merge: </span>
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span><span style="box-sizing: border-box; color: rgb(205, 0, 0);" class="">Unhandled exception in <function <lambda> at 0x276a5f0>
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span>Traceback (most recent call last):
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> File "/usr/lib/python2.7/site-<wbr class="">packages/ovirtlago/testlib.py"<wbr class="">, line 217, in assert_equals_within
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> res = func()
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> File "/home/jenkins/workspace/<wbr class="">ovirt-system-tests_manual/<wbr class="">ovirt-system-tests/basic-<wbr class="">suite-master/test-scenarios/<wbr class="">004_basic_sanity.py", line 466, in <lambda>
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> api.vms.get(VM0_NAME).disks.<wbr class="">get(disk_name).status.state == 'ok'
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span>AttributeError: 'NoneType' object has no attribute 'state'</span>
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span><span style="box-sizing: border-box; color: rgb(205, 0, 0);" class="">Error while running thread
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span>Traceback (most recent call last):
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> File "/usr/lib/python2.7/site-<wbr class="">packages/lago/utils.py", line 57, in _ret_via_queue
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> queue.put({'return': func()})
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> File "/home/jenkins/workspace/<wbr class="">ovirt-system-tests_manual/<wbr class="">ovirt-system-tests/basic-<wbr class="">suite-master/test-scenarios/<wbr class="">004_basic_sanity.py", line 465, in snapshot_live_merge
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> lambda:
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> File "/usr/lib/python2.7/site-<wbr class="">packages/ovirtlago/testlib.py"<wbr class="">, line 264, in assert_true_within_long
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> assert_equals_within_long(<wbr class="">func, True, allowed_exceptions)
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> File "/usr/lib/python2.7/site-<wbr class="">packages/ovirtlago/testlib.py"<wbr class="">, line 251, in assert_equals_within_long
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> func, value, LONG_TIMEOUT, allowed_exceptions=allowed_<wbr class="">exceptions
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> File "/usr/lib/python2.7/site-<wbr class="">packages/ovirtlago/testlib.py"<wbr class="">, line 217, in assert_equals_within
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> res = func()
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> File "/home/jenkins/workspace/<wbr class="">ovirt-system-tests_manual/<wbr class="">ovirt-system-tests/basic-<wbr class="">suite-master/test-scenarios/<wbr class="">004_basic_sanity.py", line 466, in <lambda>
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span> api.vms.get(VM0_NAME).disks.<wbr class="">get(disk_name).status.state == 'ok'
<span class="gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;" class="">08:28:31</b> </span>AttributeError: 'NoneType' object has no attribute 'state'</span>
</pre><br class="gmail-m_-3711950224702553848inbox-inbox-Apple-interchange-newline"></div></div></div><br class="">______________________________<wbr class="">_________________<br class="">Devel mailing list<br class=""><a href="mailto:Devel@ovirt.org" class="">Devel@ovirt.org</a><br class=""><a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank" class="">http://lists.ovirt.org/<wbr class="">mailman/listinfo/devel</a><br class=""></blockquote></div><br class=""></div></div><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Devel mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:Devel@ovirt.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Devel@ovirt.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://lists.ovirt.org/mailman/listinfo/devel" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.ovirt.org/mailman/listinfo/devel</a></div></blockquote></div><br class=""></body></html>