<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 17 Mar 2017, at 20:49, Yaniv Kaul <<a href="mailto:ykaul@redhat.com" class="">ykaul@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Thu, Mar 16, 2017 at 8:16 PM, Michal Skrivanek <span dir="ltr" class=""><<a href="mailto:mskrivan@redhat.com" target="_blank" class="">mskrivan@redhat.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><br class=""><div class=""><span class=""><blockquote type="cite" class=""><div class="">On 16 Mar 2017, at 15:18, Yaniv Kaul <<a href="mailto:ykaul@redhat.com" target="_blank" class="">ykaul@redhat.com</a>> wrote:</div><br class="m_-2867758101781581971Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><div class="gmail_extra"><br class="m_-2867758101781581971Apple-interchange-newline"><br class=""><div class="gmail_quote">On Thu, Mar 16, 2017 at 10:46 AM, Nir Soffer<span class="m_-2867758101781581971Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:nsoffer@redhat.com" target="_blank" class="">nsoffer@redhat.com</a>></span><span class="m_-2867758101781581971Apple-converted-space"> </span>wr<wbr class="">ote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">If found this error in system tests - looks like wrong assert - code should check<div class="">if disk is not None before checking state.<br class=""><div class=""><br class=""></div><div class="">I'm not sure who is the owner of this test, so posting here.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Theoretically, perhaps. </div><div class="">Practically, it worked (until yesterday?) and now I'm also seeing this failure - it's not a coincidence.</div></div></div></div></div></blockquote><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">However, looking at my failure[1], I'm seeing other nasty stuff, which may explain the later on failures</div><div class=""><br class=""></div><div class="">For example, new NPE I have not seen in the past:</div><div class=""><div class="">2017-03-16 09:57:57,581-04 INFO [org.ovirt.engine.core.bll.<wbr class="">AddVmTemplateCommand] (DefaultQuartzScheduler1) [5d94233] Ending command 'org.ovirt.engine.core.bll.<wbr class="">AddVmTemplateCommand' successfully.</div></div></div></div></div></div></blockquote><div class=""><br class=""></div></span>Is this IST or which TZ?</div></div></blockquote><div class=""><br class=""></div><div class="">Interesting question - I guess that's why we need the offset from UTC or use UTC in the logs.</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">Likely <a href="https://gerrit.ovirt.org/#/c/69323" target="_blank" class="">https://gerrit.ovirt.org/#/c/<wbr class="">69323</a> merged 10:33:20 CET today </div></div></blockquote><div class=""><br class=""></div><div class="">That's the cause or the fix?</div></div></div></div></div></blockquote><div><br class=""></div>oh, sorry, the cause.</div><div>we were not able to find anyone to confirm and resolve it today unfortunately</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">Y.</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""><div class="h5"><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><div class="">2017-03-16 09:57:57,591-04 INFO [org.ovirt.engine.core.<wbr class="">vdsbroker.<wbr class="">SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameter<wbr class="">s:{runAsync='true', vmId='00000000-0000-0000-0000-<wbr class="">000000000000', status='Down', exitStatus='Normal'}), log id: 30ee3299</div><div class="">2017-03-16 09:57:57,593-04 DEBUG [org.ovirt.engine.core.dal.<wbr class="">dbbroker.<wbr class="">PostgresDbEngineDialect$<wbr class="">PostgresSimpleJdbcCall] (DefaultQuartzScheduler1) [5d94233] Compiled stored procedure. Call string is [{call getvmdynamicbyvmguid(?)}]</div><div class="">2017-03-16 09:57:57,594-04 DEBUG [org.ovirt.engine.core.dal.<wbr class="">dbbroker.<wbr class="">PostgresDbEngineDialect$<wbr class="">PostgresSimpleJdbcCall] (DefaultQuartzScheduler1) [5d94233] SqlCall for procedure [GetVmDynamicByVmGuid] compiled</div><div class="">2017-03-16 09:57:57,595-04 ERROR [org.ovirt.engine.core.<wbr class="">vdsbroker.<wbr class="">SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] Command 'SetVmStatusVDSCommand( SetVmStatusVDSCommandParameter<wbr class="">s:{runAsync='true', vmId='00000000-0000-0000-0000-<wbr class="">000000000000', status='Down', exitStatus='Normal'})' execution failed: null</div><div class="">2017-03-16 09:57:57,595-04 DEBUG [org.ovirt.engine.core.<wbr class="">vdsbroker.<wbr class="">SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] Exception: java.lang.NullPointerException</div><div class=""><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.<wbr class="">vdsbroker.<wbr class="">SetVmStatusVDSCommand.<wbr class="">executeVDSCommand(<wbr class="">SetVmStatusVDSCommand.java:33) [vdsbroker.jar:]</div><div class=""><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.<wbr class="">vdsbroker.VDSCommandBase.<wbr class="">executeCommand(VDSCommandBase.<wbr class="">java:73) [vdsbroker.jar:]</div><div class=""><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.dal.<wbr class="">VdcCommandBase.execute(<wbr class="">VdcCommandBase.java:33) [dal.jar:]</div><div class=""><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.<wbr class="">vdsbroker.vdsbroker.<wbr class="">DefaultVdsCommandExecutor.<wbr class="">execute(<wbr class="">DefaultVdsCommandExecutor.<wbr class="">java:14) [vdsbroker.jar:]</div><div class=""><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.<wbr class="">vdsbroker.ResourceManager.<wbr class="">runVdsCommand(ResourceManager.<wbr class="">java:404) [vdsbroker.jar:]</div><div class=""><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.bll.<wbr class="">VDSBrokerFrontendImpl.<wbr class="">runVdsCommand(<wbr class="">VDSBrokerFrontendImpl.java:33) [bll.jar:]</div><div class=""><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.bll.<wbr class="">VmHandler.unLockVm(VmHandler.<wbr class="">java:377) [bll.jar:]</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Y.</div><div class=""><br class=""></div><div class="">[1] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/326/artifact/exported-artifacts/basic_suite_master__logs/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log" target="_blank" class="">http://jenkins.ovirt.org/job/<wbr class="">ovirt-system-tests_master_<wbr class="">check-patch-el7-x86_64/326/<wbr class="">artifact/exported-artifacts/<wbr class="">basic_suite_master__logs/test_<wbr class="">logs/basic-suite-master/post-<wbr class="">004_basic_sanity.py/lago-<wbr class="">basic-suite-master-engine/_<wbr class="">var_log/ovirt-engine/engine.<wbr class="">log</a></div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class=""><div class=""><br class=""></div><div class=""><pre class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:16px"><span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:05</b> </span><span style="box-sizing:border-box;color:rgb(0,205,205)" class=""> # snapshots_merge: </span>
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span><span style="box-sizing:border-box;color:rgb(205,0,0)" class="">Unhandled exception in <function <lambda> at 0x276a5f0>
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span>Traceback (most recent call last):
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr class="">ges/ovirtlago/testlib.py", line 217, in assert_equals_within
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> res = func()
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> File "/home/jenkins/workspace/ovirt<wbr class="">-system-tests_manual/ovirt-<wbr class="">system-tests/basic-suite-<wbr class="">master/test-scenarios/004_<wbr class="">basic_sanity.py", line 466, in <lambda>
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> api.vms.get(VM0_NAME).<a href="http://disks.ge/" class="">disks.ge</a><wbr class="">t(disk_name).status.state == 'ok'
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span>AttributeError: 'NoneType' object has no attribute 'state'</span>
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span><span style="box-sizing:border-box;color:rgb(205,0,0)" class="">Error while running thread
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span>Traceback (most recent call last):
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr class="">ges/lago/utils.py", line 57, in _ret_via_queue
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> queue.put({'return': func()})
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> File "/home/jenkins/workspace/ovirt<wbr class="">-system-tests_manual/ovirt-<wbr class="">system-tests/basic-suite-<wbr class="">master/test-scenarios/004_<wbr class="">basic_sanity.py", line 465, in snapshot_live_merge
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> lambda:
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr class="">ges/ovirtlago/testlib.py", line 264, in assert_true_within_long
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> assert_equals_within_long(func<wbr class="">, True, allowed_exceptions)
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr class="">ges/ovirtlago/testlib.py", line 251, in assert_equals_within_long
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exc<wbr class="">eptions
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr class="">ges/ovirtlago/testlib.py", line 217, in assert_equals_within
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> res = func()
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> File "/home/jenkins/workspace/ovirt<wbr class="">-system-tests_manual/ovirt-<wbr class="">system-tests/basic-suite-<wbr class="">master/test-scenarios/004_<wbr class="">basic_sanity.py", line 466, in <lambda>
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span> api.vms.get(VM0_NAME).<a href="http://disks.ge/" class="">disks.ge</a><wbr class="">t(disk_name).status.state == 'ok'
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box" class="">08:28:31</b> </span>AttributeError: 'NoneType' object has no attribute 'state'</span>
</pre><br class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-Apple-interchange-newline"></div></div></div><br class="">______________________________<wbr class="">_________________<br class="">Devel mailing list<br class=""><a href="mailto:Devel@ovirt.org" target="_blank" class="">Devel@ovirt.org</a><br class=""><a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank" class="">http://lists.ovirt.org/mailman<wbr class="">/listinfo/devel</a><br class=""></blockquote></div><br class=""></div></div><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important" class="">______________________________<wbr class="">_________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important" class="">Devel mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><a href="mailto:Devel@ovirt.org" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" class="">Devel@ovirt.org</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><a href="http://lists.ovirt.org/mailman/listinfo/devel" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" class="">http://lists.ovirt.org/<wbr class="">mailman/listinfo/devel</a></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div>
</div></blockquote></div><br class=""></body></html>