<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 16, 2017 at 8:16 PM, Michal Skrivanek <span dir="ltr"><<a href="mailto:mskrivan@redhat.com" target="_blank">mskrivan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><span class=""><blockquote type="cite"><div>On 16 Mar 2017, at 15:18, Yaniv Kaul <<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>> wrote:</div><br class="m_-2867758101781581971Apple-interchange-newline"><div><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div class="gmail_extra"><br class="m_-2867758101781581971Apple-interchange-newline"><br><div class="gmail_quote">On Thu, Mar 16, 2017 at 10:46 AM, Nir Soffer<span class="m_-2867758101781581971Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span><span class="m_-2867758101781581971Apple-converted-space"> </span>wr<wbr>ote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">If found this error in system tests - looks like wrong assert - code should check<div>if disk is not None before checking state.<br><div><br></div><div>I'm not sure who is the owner of this test, so posting here.</div></div></div></blockquote><div><br></div><div>Theoretically, perhaps. </div><div>Practically, it worked (until yesterday?) and now I'm also seeing this failure - it's not a coincidence.</div></div></div></div></div></blockquote><blockquote type="cite"><div><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div class="gmail_extra"><div class="gmail_quote"><div>However, looking at my failure[1], I'm seeing other nasty stuff, which may explain the later on failures</div><div><br></div><div>For example, new NPE I have not seen in the past:</div><div><div>2017-03-16 09:57:57,581-04 INFO [org.ovirt.engine.core.bll.<wbr>AddVmTemplateCommand] (DefaultQuartzScheduler1) [5d94233] Ending command 'org.ovirt.engine.core.bll.<wbr>AddVmTemplateCommand' successfully.</div></div></div></div></div></div></blockquote><div><br></div></span>Is this IST or which TZ?</div></div></blockquote><div><br></div><div>Interesting question - I guess that's why we need the offset from UTC or use UTC in the logs.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Likely <a href="https://gerrit.ovirt.org/#/c/69323" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>69323</a> merged 10:33:20 CET today </div></div></blockquote><div><br></div><div>That's the cause or the fix?</div><div>Y.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div class="h5"><div><br><blockquote type="cite"><div><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div class="gmail_extra"><div class="gmail_quote"><div><div>2017-03-16 09:57:57,591-04 INFO [org.ovirt.engine.core.<wbr>vdsbroker.<wbr>SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameter<wbr>s:{runAsync='true', vmId='00000000-0000-0000-0000-<wbr>000000000000', status='Down', exitStatus='Normal'}), log id: 30ee3299</div><div>2017-03-16 09:57:57,593-04 DEBUG [org.ovirt.engine.core.dal.<wbr>dbbroker.<wbr>PostgresDbEngineDialect$<wbr>PostgresSimpleJdbcCall] (DefaultQuartzScheduler1) [5d94233] Compiled stored procedure. Call string is [{call getvmdynamicbyvmguid(?)}]</div><div>2017-03-16 09:57:57,594-04 DEBUG [org.ovirt.engine.core.dal.<wbr>dbbroker.<wbr>PostgresDbEngineDialect$<wbr>PostgresSimpleJdbcCall] (DefaultQuartzScheduler1) [5d94233] SqlCall for procedure [GetVmDynamicByVmGuid] compiled</div><div>2017-03-16 09:57:57,595-04 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.<wbr>SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] Command 'SetVmStatusVDSCommand( SetVmStatusVDSCommandParameter<wbr>s:{runAsync='true', vmId='00000000-0000-0000-0000-<wbr>000000000000', status='Down', exitStatus='Normal'})' execution failed: null</div><div>2017-03-16 09:57:57,595-04 DEBUG [org.ovirt.engine.core.<wbr>vdsbroker.<wbr>SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] Exception: java.lang.NullPointerException</div><div><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.<wbr>vdsbroker.<wbr>SetVmStatusVDSCommand.<wbr>executeVDSCommand(<wbr>SetVmStatusVDSCommand.java:33) [vdsbroker.jar:]</div><div><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.<wbr>vdsbroker.VDSCommandBase.<wbr>executeCommand(VDSCommandBase.<wbr>java:73) [vdsbroker.jar:]</div><div><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.dal.<wbr>VdcCommandBase.execute(<wbr>VdcCommandBase.java:33) [dal.jar:]</div><div><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>DefaultVdsCommandExecutor.<wbr>execute(<wbr>DefaultVdsCommandExecutor.<wbr>java:14) [vdsbroker.jar:]</div><div><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.<wbr>vdsbroker.ResourceManager.<wbr>runVdsCommand(ResourceManager.<wbr>java:404) [vdsbroker.jar:]</div><div><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.bll.<wbr>VDSBrokerFrontendImpl.<wbr>runVdsCommand(<wbr>VDSBrokerFrontendImpl.java:33) [bll.jar:]</div><div><span class="m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.bll.<wbr>VmHandler.unLockVm(VmHandler.<wbr>java:377) [bll.jar:]</div></div><div><br></div><div><br></div><div><br></div><div>Y.</div><div><br></div><div>[1] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/326/artifact/exported-artifacts/basic_suite_master__logs/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-system-tests_master_<wbr>check-patch-el7-x86_64/326/<wbr>artifact/exported-artifacts/<wbr>basic_suite_master__logs/test_<wbr>logs/basic-suite-master/post-<wbr>004_basic_sanity.py/lago-<wbr>basic-suite-master-engine/_<wbr>var_log/ovirt-engine/engine.<wbr>log</a></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div><div><pre class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:16px"><span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:05</b> </span><span style="box-sizing:border-box;color:rgb(0,205,205)"> # snapshots_merge: </span>
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span><span style="box-sizing:border-box;color:rgb(205,0,0)">Unhandled exception in <function <lambda> at 0x276a5f0>
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span>Traceback (most recent call last):
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/testlib.py", line 217, in assert_equals_within
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> res = func()
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/home/jenkins/workspace/ovirt<wbr>-system-tests_manual/ovirt-<wbr>system-tests/basic-suite-<wbr>master/test-scenarios/004_<wbr>basic_sanity.py", line 466, in <lambda>
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> api.vms.get(VM0_NAME).<a href="http://disks.ge">disks.ge</a><wbr>t(disk_name).status.state == 'ok'
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span>AttributeError: 'NoneType' object has no attribute 'state'</span>
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span><span style="box-sizing:border-box;color:rgb(205,0,0)">Error while running thread
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span>Traceback (most recent call last):
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr>ges/lago/utils.py", line 57, in _ret_via_queue
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> queue.put({'return': func()})
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/home/jenkins/workspace/ovirt<wbr>-system-tests_manual/ovirt-<wbr>system-tests/basic-suite-<wbr>master/test-scenarios/004_<wbr>basic_sanity.py", line 465, in snapshot_live_merge
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> lambda:
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/testlib.py", line 264, in assert_true_within_long
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> assert_equals_within_long(func<wbr>, True, allowed_exceptions)
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/testlib.py", line 251, in assert_equals_within_long
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exc<wbr>eptions
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/testlib.py", line 217, in assert_equals_within
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> res = func()
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/home/jenkins/workspace/ovirt<wbr>-system-tests_manual/ovirt-<wbr>system-tests/basic-suite-<wbr>master/test-scenarios/004_<wbr>basic_sanity.py", line 466, in <lambda>
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> api.vms.get(VM0_NAME).<a href="http://disks.ge">disks.ge</a><wbr>t(disk_name).status.state == 'ok'
<span class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span>AttributeError: 'NoneType' object has no attribute 'state'</span>
</pre><br class="m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-Apple-interchange-newline"></div></div></div><br>______________________________<wbr>_________________<br>Devel mailing list<br><a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br><a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br></blockquote></div><br></div></div><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">______________________________<wbr>_________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">Devel mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="mailto:Devel@ovirt.org" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">Devel@ovirt.org</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://lists.ovirt.org/mailman/listinfo/devel" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a></div></blockquote></div><br></div></div></div></blockquote></div><br></div></div>