<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Fri, Mar 17, 2017 at 9:49 PM Yaniv Kaul <<a href="mailto:ykaul@redhat.com">ykaul@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg">On Thu, Mar 16, 2017 at 8:16 PM, Michal Skrivanek <span dir="ltr" class="gmail_msg"><<a href="mailto:mskrivan@redhat.com" class="gmail_msg" target="_blank">mskrivan@redhat.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="gmail_msg"><br class="gmail_msg"><div class="gmail_msg"><span class="gmail_msg"><blockquote type="cite" class="gmail_msg"><div class="gmail_msg">On 16 Mar 2017, at 15:18, Yaniv Kaul <<a href="mailto:ykaul@redhat.com" class="gmail_msg" target="_blank">ykaul@redhat.com</a>> wrote:</div><br class="m_-7415587540470853133m_-2867758101781581971Apple-interchange-newline gmail_msg"><div class="gmail_msg"><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="gmail_msg"><div class="gmail_extra gmail_msg"><br class="m_-7415587540470853133m_-2867758101781581971Apple-interchange-newline gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg">On Thu, Mar 16, 2017 at 10:46 AM, Nir Soffer<span class="m_-7415587540470853133m_-2867758101781581971Apple-converted-space gmail_msg"> </span><span dir="ltr" class="gmail_msg"><<a href="mailto:nsoffer@redhat.com" class="gmail_msg" target="_blank">nsoffer@redhat.com</a>></span><span class="m_-7415587540470853133m_-2867758101781581971Apple-converted-space gmail_msg"> </span>wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg">If found this error in system tests - looks like wrong assert - code should check<div class="gmail_msg">if disk is not None before checking state.<br class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I'm not sure who is the owner of this test, so posting here.</div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Theoretically, perhaps. </div><div class="gmail_msg">Practically, it worked (until yesterday?) and now I'm also seeing this failure - it's not a coincidence.</div></div></div></div></div></blockquote><blockquote type="cite" class="gmail_msg"><div class="gmail_msg"><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">However, looking at my failure[1], I'm seeing other nasty stuff, which may explain the later on failures</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">For example, new NPE I have not seen in the past:</div><div class="gmail_msg"><div class="gmail_msg">2017-03-16 09:57:57,581-04 INFO [org.ovirt.engine.core.bll.AddVmTemplateCommand] (DefaultQuartzScheduler1) [5d94233] Ending command 'org.ovirt.engine.core.bll.AddVmTemplateCommand' successfully.</div></div></div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></span>Is this IST or which TZ?</div></div></blockquote></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Interesting question - I guess that's why we need the offset from UTC or use UTC in the logs.</div></div></div></div></blockquote><div><br></div><div>We have offset from UTC - "-04"</div><div><br></div><div>Do we have slaves here?</div><div><a href="https://time.is/map/UTC-4">https://time.is/map/UTC-4</a><br></div><div><br></div><div>Probably machine with incorrect timezone?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="gmail_msg"><div class="gmail_msg">Likely <a href="https://gerrit.ovirt.org/#/c/69323" class="gmail_msg" target="_blank">https://gerrit.ovirt.org/#/c/69323</a> merged 10:33:20 CET today </div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">That's the cause or the fix?</div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Y.</div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="gmail_msg"><div class="gmail_msg"><div class="m_-7415587540470853133h5 gmail_msg"><div class="gmail_msg"><br class="gmail_msg"><blockquote type="cite" class="gmail_msg"><div class="gmail_msg"><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><div class="gmail_msg">2017-03-16 09:57:57,591-04 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{runAsync='true', vmId='00000000-0000-0000-0000-000000000000', status='Down', exitStatus='Normal'}), log id: 30ee3299</div><div class="gmail_msg">2017-03-16 09:57:57,593-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (DefaultQuartzScheduler1) [5d94233] Compiled stored procedure. Call string is [{call getvmdynamicbyvmguid(?)}]</div><div class="gmail_msg">2017-03-16 09:57:57,594-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (DefaultQuartzScheduler1) [5d94233] SqlCall for procedure [GetVmDynamicByVmGuid] compiled</div><div class="gmail_msg">2017-03-16 09:57:57,595-04 ERROR [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] Command 'SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{runAsync='true', vmId='00000000-0000-0000-0000-000000000000', status='Down', exitStatus='Normal'})' execution failed: null</div><div class="gmail_msg">2017-03-16 09:57:57,595-04 DEBUG [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] Exception: java.lang.NullPointerException</div><div class="gmail_msg"><span class="m_-7415587540470853133m_-2867758101781581971gmail-Apple-tab-span gmail_msg" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand.executeVDSCommand(SetVmStatusVDSCommand.java:33) [vdsbroker.jar:]</div><div class="gmail_msg"><span class="m_-7415587540470853133m_-2867758101781581971gmail-Apple-tab-span gmail_msg" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:]</div><div class="gmail_msg"><span class="m_-7415587540470853133m_-2867758101781581971gmail-Apple-tab-span gmail_msg" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:]</div><div class="gmail_msg"><span class="m_-7415587540470853133m_-2867758101781581971gmail-Apple-tab-span gmail_msg" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:]</div><div class="gmail_msg"><span class="m_-7415587540470853133m_-2867758101781581971gmail-Apple-tab-span gmail_msg" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:404) [vdsbroker.jar:]</div><div class="gmail_msg"><span class="m_-7415587540470853133m_-2867758101781581971gmail-Apple-tab-span gmail_msg" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]</div><div class="gmail_msg"><span class="m_-7415587540470853133m_-2867758101781581971gmail-Apple-tab-span gmail_msg" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.bll.VmHandler.unLockVm(VmHandler.java:377) [bll.jar:]</div></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Y.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">[1] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/326/artifact/exported-artifacts/basic_suite_master__logs/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log" class="gmail_msg" target="_blank">http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/326/artifact/exported-artifacts/basic_suite_master__logs/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log</a></div><div class="gmail_msg"><br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><pre class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-console-output gmail_msg" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:16px"><span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:05</b> </span><span style="box-sizing:border-box;color:rgb(0,205,205)" class="gmail_msg"> # snapshots_merge: </span>
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span><span style="box-sizing:border-box;color:rgb(205,0,0)" class="gmail_msg">Unhandled exception in <function <lambda> at 0x276a5f0>
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span>Traceback (most recent call last):
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217, in assert_equals_within
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> res = func()
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> File "/home/jenkins/workspace/ovirt-system-tests_manual/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 466, in <lambda>
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> api.vms.get(VM0_NAME).<a href="http://disks.ge" class="gmail_msg" target="_blank">disks.ge</a>t(disk_name).status.state == 'ok'
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span>AttributeError: 'NoneType' object has no attribute 'state'</span>
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span><span style="box-sizing:border-box;color:rgb(205,0,0)" class="gmail_msg">Error while running thread
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span>Traceback (most recent call last):
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> queue.put({'return': func()})
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> File "/home/jenkins/workspace/ovirt-system-tests_manual/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 465, in snapshot_live_merge
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> lambda:
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 264, in assert_true_within_long
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> assert_equals_within_long(func, True, allowed_exceptions)
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 251, in assert_equals_within_long
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217, in assert_equals_within
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> res = func()
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> File "/home/jenkins/workspace/ovirt-system-tests_manual/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 466, in <lambda>
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span> api.vms.get(VM0_NAME).<a href="http://disks.ge" class="gmail_msg" target="_blank">disks.ge</a>t(disk_name).status.state == 'ok'
<span class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp gmail_msg" style="box-sizing:border-box"><b style="box-sizing:border-box" class="gmail_msg">08:28:31</b> </span>AttributeError: 'NoneType' object has no attribute 'state'</span>
</pre><br class="m_-7415587540470853133m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-Apple-interchange-newline gmail_msg"></div></div></div><br class="gmail_msg">_______________________________________________<br class="gmail_msg">Devel mailing list<br class="gmail_msg"><a href="mailto:Devel@ovirt.org" class="gmail_msg" target="_blank">Devel@ovirt.org</a><br class="gmail_msg"><a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a><br class="gmail_msg"></blockquote></div><br class="gmail_msg"></div></div><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important" class="gmail_msg">_______________________________________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="gmail_msg"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important" class="gmail_msg">Devel mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="gmail_msg"><a href="mailto:Devel@ovirt.org" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="gmail_msg" target="_blank">Devel@ovirt.org</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="gmail_msg"><a href="http://lists.ovirt.org/mailman/listinfo/devel" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="gmail_msg" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a></div></blockquote></div><br class="gmail_msg"></div></div></div></blockquote></div></div></div></blockquote></div></div>