<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Mar 19, 2017 at 12:14 PM, Arik Hadas <span dir="ltr"><<a href="mailto:ahadas@redhat.com" target="_blank">ahadas@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Sat, Mar 18, 2017 at 12:57 PM, Yaniv Kaul <span dir="ltr"><<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="m_-85078513005477866gmail-">On Sat, Mar 18, 2017 at 12:05 AM, Michal Skrivanek <span dir="ltr"><<a href="mailto:mskrivan@redhat.com" target="_blank">mskrivan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><span><blockquote type="cite"><div>On 17 Mar 2017, at 20:49, Yaniv Kaul <<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>> wrote:</div><br class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472Apple-interchange-newline"><div><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 16, 2017 at 8:16 PM, Michal Skrivanek <span dir="ltr"><<a href="mailto:mskrivan@redhat.com" target="_blank">mskrivan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><span><blockquote type="cite"><div>On 16 Mar 2017, at 15:18, Yaniv Kaul <<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>> wrote:</div><br class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971Apple-interchange-newline"><div><div dir="ltr" style="font-family:helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div class="gmail_extra"><br class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971Apple-interchange-newline"><br><div class="gmail_quote">On Thu, Mar 16, 2017 at 10:46 AM, Nir Soffer<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span><span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971Apple-converted-space"> </span>wr<wbr>ote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">If found this error in system tests - looks like wrong assert - code should check<div>if disk is not None before checking state.<br><div><br></div><div>I'm not sure who is the owner of this test, so posting here.</div></div></div></blockquote><div><br></div><div>Theoretically, perhaps. </div><div>Practically, it worked (until yesterday?) and now I'm also seeing this failure - it's not a coincidence.</div></div></div></div></div></blockquote><blockquote type="cite"><div><div dir="ltr" style="font-family:helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div class="gmail_extra"><div class="gmail_quote"><div>However, looking at my failure[1], I'm seeing other nasty stuff, which may explain the later on failures</div><div><br></div><div>For example, new NPE I have not seen in the past:</div><div><div>2017-03-16 09:57:57,581-04 INFO [<a href="http://org.ovirt.engine.core.bll.Ad" target="_blank">org.ovirt.engine.core.bll.Ad</a><wbr>dVmTemplateCommand] (DefaultQuartzScheduler1) [5d94233] Ending command 'org.ovirt.engine.core.bll.Add<wbr>VmTemplateCommand' successfully.</div></div></div></div></div></div></blockquote><div><br></div></span>Is this IST or which TZ?</div></div></blockquote><div><br></div><div>Interesting question - I guess that's why we need the offset from UTC or use UTC in the logs.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Likely <a href="https://gerrit.ovirt.org/#/c/69323" target="_blank">https://gerrit.ovirt.org/#/c/6<wbr>9323</a> merged 10:33:20 CET today </div></div></blockquote><div><br></div><div>That's the cause or the fix?</div></div></div></div></div></blockquote><div><br></div></span>oh, sorry, the cause.</div><div>we were not able to find anyone to confirm and resolve it today unfortunately</div></div></blockquote></span></div></div></div></blockquote><div><br></div></span><div>A fix is posted: <a href="https://gerrit.ovirt.org/#/c/74276/" target="_blank">https://gerrit.ovirt.<wbr>org/#/c/74276/</a></div><div>The cause is explained in the commit message.</div></div></div></div></blockquote><div><br></div><div>Thanks.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>In our defense I'll say that somehow the system tests passed when we ran them ([1]) and that the command for adding a template became way too complicated than it is supposed to be with the introduction of import from glance and instance-types.</div></div></div></div></blockquote><div><br></div><div>Do we have intentions to simplify it?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>[1] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_manual/129/" target="_blank">http://jenkins.ovirt.org/<wbr>job/ovirt-system-tests_manual/<wbr>129/</a></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-85078513005477866gmail-"><div><br></div></span><div>A secondary bug is that it fails every 10 seconds or so, continuously.</div><span class="m_-85078513005477866gmail-HOEnZb"><font color="#888888"><div>Y.</div></font></span></div></div></div></blockquote><div><br></div></span><div>Right, because it fails in the end-action phase of the command and we have a retry mechanism that attempts to end the command (unless stated otherwise) periodically.</div></div></div></div></blockquote><div><br></div><div>But the end user experience is an event every few seconds (10?) that we fail an operation.</div><div>When does it give up?</div><div>Y.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="m_-85078513005477866gmail-h5"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div class="m_-85078513005477866gmail-m_-4732998431735772322h5"><div><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Y.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472h5"><div><br><blockquote type="cite"><div><div dir="ltr" style="font-family:helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div class="gmail_extra"><div class="gmail_quote"><div><div>2017-03-16 09:57:57,591-04 INFO [org.ovirt.engine.core.vdsbro<wbr>ker.SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameter<wbr>s:{runAsync='true', vmId='00000000-0000-0000-0000-<wbr>000000000000', status='Down', exitStatus='Normal'}), log id: 30ee3299</div><div>2017-03-16 09:57:57,593-04 DEBUG [org.ovirt.engine.core.dal.dbb<wbr>roker.PostgresDbEngineDialect$<wbr>PostgresSimpleJdbcCall] (DefaultQuartzScheduler1) [5d94233] Compiled stored procedure. Call string is [{call getvmdynamicbyvmguid(?)}]</div><div>2017-03-16 09:57:57,594-04 DEBUG [org.ovirt.engine.core.dal.dbb<wbr>roker.PostgresDbEngineDialect$<wbr>PostgresSimpleJdbcCall] (DefaultQuartzScheduler1) [5d94233] SqlCall for procedure [GetVmDynamicByVmGuid] compiled</div><div>2017-03-16 09:57:57,595-04 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] Command 'SetVmStatusVDSCommand( SetVmStatusVDSCommandParameter<wbr>s:{runAsync='true', vmId='00000000-0000-0000-0000-<wbr>000000000000', status='Down', exitStatus='Normal'})' execution failed: null</div><div>2017-03-16 09:57:57,595-04 DEBUG [org.ovirt.engine.core.vdsbrok<wbr>er.SetVmStatusVDSCommand] (DefaultQuartzScheduler1) [5d94233] Exception: java.lang.NullPointerException</div><div><span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.vdsbroke<wbr>r.SetVmStatusVDSCommand.execut<wbr>eVDSCommand(SetVmStatusVDSComm<wbr>and.java:33) [vdsbroker.jar:]</div><div><span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.vdsbroke<wbr>r.VDSCommandBase.executeComman<wbr>d(VDSCommandBase.java:73) [vdsbroker.jar:]</div><div><span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.dal.VdcC<wbr>ommandBase.execute(VdcCommandB<wbr>ase.java:33) [dal.jar:]</div><div><span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.vdsbroke<wbr>r.vdsbroker.DefaultVdsCommandE<wbr>xecutor.execute(DefaultVdsComm<wbr>andExecutor.java:14) [vdsbroker.jar:]</div><div><span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.vdsbroke<wbr>r.ResourceManager.runVdsComman<wbr>d(ResourceManager.java:404) [vdsbroker.jar:]</div><div><span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.bll.VDSB<wbr>rokerFrontendImpl.runVdsComman<wbr>d(VDSBrokerFrontendImpl.java:3<wbr>3) [bll.jar:]</div><div><span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>at org.ovirt.engine.core.bll.VmHa<wbr>ndler.unLockVm(VmHandler.java:<wbr>377) [bll.jar:]</div></div><div><br></div><div><br></div><div><br></div><div>Y.</div><div><br></div><div>[1] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/326/artifact/exported-artifacts/basic_suite_master__logs/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-system-tests_master_chec<wbr>k-patch-el7-x86_64/326/artifac<wbr>t/exported-artifacts/basic_sui<wbr>te_master__logs/test_logs/basi<wbr>c-suite-master/post-004_basic_<wbr>sanity.py/lago-basic-suite-mas<wbr>ter-engine/_var_log/ovirt-engi<wbr>ne/engine.log</a></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div><div><pre class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:16px"><span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:05</b> </span><span style="box-sizing:border-box;color:rgb(0,205,205)"> # snapshots_merge: </span>
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span><span style="box-sizing:border-box;color:rgb(205,0,0)">Unhandled exception in <function <lambda> at 0x276a5f0>
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span>Traceback (most recent call last):
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/testlib.py", line 217, in assert_equals_within
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> res = func()
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/home/jenkins/workspace/ovirt<wbr>-system-tests_manual/ovirt-sys<wbr>tem-tests/basic-suite-master/t<wbr>est-scenarios/004_basic_sanity<wbr>.py", line 466, in <lambda>
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> api.vms.get(VM0_NAME).<a href="http://disks.ge/" target="_blank">disks.ge</a><wbr>t(disk_name).status.state == 'ok'
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span>AttributeError: 'NoneType' object has no attribute 'state'</span>
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span><span style="box-sizing:border-box;color:rgb(205,0,0)">Error while running thread
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span>Traceback (most recent call last):
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr>ges/lago/utils.py", line 57, in _ret_via_queue
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> queue.put({'return': func()})
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/home/jenkins/workspace/ovirt<wbr>-system-tests_manual/ovirt-sys<wbr>tem-tests/basic-suite-master/t<wbr>est-scenarios/004_basic_sanity<wbr>.py", line 465, in snapshot_live_merge
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> lambda:
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/testlib.py", line 264, in assert_true_within_long
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> assert_equals_within_long(func<wbr>, True, allowed_exceptions)
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/testlib.py", line 251, in assert_equals_within_long
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exc<wbr>eptions
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/testlib.py", line 217, in assert_equals_within
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> res = func()
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> File "/home/jenkins/workspace/ovirt<wbr>-system-tests_manual/ovirt-sys<wbr>tem-tests/basic-suite-master/t<wbr>est-scenarios/004_basic_sanity<wbr>.py", line 466, in <lambda>
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span> api.vms.get(VM0_NAME).<a href="http://disks.ge/" target="_blank">disks.ge</a><wbr>t(disk_name).status.state == 'ok'
<span class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">08:28:31</b> </span>AttributeError: 'NoneType' object has no attribute 'state'</span>
</pre><br class="m_-85078513005477866gmail-m_-4732998431735772322m_-1655747182593948472m_-2867758101781581971gmail-m_-3711950224702553848inbox-inbox-Apple-interchange-newline"></div></div></div><br>______________________________<wbr>_________________<br>Devel mailing list<br><a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br><a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br></blockquote></div><br></div></div><span style="font-family:helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline">______________________________<wbr>_________________</span><br style="font-family:helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="font-family:helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline">Devel mailing list</span><br style="font-family:helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="mailto:Devel@ovirt.org" style="font-family:helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">Devel@ovirt.org</a><br style="font-family:helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://lists.ovirt.org/mailman/listinfo/devel" style="font-family:helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a></div></blockquote></div><br></div></div></div></blockquote></div><br></div></div>
</div></blockquote></div><br></div></div></div></blockquote></div></div></div><br></div></div>
<br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div>