Re: [ovirt-devel] System tests failure (wrong test code?)

--Apple-Mail=_99B42EE4-4513-4FE9-8F9A-A1E183AF4102 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii <mailto:nsoffer@redhat.com>> wrote: should check failure - it's not a coincidence. may explain the later on failures [org.ovirt.engine.core.bll.AddVmTemplateCommand] = (DefaultQuartzScheduler1) [5d94233] Ending command = 'org.ovirt.engine.core.bll.AddVmTemplateCommand' successfully. Is this IST or which TZ? Likely https://gerrit.ovirt.org/#/c/69323 = <https://gerrit.ovirt.org/#/c/69323> merged 10:33:20 CET today=20 [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] = (DefaultQuartzScheduler1) [5d94233] START, SetVmStatusVDSCommand( = SetVmStatusVDSCommandParameters:{runAsync=3D'true', = vmId=3D'00000000-0000-0000-0000-000000000000', status=3D'Down', = exitStatus=3D'Normal'}), log id: 30ee3299 [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimple= JdbcCall] (DefaultQuartzScheduler1) [5d94233] Compiled stored procedure. = Call string is [{call getvmdynamicbyvmguid(?)}] [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimple= JdbcCall] (DefaultQuartzScheduler1) [5d94233] SqlCall for procedure = [GetVmDynamicByVmGuid] compiled [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] = (DefaultQuartzScheduler1) [5d94233] Command 'SetVmStatusVDSCommand( = SetVmStatusVDSCommandParameters:{runAsync=3D'true', = vmId=3D'00000000-0000-0000-0000-000000000000', status=3D'Down', = exitStatus=3D'Normal'})' execution failed: null [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] = (DefaultQuartzScheduler1) [5d94233] Exception: = java.lang.NullPointerException org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand.executeVDSCommand(Se= tVmStatusVDSCommand.java:33) [vdsbroker.jar:] org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBa= se.java:73) [vdsbroker.jar:] org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) = [dal.jar:] org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execut= e(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceMana= ger.java:404) [vdsbroker.jar:] org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFro= ntendImpl.java:33) [bll.jar:] org.ovirt.engine.core.bll.VmHandler.unLockVm(VmHandler.java:377) = [bll.jar:] http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86= _64/326/artifact/exported-artifacts/basic_suite_master__logs/test_logs/bas= ic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-engine/_v= ar_log/ovirt-engine/engine.log = <http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x8= 6_64/326/artifact/exported-artifacts/basic_suite_master__logs/test_logs/ba= sic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-engine/_= var_log/ovirt-engine/engine.log> "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217, in = assert_equals_within "/home/jenkins/workspace/ovirt-system-tests_manual/ovirt-system-tests/basi= c-suite-master/test-scenarios/004_basic_sanity.py", line 466, in = <lambda> =3D=3D 'ok' 57, in _ret_via_queue "/home/jenkins/workspace/ovirt-system-tests_manual/ovirt-system-tests/basi= c-suite-master/test-scenarios/004_basic_sanity.py", line 465, in = snapshot_live_merge "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 264, in = assert_true_within_long "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 251, in = assert_equals_within_long allowed_exceptions=3Dallowed_exceptions "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217, in = assert_equals_within "/home/jenkins/workspace/ovirt-system-tests_manual/ovirt-system-tests/basi= c-suite-master/test-scenarios/004_basic_sanity.py", line 466, in = <lambda> =3D=3D 'ok' <http://lists.ovirt.org/mailman/listinfo/devel> <http://lists.ovirt.org/mailman/listinfo/devel> --Apple-Mail=_99B42EE4-4513-4FE9-8F9A-A1E183AF4102 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 16 Mar 2017, at 15:18, Yaniv Kaul &lt;<a = href=3D"mailto:ykaul@redhat.com" class=3D&quot;&quot;&gt;ykaul(a)redhat.com&lt;/a&gt;&amp;gt; = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" style=3D"font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><div = class=3D"gmail_extra"><br class=3D"Apple-interchange-newline"><br = class=3D""><div class=3D"gmail_quote">On Thu, Mar 16, 2017 at 10:46 AM, = Nir Soffer<span class=3D"Apple-converted-space">&nbsp;</span><span = dir=3D"ltr" class=3D"">&lt;<a href=3D"mailto:nsoffer@redhat.com" = target=3D"_blank" class=3D&quot;&quot;&gt;nsoffer(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt;&lt;span = class=3D"Apple-converted-space">&nbsp;</span>wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px = 0px 0.8ex; border-left-width: 1px; border-left-style: solid; = border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div = dir=3D"ltr" class=3D"">If found this error in system tests - looks like = wrong assert - code should check<div class=3D"">if disk is not None = before checking state.<br class=3D""><div class=3D""><br = class=3D""></div><div class=3D"">I'm not sure who is the owner of this = test, so posting here.</div></div></div></blockquote><div class=3D""><br = class=3D""></div><div class=3D"">Theoretically, perhaps.&nbsp;</div><div = class=3D"">Practically, it worked (until yesterday?) and now I'm also = seeing this failure - it's not a = coincidence.</div></div></div></div></div></blockquote><blockquote = type=3D"cite" class=3D""><div class=3D""><div dir=3D"ltr" = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><div = class=3D"gmail_extra"><div class=3D"gmail_quote"><div class=3D"">However, = looking at my failure[1], I'm seeing other nasty stuff, which may = explain the later on failures</div><div class=3D""><br = class=3D""></div><div class=3D"">For example, new NPE I have not seen in = the past:</div><div class=3D""><div class=3D"">2017-03-16 = 09:57:57,581-04 INFO = &nbsp;[org.ovirt.engine.core.bll.AddVmTemplateCommand] = (DefaultQuartzScheduler1) [5d94233] Ending command = 'org.ovirt.engine.core.bll.AddVmTemplateCommand' = successfully.</div></div></div></div></div></div></blockquote><div><br = class=3D""></div>Is this IST or which TZ?</div><div>Likely <a = href=3D"https://gerrit.ovirt.org/#/c/69323" = class=3D"">https://gerrit.ovirt.org/#/c/69323</a>&... 10:33:20 = CET today&nbsp;</div><div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div dir=3D"ltr" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><div class=3D"gmail_extra"><div = class=3D"gmail_quote"><div class=3D""><div class=3D"">2017-03-16 = 09:57:57,591-04 INFO = &nbsp;[org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] = (DefaultQuartzScheduler1) [5d94233] START, SetVmStatusVDSCommand( = SetVmStatusVDSCommandParameters:{runAsync=3D'true', = vmId=3D'00000000-0000-0000-0000-000000000000', status=3D'Down', = exitStatus=3D'Normal'}), log id: 30ee3299</div><div class=3D"">2017-03-16 = 09:57:57,593-04 DEBUG = [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimple= JdbcCall] (DefaultQuartzScheduler1) [5d94233] Compiled stored procedure. = Call string is [{call getvmdynamicbyvmguid(?)}]</div><div = class=3D"">2017-03-16 09:57:57,594-04 DEBUG = [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimple= JdbcCall] (DefaultQuartzScheduler1) [5d94233] SqlCall for procedure = [GetVmDynamicByVmGuid] compiled</div><div class=3D"">2017-03-16 = 09:57:57,595-04 ERROR = [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] = (DefaultQuartzScheduler1) [5d94233] Command 'SetVmStatusVDSCommand( = SetVmStatusVDSCommandParameters:{runAsync=3D'true', = vmId=3D'00000000-0000-0000-0000-000000000000', status=3D'Down', = exitStatus=3D'Normal'})' execution failed: null</div><div = class=3D"">2017-03-16 09:57:57,595-04 DEBUG = [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] = (DefaultQuartzScheduler1) [5d94233] Exception: = java.lang.NullPointerException</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space: pre;"> = </span>at = org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand.executeVDSCommand(Se= tVmStatusVDSCommand.java:33) [vdsbroker.jar:]</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space: pre;"> = </span>at = org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBa= se.java:73) [vdsbroker.jar:]</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space: pre;"> = </span>at = org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) = [dal.jar:]</div><div class=3D""><span class=3D"gmail-Apple-tab-span" = style=3D"white-space: pre;"> </span>at = org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execut= e(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:]</div><div = class=3D""><span class=3D"gmail-Apple-tab-span" style=3D"white-space: = pre;"> </span>at = org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceMana= ger.java:404) [vdsbroker.jar:]</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space: pre;"> = </span>at = org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFro= ntendImpl.java:33) [bll.jar:]</div><div class=3D""><span = class=3D"gmail-Apple-tab-span" style=3D"white-space: pre;"> = </span>at = org.ovirt.engine.core.bll.VmHandler.unLockVm(VmHandler.java:377) = [bll.jar:]</div></div><div class=3D""><br class=3D""></div><div = class=3D""><br class=3D""></div><div class=3D""><br class=3D""></div><div = class=3D"">Y.</div><div class=3D""><br class=3D""></div><div = class=3D"">[1] &nbsp;<a = href=3D"http://jenkins.ovirt.org/job/ovirt-system-tests_master_check... -el7-x86_64/326/artifact/exported-artifacts/basic_suite_master__logs/test_= logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-e= ngine/_var_log/ovirt-engine/engine.log" = class=3D"">http://jenkins.ovirt.org/job/ovirt-system-tests_m... tch-el7-x86_64/326/artifact/exported-artifacts/basic_suite_master__logs/te= st_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-maste= r-engine/_var_log/ovirt-engine/engine.log</a></div><div class=3D""><br = class=3D""></div><blockquote class=3D"gmail_quote" style=3D"margin: 0px = 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; = border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div = dir=3D"ltr" class=3D""><div class=3D""><div class=3D""><br = class=3D""></div><div class=3D""><pre = class=3D"gmail-m_-3711950224702553848inbox-inbox-console-output" = style=3D"box-sizing: border-box; white-space: pre-wrap; word-wrap: = break-word; margin-top: 0px; margin-bottom: 0px; color: rgb(51, 51, 51); = font-size: 16px;"><span = class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:05</b> </span><span style=3D"box-sizing: border-box; = color: rgb(0, 205, 205);" class=3D""> # snapshots_merge: </span> <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span><span style=3D"box-sizing: border-box; = color: rgb(205, 0, 0);" class=3D"">Unhandled exception in &lt;function = &lt;lambda&gt; at 0x276a5f0&gt; <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span>Traceback (most recent call last): <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> File "/usr/lib/python2.7/site-<wbr = class=3D"">packages/ovirtlago/testlib.py"<wbr class=3D"">, line 217, in = assert_equals_within <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> res =3D func() <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> File "/home/jenkins/workspace/<wbr = class=3D"">ovirt-system-tests_manual/<wbr = class=3D"">ovirt-system-tests/basic-<wbr = class=3D"">suite-master/test-scenarios/<wbr = class=3D"">004_basic_sanity.py", line 466, in &lt;lambda&gt; <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> api.vms.get(VM0_NAME).disks.<wbr = class=3D"">get(disk_name).status.state =3D=3D 'ok' <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span>AttributeError: 'NoneType' object has no = attribute 'state'</span> <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span><span style=3D"box-sizing: border-box; = color: rgb(205, 0, 0);" class=3D"">Error while running thread <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span>Traceback (most recent call last): <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> File "/usr/lib/python2.7/site-<wbr = class=3D"">packages/lago/utils.py", line 57, in _ret_via_queue <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> queue.put({'return': func()}) <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> File "/home/jenkins/workspace/<wbr = class=3D"">ovirt-system-tests_manual/<wbr = class=3D"">ovirt-system-tests/basic-<wbr = class=3D"">suite-master/test-scenarios/<wbr = class=3D"">004_basic_sanity.py", line 465, in snapshot_live_merge <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> lambda: <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> File "/usr/lib/python2.7/site-<wbr = class=3D"">packages/ovirtlago/testlib.py"<wbr class=3D"">, line 264, in = assert_true_within_long <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> assert_equals_within_long(<wbr = class=3D"">func, True, allowed_exceptions) <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> File "/usr/lib/python2.7/site-<wbr = class=3D"">packages/ovirtlago/testlib.py"<wbr class=3D"">, line 251, in = assert_equals_within_long <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> func, value, LONG_TIMEOUT, = allowed_exceptions=3Dallowed_<wbr class=3D"">exceptions <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> File "/usr/lib/python2.7/site-<wbr = class=3D"">packages/ovirtlago/testlib.py"<wbr class=3D"">, line 217, in = assert_equals_within <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> res =3D func() <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> File "/home/jenkins/workspace/<wbr = class=3D"">ovirt-system-tests_manual/<wbr = class=3D"">ovirt-system-tests/basic-<wbr = class=3D"">suite-master/test-scenarios/<wbr = class=3D"">004_basic_sanity.py", line 466, in &lt;lambda&gt; <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span> api.vms.get(VM0_NAME).disks.<wbr = class=3D"">get(disk_name).status.state =3D=3D 'ok' <span class=3D"gmail-m_-3711950224702553848inbox-inbox-timestamp" = style=3D"box-sizing: border-box;"><b style=3D"box-sizing: border-box;" = class=3D"">08:28:31</b> </span>AttributeError: 'NoneType' object has no = attribute 'state'</span> </pre><br = class=3D"gmail-m_-3711950224702553848inbox-inbox-Apple-interchange-newline= "></div></div></div><br class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D"">Devel mailing list<br = class=3D""><a href=3D"mailto:Devel@ovirt.org" = class=3D&quot;&quot;&gt;Devel(a)ovirt.org&lt;/a&gt;&lt;br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" rel=3D"noreferrer" = target=3D"_blank" class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""></blockquote></div><br= class=3D""></div></div><span style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: = none; display: inline !important;" = class=3D"">_______________________________________________</span><br = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><span style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = float: none; display: inline !important;" class=3D"">Devel mailing = list</span><br style=3D"font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><a = href=3D"mailto:Devel@ovirt.org" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px;" class=3D&quot;&quot;&gt;Devel(a)ovirt.org&lt;/a&gt;&lt;br = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" = class=3D"">http://lists.ovirt.org/mailman/listinfo/devel<... uote></div><br class=3D""></body></html>= --Apple-Mail=_99B42EE4-4513-4FE9-8F9A-A1E183AF4102--