--Apple-Mail=_709B8D33-A1EA-4482-9A7A-2C59BF76F652 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8
On 12 Apr 2018, at 17:15, Dafna Ron <dron@redhat.com> wrote: =20 hi, =20 we are failing randomly on test 006_migrations.migrate_vm with what = seems to be the same issue.=20 =20 the vm seems to be migrated successfully but engine thinks that it = failed and re-calls migration getting a response of vm already exists.=20=
=20 I don't think this is an issue with the test but rather a regression = so I opened a bug:=20
=20 https://bugzilla.redhat.com/show_bug.cgi?id=3D1566594 = <https://bugzilla.redhat.com/show_bug.cgi?id=3D1566594> =20 Thanks,=20 Dafna =20 =20 On Wed, Apr 11, 2018 at 1:52 PM, Milan Zamazal <mzamazal@redhat.com = <mailto:mzamazal@redhat.com>> wrote: Arik Hadas <ahadas@redhat.com <mailto:ahadas@redhat.com>> writes: =20
On Wed, Apr 11, 2018 at 12:45 PM, Alona Kaplan <alkaplan@redhat.com = <mailto:alkaplan@redhat.com>> wrote:
On Tue, Apr 10, 2018 at 6:52 PM, Gal Ben Haim <gbenhaim@redhat.com =
<mailto:gbenhaim@redhat.com>> wrote:
I'm seeing the same error in [1], during =
006_migrations.migrate_vm.
[1] =
http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1650/ = <http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1650/>
Seems like another bug. The migration failed since for some reason =
is already defined on the destination host.
2018-04-10 11:08:08,685-0400 ERROR (jsonrpc/0) [api] FINISH create error=3DVirtual machine already exists (api:129) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line = 122, in method ret =3D func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 191, in = create raise exception.VMExists() VMExists: Virtual machine already exists
Milan, Francesco, could it be that because of [1] that appears on =
destination host right after shutting down the VM, it remained = defined on that host? =20 I can't see any destroy call in the logs after the successful =
migration from the given host. That would explain =E2=80=9CVMExists=E2=80= =9D error. =20
[1] 2018-04-10 11:01:40,005-0400 ERROR (libvirt/events) [vds] Error = running VM callback (clientIF:683)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line = 646, in dispatchLibvirtEvents
v.onLibvirtLifecycleEvent(event, detail, None)
AttributeError: 'NoneType' object has no attribute = 'onLibvirtLifecycleEvent' =20 That means that a life cycle event on an unknown VM has arrived, in =
case apparently destroy event, following the destroy call after the failed incoming migration. The reported AttributeError is a minor = bug, already fixed in master. So it's most likely unrelated to the = discussed problem. =20
On Tue, Apr 10, 2018 at 4:14 PM, Alona Kaplan <alkaplan@redhat.com = <mailto:alkaplan@redhat.com>> wrote:
Hi all,
Looking at the log it seems that the new GetCapabilitiesAsync is responsible for the mess.
- * 08:29:47 - engine loses connectivity to host = 'lago-basic-suite-4-2-host-0'.*
*- Every 3 seconds a getCapabalititiesAsync request is sent to =
i do not think so, I=E2=80=99ve heard someone removed a test in between = migrating A->B and B->A? If that=E2=80=99s the case that is the real issue.=20 You can=E2=80=99t migrate back to A without waiting for A to be cleared = out properly https://gerrit.ovirt.org/#/c/90166/ = <https://gerrit.ovirt.org/#/c/90166/> should fix it the vm the preceding this the host (unsuccessfully).*
* before each "getCapabilitiesAsync" the monitoring lock is =
taken (VdsManager,refreshImpl)
* "getCapabilitiesAsync" immediately fails and throws =
'VDSNetworkException: java.net.ConnectException: Connection refused'. = The exception is caught by = 'GetCapabilitiesAsyncVDSCommand.executeVdsBrokerCommand' which calls = 'onFailure' of the callback and re-throws the exception.
catch (Throwable t) { getParameters().getCallback().onFailure(t); throw t; }
* The 'onFailure' of the callback releases the =
"monitoringLock" ('postProcessRefresh()->afterRefreshTreatment()-> if = (!succeeded) lockManager.releaseLock(monitoringLock);')
* 'VdsManager,refreshImpl' catches the network exception, =
marks 'releaseLock =3D true' and *tries to release the already released = lock*.
The following warning is printed to the log -
WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] =
(EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Trying to release = exclusive lock which does not exist, lock key: = 'ecf53d69-eb68-4b11-8df2-c4aa4e19bd93VDS_INIT'
*- 08:30:51 a successful getCapabilitiesAsync is sent.*
*- 08:32:55 - The failing test starts (Setup Networks for setting =
ipv6). *
* SetupNetworks takes the monitoring lock.
*- 08:33:00 - ResponseTracker cleans the getCapabilitiesAsync =
requests from 4 minutes ago from its queue and prints a = VDSNetworkException: Vds timeout occured.*
* When the first request is removed from the queue =
('ResponseTracker.remove()'), the
*'Callback.onFailure' is invoked (for the second time) -> = monitoring lock is released (the lock taken by the SetupNetworks!).*
* *The other requests removed from the queue also try to = release the monitoring lock*, but there is nothing to release.
* The following warning log is printed - WARN = [org.ovirt.engine.core.bll.lock.InMemoryLockManager] = (EE-ManagedThreadFactory-engineScheduled-Thread-14) [] Trying to release = exclusive lock which does not exist, lock key: = 'ecf53d69-eb68-4b11-8df2-c4aa4e19bd93VDS_INIT'
- *08:33:00 - SetupNetwork fails on Timeout ~4 seconds after is = started*. Why? I'm not 100% sure but I guess the late processing of the = 'getCapabilitiesAsync' that causes losing of the monitoring lock and the = late + mupltiple processing of failure is root cause.
Ravi, 'getCapabilitiesAsync' failure is treated twice and the = lock is trying to be released three times. Please share your opinion = regarding how it should be fixed.
Thanks,
Alona.
On Sun, Apr 8, 2018 at 1:21 PM, Dan Kenigsberg <danken@redhat.com = <mailto:danken@redhat.com>> wrote:
On Sun, Apr 8, 2018 at 9:21 AM, Edward Haas <ehaas@redhat.com = <mailto:ehaas@redhat.com>> wrote:
> > > On Sun, Apr 8, 2018 at 9:15 AM, Eyal Edri <eedri@redhat.com = <mailto:eedri@redhat.com>> wrote: > >> Was already done by Yaniv - https://gerrit.ovirt.org/#/c/89851 = <https://gerrit.ovirt.org/#/c/89851>. >> Is it still failing? >> >> On Sun, Apr 8, 2018 at 8:59 AM, Barak Korren = <bkorren@redhat.com <mailto:bkorren@redhat.com>> >> wrote: >> >>> On 7 April 2018 at 00:30, Dan Kenigsberg <danken@redhat.com = <mailto:danken@redhat.com>> wrote: >>> > No, I am afraid that we have not managed to understand why = setting >>> and >>> > ipv6 address too the host off the grid. We shall continue >>> researching >>> > this next week. >>> > >>> > Edy, https://gerrit.ovirt.org/#/c/88637/ = <https://gerrit.ovirt.org/#/c/88637/> is already 4 weeks old, >>> but >>> > could it possibly be related (I really doubt that)? >>> > >>> >> > Sorry, but I do not see how this problem is related to VDSM. > There is nothing that indicates that there is a VDSM problem. > > Has the RPC connection between Engine and VDSM failed? > > Further up the thread, Piotr noticed that (at least on one = failure of this test) that the Vdsm host lost connectivity to its storage, = and Vdsm process was restarted. However, this does not seems to happen in = all cases where this test fails.
_______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel>
_______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel>
-- *GAL bEN HAIM* RHV DEVOPS
_______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel>
Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--Apple-Mail=_709B8D33-A1EA-4482-9A7A-2C59BF76F652 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><br = class=3D""><div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On 12 Apr 2018, at 17:15, Dafna Ron <<a = href=3D"mailto:dron@redhat.com" class=3D"">dron@redhat.com</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D"">hi<span style=3D"font-weight:bold" = class=3D"">,<br class=3D""><br class=3D""></span></div><div class=3D"">we = are failing randomly on test 006_migrations.migrate_vm with what seems = to be the same issue. <br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D"">the vm seems to be migrated = successfully but engine thinks that it failed and re-calls migration = getting a response of vm already exists. <br class=3D""><br = class=3D""></div><div class=3D"">I don't think this is an issue with the = test but rather a regression so I opened a bug: <br = class=3D""></div></div></div></blockquote><div><br class=3D""></div>i do = not think so, I=E2=80=99ve heard someone removed a test in between = migrating A->B and B->A?</div><div>If that=E2=80=99s the case that = is the real issue. </div><div>You can=E2=80=99t migrate back to A = without waiting for A to be cleared out properly</div><div><a = href=3D"https://gerrit.ovirt.org/#/c/90166/" = class=3D"">https://gerrit.ovirt.org/#/c/90166/</a> should fix = it</div><div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D""><div dir=3D"ltr" class=3D""><div class=3D""><br class=3D""><a = href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1566594" = class=3D"">https://bugzilla.redhat.com/show_bug.cgi?id=3D1566594</a><br = class=3D""><br class=3D""></div><div class=3D"">Thanks, <br = class=3D""></div><div class=3D"">Dafna<br class=3D""><span = style=3D"font-weight:bold" class=3D""><span style=3D"font-weight:bold" = class=3D""><br class=3D""></span></span></div><span = style=3D"font-weight:bold" class=3D""></span></div><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On Wed, = Apr 11, 2018 at 1:52 PM, Milan Zamazal <span dir=3D"ltr" class=3D""><<a= href=3D"mailto:mzamazal@redhat.com" target=3D"_blank" = class=3D"">mzamazal@redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=3D"">Arik = Hadas <<a href=3D"mailto:ahadas@redhat.com" = class=3D"">ahadas@redhat.com</a>> writes:<br class=3D""> <br class=3D""> > On Wed, Apr 11, 2018 at 12:45 PM, Alona Kaplan <<a = href=3D"mailto:alkaplan@redhat.com" class=3D"">alkaplan@redhat.com</a>>= wrote:<br class=3D""> ><br class=3D""> >><br class=3D""> >><br class=3D""> >> On Tue, Apr 10, 2018 at 6:52 PM, Gal Ben Haim <<a = href=3D"mailto:gbenhaim@redhat.com" class=3D"">gbenhaim@redhat.com</a>>= wrote:<br class=3D""> >><br class=3D""> >>> I'm seeing the same error in [1], during = 006_migrations.migrate_vm.<br class=3D""> >>><br class=3D""> >>> [1] <a = href=3D"http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1650/" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://jenkins.ovirt.org/job/<wbr = class=3D"">ovirt-4.2_change-queue-tester/<wbr class=3D"">1650/</a><br = class=3D""> >>><br class=3D""> >><br class=3D""> >> Seems like another bug. The migration failed since for some = reason the vm<br class=3D""> >> is already defined on the destination host.<br class=3D""> >><br class=3D""> >> 2018-04-10 11:08:08,685-0400 ERROR (jsonrpc/0) [api] FINISH = create<br class=3D""> >> error=3DVirtual machine already exists (api:129)<br class=3D""> >> Traceback (most recent call last):<br class=3D""> >> File "/usr/lib/python2.7/site-<wbr = class=3D"">packages/vdsm/common/api.py", line 122, in<br class=3D""> >> method<br class=3D""> >> ret =3D func(*args, **kwargs)<br class=3D""> >> File "/usr/lib/python2.7/site-<wbr = class=3D"">packages/vdsm/API.py", line 191, in create<br class=3D""> >> raise exception.VMExists()<br class=3D""> >> VMExists: Virtual machine already exists<br class=3D""> >><br class=3D""> >><br class=3D""> > Milan, Francesco, could it be that because of [1] that appears on = the<br class=3D""> > destination host right after shutting down the VM, it remained = defined on<br class=3D""> > that host?<br class=3D""> <br class=3D""> </span>I can't see any destroy call in the logs after the successful = preceding<br class=3D""> migration from the given host. That would explain =E2=80=9CVMExists=E2= =80=9D error.<br class=3D""> <span class=3D""><br class=3D""> > [1] 2018-04-10 11:01:40,005-0400 ERROR (libvirt/events) [vds] Error = running<br class=3D""> > VM callback (clientIF:683)<br class=3D""> ><br class=3D""> > Traceback (most recent call last):<br class=3D""> ><br class=3D""> > File "/usr/lib/python2.7/site-<wbr = class=3D"">packages/vdsm/clientIF.py", line 646, in<br class=3D""> > dispatchLibvirtEvents<br class=3D""> ><br class=3D""> > v.onLibvirtLifecycleEvent(<wbr class=3D"">event, = detail, None)<br class=3D""> ><br class=3D""> > AttributeError: 'NoneType' object has no attribute = 'onLibvirtLifecycleEvent'<br class=3D""> <br class=3D""> </span>That means that a life cycle event on an unknown VM has arrived, = in this<br class=3D""> case apparently destroy event, following the destroy call after the<br = class=3D""> failed incoming migration. The reported AttributeError is a minor = bug,<br class=3D""> already fixed in master. So it's most likely unrelated to the = discussed<br class=3D""> problem.<br class=3D""> <span class=3D""><br class=3D""> >>> On Tue, Apr 10, 2018 at 4:14 PM, Alona Kaplan <<a = href=3D"mailto:alkaplan@redhat.com" = class=3D"">alkaplan@redhat.com</a>><br class=3D""> >>> wrote:<br class=3D""> >>><br class=3D""> >>>> Hi all,<br class=3D""> >>>><br class=3D""> >>>> Looking at the log it seems that the new = GetCapabilitiesAsync is<br class=3D""> >>>> responsible for the mess.<br class=3D""> >>>><br class=3D""> >>>> -<br class=3D""> </span>>>>> * 08:29:47 - engine loses connectivity to host = 'lago-basic-suite-4-2-host-0'.<wbr class=3D"">*<br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>> *- Every 3 seconds a getCapabalititiesAsync request is = sent to the host (unsuccessfully).*<br class=3D""> <span class=3D"">>>>><br class=3D""> >>>> * before each = "getCapabilitiesAsync" the monitoring lock is taken = (VdsManager,refreshImpl)<br class=3D""> >>>><br class=3D""> >>>> * "getCapabilitiesAsync" = immediately fails and throws 'VDSNetworkException: = java.net.ConnectException: Connection refused'. The exception is caught = by '<wbr class=3D"">GetCapabilitiesAsyncVDSCommand<wbr = class=3D"">.executeVdsBrokerCommand' which calls 'onFailure' of the = callback and re-throws the exception.<br class=3D""> >>>><br class=3D""> >>>> catch (Throwable t) = {<br class=3D""> >>>> = getParameters().getCallback().<wbr class=3D"">onFailure(t);<br = class=3D""> >>>> throw = t;<br class=3D""> >>>> }<br class=3D""> >>>><br class=3D""> >>>> * The 'onFailure' of the callback = releases the "monitoringLock" ('postProcessRefresh()-><wbr = class=3D"">afterRefreshTreatment()-> if (!succeeded) = lockManager.releaseLock(<wbr class=3D"">monitoringLock);')<br class=3D""> >>>><br class=3D""> </span>>>>> * 'VdsManager,refreshImpl' = catches the network exception, marks 'releaseLock =3D true' and *tries = to release the already released lock*.<br class=3D""> <span class=3D"">>>>><br class=3D""> >>>> The following warning is = printed to the log -<br class=3D""> >>>><br class=3D""> >>>> WARN = [org.ovirt.engine.core.bll.<wbr class=3D"">lock.InMemoryLockManager] = (EE-ManagedThreadFactory-<wbr class=3D"">engineScheduled-Thread-53) [] = Trying to release exclusive lock which does not exist, lock key: = 'ecf53d69-eb68-4b11-8df2-<wbr class=3D"">c4aa4e19bd93VDS_INIT'<br = class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>><br class=3D""> </span>>>>> *- 08:30:51 a successful getCapabilitiesAsync is = sent.*<br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>> *- 08:32:55 - The failing test starts (Setup Networks = for setting ipv6). *<br class=3D""> <span class=3D"">>>>><br class=3D""> >>>> * SetupNetworks takes the monitoring = lock.<br class=3D""> >>>><br class=3D""> </span>>>>> *- 08:33:00 - ResponseTracker cleans the = getCapabilitiesAsync requests from 4 minutes ago from its queue and = prints a VDSNetworkException: Vds timeout occured.*<br class=3D""> <span class=3D"">>>>><br class=3D""> >>>> * When the first request is = removed from the queue ('ResponseTracker.remove()'), the<br class=3D""> </span>>>>> *'Callback.onFailure' is invoked (for the second = time) -> monitoring lock is released (the lock taken by the = SetupNetworks!).*<br class=3D""> >>>><br class=3D""> >>>> * *The other requests removed = from the queue also try to release the monitoring lock*, but there is = nothing to release.<br class=3D""> <span class=3D"">>>>><br class=3D""> >>>> * The following warning log = is printed -<br class=3D""> >>>> WARN = [org.ovirt.engine.core.bll.<wbr class=3D"">lock.InMemoryLockManager] = (EE-ManagedThreadFactory-<wbr class=3D"">engineScheduled-Thread-14) [] = Trying to release exclusive lock which does not exist, lock key: = 'ecf53d69-eb68-4b11-8df2-<wbr class=3D"">c4aa4e19bd93VDS_INIT'<br = class=3D""> >>>><br class=3D""> </span>>>>> - *08:33:00 - SetupNetwork fails on Timeout ~4 = seconds after is started*. Why? I'm not 100% sure but I guess the late = processing of the 'getCapabilitiesAsync' that causes losing of the = monitoring lock and the late + mupltiple processing of failure is root = cause.<br class=3D""> <div class=3D""><div class=3D"h5">>>>><br class=3D""> >>>><br class=3D""> >>>> Ravi, 'getCapabilitiesAsync' failure is treated twice = and the lock is trying to be released three times. Please share your = opinion regarding how it should be fixed.<br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>> Thanks,<br class=3D""> >>>><br class=3D""> >>>> Alona.<br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>> On Sun, Apr 8, 2018 at 1:21 PM, Dan Kenigsberg <<a = href=3D"mailto:danken@redhat.com" class=3D"">danken@redhat.com</a>><br = class=3D""> >>>> wrote:<br class=3D""> >>>><br class=3D""> >>>>> On Sun, Apr 8, 2018 at 9:21 AM, Edward Haas <<a = href=3D"mailto:ehaas@redhat.com" class=3D"">ehaas@redhat.com</a>> = wrote:<br class=3D""> >>>>><br class=3D""> >>>>>><br class=3D""> >>>>>><br class=3D""> >>>>>> On Sun, Apr 8, 2018 at 9:15 AM, Eyal Edri = <<a href=3D"mailto:eedri@redhat.com" = class=3D"">eedri@redhat.com</a>> wrote:<br class=3D""> >>>>>><br class=3D""> >>>>>>> Was already done by Yaniv - <a = href=3D"https://gerrit.ovirt.org/#/c/89851" rel=3D"noreferrer" = target=3D"_blank" class=3D"">https://gerrit.ovirt.org/#/c/<wbr = class=3D"">89851</a>.<br class=3D""> >>>>>>> Is it still failing?<br class=3D""> >>>>>>><br class=3D""> >>>>>>> On Sun, Apr 8, 2018 at 8:59 AM, Barak = Korren <<a href=3D"mailto:bkorren@redhat.com" = class=3D"">bkorren@redhat.com</a>><br class=3D""> >>>>>>> wrote:<br class=3D""> >>>>>>><br class=3D""> >>>>>>>> On 7 April 2018 at 00:30, Dan = Kenigsberg <<a href=3D"mailto:danken@redhat.com" = class=3D"">danken@redhat.com</a>> wrote:<br class=3D""> >>>>>>>> > No, I am afraid that we have not = managed to understand why setting<br class=3D""> >>>>>>>> and<br class=3D""> >>>>>>>> > ipv6 address too the host off the = grid. We shall continue<br class=3D""> >>>>>>>> researching<br class=3D""> >>>>>>>> > this next week.<br class=3D""> >>>>>>>> ><br class=3D""> >>>>>>>> > Edy, <a = href=3D"https://gerrit.ovirt.org/#/c/88637/" rel=3D"noreferrer" = target=3D"_blank" class=3D"">https://gerrit.ovirt.org/#/c/<wbr = class=3D"">88637/</a> is already 4 weeks old,<br class=3D""> >>>>>>>> but<br class=3D""> >>>>>>>> > could it possibly be related (I = really doubt that)?<br class=3D""> >>>>>>>> ><br class=3D""> >>>>>>>><br class=3D""> >>>>>>><br class=3D""> >>>>>> Sorry, but I do not see how this problem is = related to VDSM.<br class=3D""> >>>>>> There is nothing that indicates that there is a = VDSM problem.<br class=3D""> >>>>>><br class=3D""> >>>>>> Has the RPC connection between Engine and VDSM = failed?<br class=3D""> >>>>>><br class=3D""> >>>>>><br class=3D""> >>>>> Further up the thread, Piotr noticed that (at least = on one failure of<br class=3D""> >>>>> this test) that the Vdsm host lost connectivity to = its storage, and Vdsm<br class=3D""> >>>>> process was restarted. However, this does not seems = to happen in all cases<br class=3D""> >>>>> where this test fails.<br class=3D""> >>>>><br class=3D""> >>>>> ______________________________<wbr = class=3D"">_________________<br class=3D""> >>>>> Devel mailing list<br class=3D""> >>>>> <a href=3D"mailto:Devel@ovirt.org" = class=3D"">Devel@ovirt.org</a><br class=3D""> >>>>> <a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" rel=3D"noreferrer" = target=3D"_blank" class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""> >>>>><br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>> ______________________________<wbr = class=3D"">_________________<br class=3D""> >>>> Devel mailing list<br class=3D""> >>>> <a href=3D"mailto:Devel@ovirt.org" = class=3D"">Devel@ovirt.org</a><br class=3D""> >>>> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel"= rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""> >>>><br class=3D""> >>><br class=3D""> >>><br class=3D""> >>><br class=3D""> >>> --<br class=3D""> </div></div>>>> *GAL bEN HAIM*<br class=3D""> <span class=3D"im HOEnZb">>>> RHV DEVOPS<br class=3D""> >>><br class=3D""> >><br class=3D""> >><br class=3D""> >> ______________________________<wbr = class=3D"">_________________<br class=3D""> >> Devel mailing list<br class=3D""> >> <a href=3D"mailto:Devel@ovirt.org" = class=3D"">Devel@ovirt.org</a><br class=3D""> >> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""> >><br class=3D""> </span><div class=3D"HOEnZb"><div = class=3D"h5">______________________________<wbr = class=3D"">_________________<br class=3D""> Devel mailing list<br class=3D""> <a href=3D"mailto:Devel@ovirt.org" class=3D"">Devel@ovirt.org</a><br = class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a></div></div></blockquote></div><br = class=3D""></div> _______________________________________________<br class=3D"">Devel = mailing list<br class=3D""><a href=3D"mailto:Devel@ovirt.org" = class=3D"">Devel@ovirt.org</a><br = class=3D"">http://lists.ovirt.org/mailman/listinfo/devel</div></blockquote=
</div><br class=3D""></body></html>=
--Apple-Mail=_709B8D33-A1EA-4482-9A7A-2C59BF76F652--