<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 11, 2018 at 12:45 PM, Alona Kaplan <span dir="ltr"><<a href="mailto:alkaplan@redhat.com" target="_blank">alkaplan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-">On Tue, Apr 10, 2018 at 6:52 PM, Gal Ben Haim <span dir="ltr"><<a href="mailto:gbenhaim@redhat.com" target="_blank">gbenhaim@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">I'm seeing the same error in [1], during <span style="box-sizing:border-box;color:rgb(51,51,51);font-family:Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold">006_migrations.migrate_<wbr>vm.</span><div><br></div><div>[1] <a href="http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1650/" target="_blank">http://jenkins.ovirt.org/job/o<wbr>virt-4.2_change-queue-tester/1<wbr>650/</a></div></div></blockquote><div><span style="color:rgb(0,0,0)"><br></span></div></span><div><span style="color:rgb(0,0,0)">Seems like another bug.<font size="2"> The migration failed since for </font>some </span>reason the vm is already defined on the destination host.<br></div><div><br></div><div><span style="font-weight:bold;color:rgb(204,0,0)"></span>2018-04-10 11:08:08,685-0400 ERROR (jsonrpc/0) [api] FINISH create error=Virtual machine already exists (api:129)<br>Traceback (most recent call last):<br><span style="font-weight:bold;color:rgb(204,0,0)"></span>File "/usr/lib/python2.7/site-<wbr>packages/vdsm/common/api.py", line 122, in method<br><span style="font-weight:bold;color:rgb(204,0,0)"></span>ret = func(*args, **kwargs)<br><span style="font-weight:bold;color:rgb(204,0,0)"></span>File "/usr/lib/python2.7/site-<wbr>packages/vdsm/API.py", line 191, in create<br>raise exception.VMExists()<br><span style="font-weight:bold;color:rgb(204,0,0)"></span>VMExists: Virtual machine already exists<br><br></div></div></div></div></blockquote><div><br></div><div>Milan, Francesco, could it be that because of [1] that appears on the destination host right after shutting down the VM, it remained defined on that host?</div><div> </div><div>[1] <span style="color:rgb(0,0,0);font-family:Menlo;font-size:11px">2018-04-10 11:01:40,005-0400 ERROR (libvirt/events) [vds] Error running VM callback (clientIF:683)</span></div>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Traceback (most recent call last):</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 646, in dispatchLibvirtEvents</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> v.onLibvirtLifecycleEvent(event, detail, None)</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">AttributeError: 'NoneType' object has no attribute 'onLibvirtLifecycleEvent'</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div><div class="gmail-h5"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><span>On Tue, Apr 10, 2018 at 4:14 PM, Alona Kaplan <span dir="ltr"><<a href="mailto:alkaplan@redhat.com" target="_blank">alkaplan@redhat.com</a>></span> wrote:<br></span><div><div class="gmail-m_-4522529395583031909h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi all,</div><div><br></div><div>Looking at the log it seems that the new GetCapabilitiesAsync is responsible for the mess.<br></div><pre><pre>-<b> 08:29:47 - engine loses connectivity to host 'lago-basic-suite-4-2-host-0'.<br></b></pre><pre><b>- Every 3 seconds a getCapabalititiesAsync request is sent to the host (unsuccessfully).<br><br></b></pre><pre> * before each "getCapabilitiesAsync" the monitoring lock is taken (VdsManager,refreshImpl)<br></pre><pre> * "getCapabilitiesAsync" immediately fails and throws 'VDSNetworkException: java.net.ConnectException: Connection refused'. The exception is caught by 'GetCapabilitiesAsyncVDSComman<wbr>d.executeVdsBrokerCommand' which calls 'onFailure' of the callback and re-throws the exception.<br><br> catch (Throwable t) {<br> getParameters().getCallback().<wbr>onFailure(t);<br> throw t;<br> }<br></pre><pre> * The 'onFailure' of the callback releases the "monitoringLock" ('postProcessRefresh()->afterR<wbr>efreshTreatment()-> if (!succeeded) lockManager.releaseLock(monito<wbr>ringLock);')<br></pre><pre> * 'VdsManager,refreshImpl' catches the network exception, marks 'releaseLock = true' and <span style="color:rgb(255,0,0)"><b>tries to release the already released lock</b>.</span><br></pre><pre> The following warning is printed to the log - <pre> WARN [org.ovirt.engine.core.bll.loc<wbr>k.InMemoryLockManager] (EE-ManagedThreadFactory-engin<wbr>eScheduled-Thread-53) [] Trying to release exclusive lock which does not exist, lock key: 'ecf53d69-eb68-4b11-8df2-c4aa4<wbr>e19bd93VDS_INIT'</pre><b><br></b></pre><pre><b>- 08:30:51 a successful getCapabilitiesAsync is sent.<br><br></b></pre><pre><b>- 08:32:55 - The failing test starts (Setup Networks for setting ipv6).<br> </b><br></pre><pre> * SetupNetworks takes the monitoring lock.<br><br><pre><pre><b>- 08:33:00 - ResponseTracker cleans the getCapabilitiesAsync requests from 4 minutes ago from its queue and prints a VDSNetworkException: Vds timeout occured.</b><br></pre><pre> * When the first request is removed from the queue ('ResponseTracker.remove()'), the <span style="color:rgb(255,0,0)"><b>'Callback.onFailure' is invoked (for the second time) -> monitoring lock is released (the lock taken by the SetupNetworks!).<br></b></span></pre><pre> *<span style="color:rgb(255,0,0)"> <b>The other requests removed from the queue also try to release the monitoring lock</b></span>, but there is nothing to release.<br><br> * The following warning log is printed - <br> WARN [org.ovirt.engine.core.bll.loc<wbr>k.InMemoryLockManager] (EE-ManagedThreadFactory-engin<wbr>eScheduled-Thread-14) [] Trying to release exclusive lock which does not exist, lock key: 'ecf53d69-eb68-4b11-8df2-c4aa4<wbr>e19bd93VDS_INIT' <br></pre><pre><b></b></pre><pre>- <b>08:33:00 - SetupNetwork fails on Timeout ~4 seconds after is started</b>. Why? I'm not 100% sure but I guess the late processing of the 'getCapabilitiesAsync' that causes losing of the monitoring lock and the late + mupltiple processing of failure is root cause.<br><br><br></pre><pre>Ravi, 'getCapabilitiesAsync' failure is treated twice and the lock is trying to be released three times. Please share your opinion regarding how it should be fixed.<br><br><br></pre><pre>Thanks,<br></pre><pre>Alona.<br></pre><pre><b></b></pre></pre></pre><pre><br></pre><pre><b></b><br><br></pre><pre><br></pre></pre><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-m_-4522529395583031909m_6800077040582253067h5">On Sun, Apr 8, 2018 at 1:21 PM, Dan Kenigsberg <span dir="ltr"><<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_-4522529395583031909m_6800077040582253067h5"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-m_-4522529395583031909m_6800077040582253067m_-9018820350375279086gmail-">On Sun, Apr 8, 2018 at 9:21 AM, Edward Haas <span dir="ltr"><<a href="mailto:ehaas@redhat.com" target="_blank">ehaas@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Sun, Apr 8, 2018 at 9:15 AM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Was already done by Yaniv - <a href="https://gerrit.ovirt.org/#/c/89851" target="_blank">https://gerrit.ovirt.org/#/c<wbr>/89851</a>.<div>Is it still failing? </div></div><div class="gmail_extra"><div><div class="gmail-m_-4522529395583031909m_6800077040582253067m_-9018820350375279086gmail-m_2954871527889988252m_-7799404494298195144h5"><br><div class="gmail_quote">On Sun, Apr 8, 2018 at 8:59 AM, Barak Korren <span dir="ltr"><<a href="mailto:bkorren@redhat.com" target="_blank">bkorren@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><span>On 7 April 2018 at 00:30, Dan Kenigsberg <<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>> wrote:<br>
> No, I am afraid that we have not managed to understand why setting and<br>
> ipv6 address too the host off the grid. We shall continue researching<br>
> this next week.<br>
><br>
> Edy, <a href="https://gerrit.ovirt.org/#/c/88637/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/8<wbr>8637/</a> is already 4 weeks old, but<br>
> could it possibly be related (I really doubt that)?<br>
><br></span></blockquote></div></div></div></div></blockquote><div><br></div></span><div>Sorry, but I do not see how this problem is related to VDSM.<br></div><div>There is nothing that indicates that there is a VDSM problem.<br><br></div><div>Has the RPC connection between Engine and VDSM failed?<br></div><br></div></div></div></blockquote></span><div><br>Further up the thread, Piotr noticed that (at least on one failure of this test) that the Vdsm host lost connectivity to its storage, and Vdsm process was restarted. However, this does not seems to happen in all cases where this test fails.<br></div></div></div></div>
<br></div></div><span>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br></span></blockquote></div><br></div></div>
<br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br></blockquote></div></div></div><br><br clear="all"><span><div><br></div>-- <br><div class="gmail-m_-4522529395583031909m_6800077040582253067gmail_signature"><div dir="ltr"><font size="2" face="overpass, sans-serif" color="#000000"><span style="text-transform:uppercase"><b>GAL bEN HAIM</b></span></font><div><span style="color:rgb(0,0,0);font-family:overpass,sans-serif;text-transform:uppercase"><font size="2">RHV DEVOPS</font></span><br></div></div></div>
</span></div>
</blockquote></div></div></div><br></div></div>
<br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br></blockquote></div><br></div></div>