<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_extra"><div class="gmail_quote">On Mon, Sep 16, 2013 at 5:41 PM, Omer Frenkel <span dir="ltr">&lt;<a href="mailto:ofrenkel@redhat.com" target="_blank">ofrenkel@redhat.com</a>&gt;</span> wrote:<br>

<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:&#39;times new roman&#39;,&#39;new york&#39;,times,serif">

<div><br></div><div><br></div><hr><blockquote style="padding-left:5px;font-size:12pt;font-style:normal;margin-left:5px;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal;border-left-width:2px;border-left-style:solid;border-left-color:rgb(16,16,255)">

<b>From: </b>&quot;Andrew Lau&quot; &lt;<a href="mailto:andrew@andrewklau.com" target="_blank">andrew@andrewklau.com</a>&gt;<br><b>To: </b>&quot;Omer Frenkel&quot; &lt;<a href="mailto:ofrenkel@redhat.com" target="_blank">ofrenkel@redhat.com</a>&gt;<br>

<b>Cc: </b>&quot;Dan Kenigsberg&quot; &lt;<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>&gt;, <a href="mailto:libvir-list@redhat.com" target="_blank">libvir-list@redhat.com</a>, &quot;users&quot; &lt;<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>&gt;<br>

<b>Sent: </b>Monday, September 16, 2013 1:38:53 AM<div><div class="h5"><br><b>Subject: </b>Re: [Users] Live Migration failed oVirt 3.3 Nightly<br><div><br></div><div dir="ltr"><div style="font-family:tahoma,sans-serif"><span style="font-family:arial">On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel </span><span dir="ltr" style="font-family:arial">&lt;<a href="mailto:ofrenkel@redhat.com" target="_blank">ofrenkel@redhat.com</a>&gt;</span><span style="font-family:arial"> wrote:</span><br>

</div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>

<div><br> <br> ----- Original Message -----<br> &gt; From: &quot;Dan Kenigsberg&quot; &lt;<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>&gt;<br> &gt; To: &quot;Andrew Lau&quot; &lt;<a href="mailto:andrew@andrewklau.com" target="_blank">andrew@andrewklau.com</a>&gt;<br>

 &gt; Cc: <a href="mailto:libvir-list@redhat.com" target="_blank">libvir-list@redhat.com</a>, &quot;users&quot; &lt;<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>&gt;<br> &gt; Sent: Sunday, September 15, 2013 3:47:03 PM<br>

 &gt; Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly<br> &gt;<br> &gt; On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:<br> &gt; &gt; On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg &lt;<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>&gt; wrote:<br>

 &gt; &gt;<br> &gt; &gt; &gt; On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:<br> &gt; &gt; &gt; &gt; On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg &lt;<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>&gt;<br>

 &gt; &gt; &gt; wrote:<br> &gt; &gt; &gt; &gt;<br> &gt; &gt; &gt; &gt; &gt; On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:<br> &gt; &gt; &gt; &gt; &gt; &gt; Hi Dan,<br> &gt; &gt; &gt; &gt; &gt; &gt;<br> &gt; &gt; &gt; &gt; &gt; &gt; Certainly, I&#39;ve uploaded them to fedora&#39;s paste bin and tried to<br>

 &gt; &gt; &gt; &gt; &gt; &gt; snip<br> &gt; &gt; &gt; &gt; &gt; just<br> &gt; &gt; &gt; &gt; &gt; &gt; the relevant details.<br> &gt; &gt; &gt; &gt; &gt; &gt;<br> &gt; &gt; &gt; &gt; &gt; &gt; Sender (<a href="http://hv01.melb.domain.net" target="_blank">hv01.melb.domain.net</a>):<br>

 &gt; &gt; &gt; &gt; &gt; &gt; <a href="http://paste.fedoraproject.org/39660/92339651/" target="_blank">http://paste.fedoraproject.org/39660/92339651/</a><br> &gt; &gt; &gt; &gt; &gt;<br> &gt; &gt; &gt; &gt; &gt; This one has<br>

 &gt; &gt; &gt; &gt; &gt;<br> &gt; &gt; &gt; &gt; &gt;     libvirtError: operation failed: Failed to connect to remote<br> &gt; &gt; &gt; &gt; &gt;     libvirt<br> &gt; &gt; &gt; &gt; &gt; URI qemu+tls://<a href="http://hv02.melb.domain.net/system" target="_blank">hv02.melb.domain.net/system</a><br>

 &gt; &gt; &gt; &gt; &gt;<br> &gt; &gt; &gt; &gt; &gt; which is most often related to firewall issues, and some time to key<br> &gt; &gt; &gt; &gt; &gt; mismatch.<br> &gt; &gt; &gt; &gt; &gt;<br> &gt; &gt; &gt; &gt; &gt; Does<br>

 &gt; &gt; &gt; &gt; &gt;     virsh -c qemu+tls://<a href="http://hv02.melb.domain.net/system" target="_blank">hv02.melb.domain.net/system</a> capabilities<br> &gt; &gt; &gt; &gt; &gt; work when run from the command line of hv01?<br>

 &gt; &gt; &gt; &gt; &gt;<br> &gt; &gt; &gt; &gt; &gt; Dan.<br> &gt; &gt; &gt; &gt; &gt; &gt; Receiver (<a href="http://hv02.melb.domain.net" target="_blank">hv02.melb.domain.net</a>): `<br> &gt; &gt; &gt; &gt; &gt; &gt; <a href="http://paste.fedoraproject.org/39661/23406913/" target="_blank">http://paste.fedoraproject.org/39661/23406913/</a><br>

 &gt; &gt; &gt; &gt; &gt; &gt;<br> &gt; &gt; &gt; &gt; &gt; &gt; VM being transfered is ovirt_guest_vm<br> &gt; &gt; &gt; &gt; &gt; &gt;<br> &gt; &gt; &gt; &gt; &gt; &gt; Thanks,<br> &gt; &gt; &gt; &gt; &gt; &gt; Andrew<br>

 &gt; &gt; &gt; &gt; &gt;<br> &gt; &gt; &gt; &gt;<br> &gt; &gt; &gt; &gt; virsh -c qemu+tls://<a href="http://hv02.melb.domain.net/system" target="_blank">hv02.melb.domain.net/system</a><br> &gt; &gt; &gt; &gt; 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: 0.10.2,<br>

 &gt; &gt; &gt; &gt; package: 18.el6_4.9 (CentOS BuildSystem &lt;<a href="http://bugs.centos.org" target="_blank">http://bugs.centos.org</a>&gt;,<br> &gt; &gt; &gt; &gt; 2013-07-02-11:19:29, <a href="http://c6b8.bsys.dev.centos.org" target="_blank">c6b8.bsys.dev.centos.org</a>)<br>

 &gt; &gt; &gt; &gt; 2013-09-15 10:41:10.620+0000: 23994: warning :<br> &gt; &gt; &gt; &gt; virNetTLSContextCheckCertificate:1102 : Certificate check failed<br> &gt; &gt; &gt; &gt; Certificate failed validation: The certificate hasn&#39;t got a known<br>

 &gt; &gt; &gt; &gt; issuer.<br> &gt; &gt; &gt;<br> &gt; &gt; &gt; Would you share your<br> &gt; &gt; &gt;<br> &gt; &gt; &gt;<br> &gt; &gt; &gt; openssl x509 -in<br> &gt; &gt; &gt; /etc/pki/vdsm/certs/cacert.pem -text<br>

 &gt; &gt; &gt;<br> &gt; &gt; &gt; openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text<br> &gt; &gt; &gt;<br> &gt; &gt; &gt; on both hosts? This content may be sensitive, and may not<br> &gt; &gt; &gt; provide an answer why libvirt on src cannot contact libvirtd on the<br>

 &gt; &gt; &gt; other host. So before you do that, would you test if<br> &gt; &gt; &gt;<br> &gt; &gt; &gt;<br> &gt; &gt; &gt;   vdsClient -s <a href="http://hv02.melb.domain.net" target="_blank">hv02.melb.domain.net</a> getVdsCapabilities<br>

 &gt; &gt; &gt;<br> &gt; &gt; &gt; works when run on hv01? It may be that the certificates are fine, but<br> &gt; &gt; &gt; libvirt is not configured to use the correct ones.<br> &gt; &gt; &gt;<br> &gt; &gt; &gt; Dan.<br>

 &gt; &gt; &gt;<br> &gt; &gt; &gt;<br> &gt; &gt; vdsClient -s <a href="http://hv02.melb.domain.net" target="_blank">hv02.melb.domain.net</a> getVdsCapabilities runs fine<br> &gt; &gt;<br> &gt; &gt; I did a quick comparison between the files on both hosts, they seem to have<br>

 &gt; &gt; the right details (host names, authority etc.)<br> &gt; &gt; cacert.pem matches<br> &gt; &gt;<br> &gt; &gt; /etc/libvirt/libvirtd.conf<br> &gt; &gt;<br> &gt; &gt; ca_file=&quot;/etc/pki/vdsm/certs/cacert.pem&quot;<br>

 &gt; &gt; cert_file=&quot;/etc/pki/vdsm/certs/vdsmcert.pem&quot;<br> &gt; &gt; key_file=&quot;/etc/pki/vdsm/keys/vdsmkey.pem&quot;<br> &gt;<br> <br></div></div>this sounds a little like<br> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=996146" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=996146</a><br>

 <br> can you try to restart libvirt (on both hosts just to be sure) and try again?<br><div><br> &gt; Maybe someone on libvir-list could guess why this could be happening?<br></div>&gt; _______________________________________________<br>

 &gt; Users mailing list<br> &gt; <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br> &gt; <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>

 &gt;<br></blockquote></div><br></div><div class="gmail_extra"><div style="font-family:tahoma,sans-serif">I did try that already</div><div style="font-family:tahoma,sans-serif"><br></div><div style="font-family:tahoma,sans-serif">

service vdsmd restart</div><div style="font-family:tahoma,sans-serif"><br></div><div><div><span style="font-family:tahoma,sans-serif" face="tahoma, sans-serif">[root@hv02 ~]# service vdsmd restart</span></div><div><span style="font-family:tahoma,sans-serif" face="tahoma, sans-serif">Shutting down vdsm daemon: </span></div>

<div><span style="font-family:tahoma,sans-serif" face="tahoma, sans-serif">vdsm watchdog stop                                         [  OK  ]</span></div><div><span style="font-family:tahoma,sans-serif" face="tahoma, sans-serif">vdsm stop                                                  [  OK  ]</span></div>

<div><span style="font-family:tahoma,sans-serif" face="tahoma, sans-serif">Starting configure libvirt to VDSM ...</span></div><div><span style="font-family:tahoma,sans-serif" face="tahoma, sans-serif">libvirt is already configured for vdsm</span></div>

<div><span style="font-family:tahoma,sans-serif" face="tahoma, sans-serif">=Done configuring libvirt=</span></div><div><span style="font-family:tahoma,sans-serif" face="tahoma, sans-serif">libvir: Network Filter Driver error : Requested operation is not valid: nwfilter is in use</span></div>

<div><span style="font-family:tahoma,sans-serif" face="tahoma, sans-serif">Checking conflicts ...</span></div><div><span style="font-family:tahoma,sans-serif" face="tahoma, sans-serif">SUCCESS: ssl configured to true. No conflicts</span></div>

<div><span style="font-family:tahoma,sans-serif" face="tahoma, sans-serif">Starting up vdsm daemon: </span></div><div><span style="font-family:tahoma,sans-serif" face="tahoma, sans-serif">vdsm start                                                 [  OK  ]</span></div>

<div style="font-family:tahoma,sans-serif"><br></div><div style="font-family:tahoma,sans-serif"><br></div><div style="font-family:tahoma,sans-serif">Migration still failed. Keep in mind, when I had oVirt 3.3 on these nodes migration was working fine. Only when I upgraded to the nightly and it picked up the new vdsm packages it started to fail.</div>

<div style="font-family:tahoma,sans-serif"><br></div></div><br></div></div></div></div></blockquote><div>can you try to restart the libvirtd service itself? not vdsm<br></div></div></div></blockquote><div><br></div><div>
<br>
</div><div class="gmail_default"><div class="gmail_default"><font face="tahoma, sans-serif">[root@hv01 ~]# service libvirtd restart</font></div><div class="gmail_default"><font face="tahoma, sans-serif">Stopping libvirtd daemon: libvirtd: libvirtd is managed by upstart and started, use initctl instead</font></div>

<div class="gmail_default"><font face="tahoma, sans-serif">[root@hv01 ~]# initctl restart libvirtd</font></div><div class="gmail_default"><font face="tahoma, sans-serif">libvirtd start/running, process 4538</font></div><div style="font-family:tahoma,sans-serif">

<br></div><div style="font-family:tahoma,sans-serif"><br></div><div style="font-family:tahoma,sans-serif">Migration was successful, thanks!</div></div></div><br></div></div>