Re: [Users] Live Migration failed oVirt 3.3 Nightly

16 Sep 2013

      ------=_Part_21768893_1437824270.1379317267664
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

----- Original Message -----
...
From: "Andrew Lau" <andrew@andrewklau.com>
To: "Omer Frenkel" <ofrenkel@redhat.com>
Cc: "Dan Kenigsberg" <danken@redhat.com>, libvir-list@redhat.com, "users"
<users@ovirt.org>
Sent: Monday, September 16, 2013 1:38:53 AM
Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
...
On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel < ofrenkel@redhat.com > wrote:
...
...
----- Original Message -----
...
...
From: "Dan Kenigsberg" < danken@redhat.com >
...
...
To: "Andrew Lau" < andrew@andrewklau.com >
...
...
Cc: libvir-list@redhat.com , "users" < users@ovirt.org >
...
...
Sent: Sunday, September 15, 2013 3:47:03 PM
...
...
Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
...
...
...
...
On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
...
...
...
On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg < danken@redhat.com >
wrote:
...
...
...
...
...
...
...
On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
...
...
...
...
...
On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg < danken@redhat.com
>
...
...
...
...
wrote:
...
...
...
...
...
...
...
...
...
...
> On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:
...
...
...
...
...
> > Hi Dan,
...
...
...
...
...
> >
...
...
...
...
...
> > Certainly, I've uploaded them to fedora's paste bin and tried
> > to
...
...
...
...
...
> > snip
...
...
...
...
...
> just
...
...
...
...
...
> > the relevant details.
...
...
...
...
...
> >
...
...
...
...
...
> > Sender ( hv01.melb.domain.net ):
...
...
...
...
...
> > http://paste.fedoraproject.org/39660/92339651/
...
...
...
...
...
>
...
...
...
...
...
> This one has
...
...
...
...
...
>
...
...
...
...
...
> libvirtError: operation failed: Failed to connect to remote
...
...
...
...
...
> libvirt
...
...
...
...
...
> URI qemu+tls:// hv02.melb.domain.net/system
...
...
...
...
...
>
...
...
...
...
...
> which is most often related to firewall issues, and some time to
> key
...
...
...
...
...
> mismatch.
...
...
...
...
...
>
...
...
...
...
...
> Does
...
...
...
...
...
> virsh -c qemu+tls:// hv02.melb.domain.net/system capabilities
...
...
...
...
...
> work when run from the command line of hv01?
...
...
...
...
...
>
...
...
...
...
...
> Dan.
...
...
...
...
...
> > Receiver ( hv02.melb.domain.net ): `
...
...
...
...
...
> > http://paste.fedoraproject.org/39661/23406913/
...
...
...
...
...
> >
...
...
...
...
...
> > VM being transfered is ovirt_guest_vm
...
...
...
...
...
> >
...
...
...
...
...
> > Thanks,
...
...
...
...
...
> > Andrew
...
...
...
...
...
>
...
...
...
...
...
...
...
...
...
...
virsh -c qemu+tls:// hv02.melb.domain.net/system
...
...
...
...
...
2013-09-15 10:41:10.620+0000: 23994: info : libvirt version:
0.10.2,
...
...
...
...
...
package: 18.el6_4.9 (CentOS BuildSystem < http://bugs.centos.org >,
...
...
...
...
...
2013-07-02-11:19:29, c6b8.bsys.dev.centos.org )
...
...
...
...
...
2013-09-15 10:41:10.620+0000: 23994: warning :
...
...
...
...
...
virNetTLSContextCheckCertificate:1102 : Certificate check failed
...
...
...
...
...
Certificate failed validation: The certificate hasn't got a known
...
...
...
...
...
issuer.
...
...
...
...
...
...
...
...
Would you share your
...
...
...
...
...
...
...
...
...
...
...
...
openssl x509 -in
...
...
...
...
/etc/pki/vdsm/certs/cacert.pem -text
...
...
...
...
...
...
...
...
openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
...
...
...
...
...
...
...
...
on both hosts? This content may be sensitive, and may not
...
...
...
...
provide an answer why libvirt on src cannot contact libvirtd on the
...
...
...
...
other host. So before you do that, would you test if
...
...
...
...
...
...
...
...
...
...
...
...
vdsClient -s hv02.melb.domain.net getVdsCapabilities
...
...
...
...
...
...
...
...
works when run on hv01? It may be that the certificates are fine, but
...
...
...
...
libvirt is not configured to use the correct ones.
...
...
...
...
...
...
...
...
Dan.
...
...
...
...
...
...
...
...
...
...
...
vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
...
...
...
...
...
...
I did a quick comparison between the files on both hosts, they seem to
have
...
...
...
the right details (host names, authority etc.)
...
...
...
cacert.pem matches
...
...
...
...
...
...
/etc/libvirt/libvirtd.conf
...
...
...
...
...
...
ca_file="/etc/pki/vdsm/certs/cacert.pem"
...
...
...
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
...
...
...
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
...
...

...
...
this sounds a little like
...
https://bugzilla.redhat.com/show_bug.cgi?id=996146

...
...
can you try to restart libvirt (on both hosts just to be sure) and try
again?

...
...
...
Maybe someone on libvir-list could guess why this could be happening?
...
...
_______________________________________________
...
...
Users mailing list
...
...
Users@ovirt.org
...
...
http://lists.ovirt.org/mailman/listinfo/users
...
...

...
I did try that already
...
service vdsmd restart
...
[root@hv02 ~]# service vdsmd restart
Shutting down vdsm daemon:
vdsm watchdog stop [ OK ]
vdsm stop [ OK ]
Starting configure libvirt to VDSM ...
libvirt is already configured for vdsm
=Done configuring libvirt=
libvir: Network Filter Driver error : Requested operation is not valid:
nwfilter is in use
Checking conflicts ...
SUCCESS: ssl configured to true. No conflicts
Starting up vdsm daemon:
vdsm start [ OK ]
...
Migration still failed. Keep in mind, when I had oVirt 3.3 on these nodes
migration was working fine. Only when I upgraded to the nightly and it
picked up the new vdsm packages it started to fail.
...
Sent: </b>Monday, September 16, 2013 1:38:53 AM<br><b>Subject: </b>Re: [Us=
ers] Live Migration failed oVirt 3.3 Nightly<br><div><br></div><div dir=3D"=
ltr"><div class=3D"gmail_default" style=3D"font-family:tahoma,sans-serif" d=
ata-mce-style=3D"font-family: tahoma,sans-serif;"><span style=3D"font-famil=
y:arial" data-mce-style=3D"font-family: arial;">On Sun, Sep 15, 2013 at 11:=
51 PM, Omer Frenkel </span><span dir=3D"ltr" style=3D"font-family:arial" da=
ta-mce-style=3D"font-family: arial;"><<a href=3D"mailto:ofrenkel@redhat.=
com" target=3D"_blank" data-mce-href=3D"mailto:ofrenkel@redhat.com">ofrenke=
l@redhat.com</a>></span><span style=3D"font-family:arial" data-mce-style=
=3D"font-family: arial;"> wrote:</span><br></div><div class=3D"gmail_extra"=
<div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,20=
4);border-left-style:solid;padding-left:1ex" data-mce-style=3D"margin: 0px =
0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-l=
eft-style: solid; padding-left: 1ex;"><div class=3D""><div class=3D"h5"><br=
<br> ----- Original Message -----<br> > From: "Dan Kenigsberg" <<a =
...
> > > > > > snip<br> > > > > > just<br> =
> > > > > > the relevant details.<br> > > > >=
 > ><br> > > > > > > Sender (<a href=3D"http://hv01=
.melb.domain.net" target=3D"_blank" data-mce-href=3D"http://hv01.melb.domai=
n.net">hv01.melb.domain.net</a>):<br> > > > > > > <a href=
=3D"http://paste.fedoraproject.org/39660/92339651/" target=3D"_blank" data-=
mce-href=3D"http://paste.fedoraproject.org/39660/92339651/">http://paste.fe=
doraproject.org/39660/92339651/</a><br> > > > > ><br> > &=
gt; > > > This one has<br> > > > > ><br> > > =
> > >     libvirtError: operation failed: Failed to conn=
ect to remote<br> > > > > >     libvirt<br> > &=
gt; > > > URI qemu+tls://<a href=3D"http://hv02.melb.domain.net/sy=
stem" target=3D"_blank" data-mce-href=3D"http://hv02.melb.domain.net/system=
">hv02.melb.domain.net/system</a><br> > > > > ><br> > >=
; > > > which is most often related to firewall issues, and some t=
ime to key<br> > > > > > mismatch.<br> > > > > &=
gt;<br> > > > > > Does<br> > > > > >   &=
nbsp; virsh -c qemu+tls://<a href=3D"http://hv02.melb.domain.net/system" ta=
rget=3D"_blank" data-mce-href=3D"http://hv02.melb.domain.net/system">hv02.m=
elb.domain.net/system</a> capabilities<br> > > > > > work wh=
en run from the command line of hv01?<br> > > > > ><br> >=
 > > > > Dan.<br> > > > > > > Receiver (<a hr=
ef=3D"http://hv02.melb.domain.net" target=3D"_blank" data-mce-href=3D"http:=
//hv02.melb.domain.net">hv02.melb.domain.net</a>): `<br> > > > >=
; > > <a href=3D"http://paste.fedoraproject.org/39661/23406913/" targ=
et=3D"_blank" data-mce-href=3D"http://paste.fedoraproject.org/39661/2340691=
3/">http://paste.fedoraproject.org/39661/23406913/</a><br> > > > &=
gt; > ><br> > > > > > > VM being transfered is ovir=
t_guest_vm<br> > > > > > ><br> > > > > > &=
gt; Thanks,<br> > > > > > > Andrew<br> > > > >=
; ><br> > > > ><br> > > > > virsh -c qemu+tls://=
<a href=3D"http://hv02.melb.domain.net/system" target=3D"_blank" data-mce-h=
ref=3D"http://hv02.melb.domain.net/system">hv02.melb.domain.net/system</a><=
br> > > > > 2013-09-15 10:41:10.620+0000: 23994: info : libvirt=
 version: 0.10.2,<br> > > > > package: 18.el6_4.9 (CentOS Build=
System <<a href=3D"http://bugs.centos.org" target=3D"_blank" data-mce-hr=
ef=3D"http://bugs.centos.org">http://bugs.centos.org</a>>,<br> > >=
 > > 2013-07-02-11:19:29, <a href=3D"http://c6b8.bsys.dev.centos.org"=
 target=3D"_blank" data-mce-href=3D"http://c6b8.bsys.dev.centos.org">c6b8.b=
sys.dev.centos.org</a>)<br> > > > > 2013-09-15 10:41:10.620+000=
0: 23994: warning :<br> > > > > virNetTLSContextCheckCertificat=
e:1102 : Certificate check failed<br> > > > > Certificate faile=
d validation: The certificate hasn't got a known<br> > > > > is=
suer.<br> > > ><br> > > > Would you share your<br> > &=
gt; ><br> > > ><br> > > > openssl x509 -in<br> > &g=
t; > /etc/pki/vdsm/certs/cacert.pem -text<br> > > ><br> > &g=
t; > openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text<br> > &g=
t; ><br> > > > on both hosts? This content may be sensitive, an=
d may not<br> > > > provide an answer why libvirt on src cannot co=
ntact libvirtd on the<br> > > > other host. So before you do that,=
 would you test if<br> > > ><br> > > ><br> > > >=
   vdsClient -s <a href=3D"http://hv02.melb.domain.net" target=3D"_bla=
nk" data-mce-href=3D"http://hv02.melb.domain.net">hv02.melb.domain.net</a> =
getVdsCapabilities<br> > > ><br> > > > works when run on =
hv01? It may be that the certificates are fine, but<br> > > > libv=
irt is not configured to use the correct ones.<br> > > ><br> > =
> > Dan.<br> > > ><br> > > ><br> > > vdsClien=
t -s <a href=3D"http://hv02.melb.domain.net" target=3D"_blank" data-mce-hre=
f=3D"http://hv02.melb.domain.net">hv02.melb.domain.net</a> getVdsCapabiliti=
es runs fine<br> > ><br> > > I did a quick comparison between t=
he files on both hosts, they seem to have<br> > > the right details (=
host names, authority etc.)<br> > > cacert.pem matches<br> > ><=
br> > > /etc/libvirt/libvirtd.conf<br> > ><br> > > ca_fil=
e=3D"/etc/pki/vdsm/certs/cacert.pem"<br> > > cert_file=3D"/etc/pki/vd=
sm/certs/vdsmcert.pem"<br> > > key_file=3D"/etc/pki/vdsm/keys/vdsmkey=
.pem"<br> ><br> <br></div></div>this sounds a little like<br> <a href=3D=
"https://bugzilla.redhat.com/show_bug.cgi?id=3D996146" target=3D"_blank" da=
ta-mce-href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D996146">https:=
//bugzilla.redhat.com/show_bug.cgi?id=3D996146</a><br> <br> can you try to =
restart libvirt (on both hosts just to be sure) and try again?<br><div clas=
s=3D"im"><br> > Maybe someone on libvir-list could guess why this could =
be happening?<br></div>> _______________________________________________=
<br> > Users mailing list<br> > <a href=3D"mailto:Users@ovirt.org" ta=
rget=3D"_blank" data-mce-href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a=
<br> > <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" target=
=3D"_blank" data-mce-href=3D"http://lists.ovirt.org/mailman/listinfo/users"=
http://lists.ovirt.org/mailman/listinfo/users</a><br> ><br></blockquote=
</div><br></div><div class=3D"gmail_extra"><div class=3D"gmail_default" st=
yle=3D"font-family:tahoma,sans-serif" data-mce-style=3D"font-family: tahoma=
,sans-serif;">I did try that already</div><div class=3D"gmail_default" styl=
e=3D"font-family:tahoma,sans-serif" data-mce-style=3D"font-family: tahoma,s=
ans-serif;"><br></div><div class=3D"gmail_default" style=3D"font-family:tah=
oma,sans-serif" data-mce-style=3D"font-family: tahoma,sans-serif;">service =
vdsmd restart</div><div class=3D"gmail_default" style=3D"font-family:tahoma=
,sans-serif" data-mce-style=3D"font-family: tahoma,sans-serif;"><br></div><=
...
Migration still failed. Keep in mind, when I had oVirt 3.3 on these nodes =
migration was working fine. Only when I upgraded to the nightly and it pick=
ed up the new vdsm packages it started to fail.</div><div style=3D"font-fam=
ily:tahoma,sans-serif" data-mce-style=3D"font-family: tahoma,sans-serif;"><=
br></div></div><br></div></div></blockquote><div>can you try to restart the=
can you try to restart the libvirtd service itself? not vdsm 

------=_Part_21768893_1437824270.1379317267664
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><body><div style=3D"font-family: times new roman, new york, times, se=
rif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><hr id=
=3D"zwchr"><blockquote style=3D"border-left:2px solid #1010FF;margin-left:5=
px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-de=
coration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-=
mce-style=3D"border-left: 2px solid #1010FF; margin-left: 5px; padding-left=
: 5px; color: #000; font-weight: normal; font-style: normal; text-decoratio=
n: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From=
: </b>"Andrew Lau" <andrew@andrewklau.com><br><b>To: </b>"Omer Frenke=
l" <ofrenkel@redhat.com><br><b>Cc: </b>"Dan Kenigsberg" <danken@re=
dhat.com>, libvir-list@redhat.com, "users" <users@ovirt.org><br><b=
href=3D"mailto:danken@redhat.com" target=3D"_blank" data-mce-href=3D"mailto=
:danken@redhat.com">danken@redhat.com</a>><br> > To: "Andrew Lau" <=
;<a href=3D"mailto:andrew@andrewklau.com" target=3D"_blank" data-mce-href=
=3D"mailto:andrew@andrewklau.com">andrew@andrewklau.com</a>><br> > Cc=
: <a href=3D"mailto:libvir-list@redhat.com" target=3D"_blank" data-mce-href=
=3D"mailto:libvir-list@redhat.com">libvir-list@redhat.com</a>, "users" <=
<a href=3D"mailto:users@ovirt.org" target=3D"_blank" data-mce-href=3D"mailt=
o:users@ovirt.org">users@ovirt.org</a>><br> > Sent: Sunday, September=
 15, 2013 3:47:03 PM<br> > Subject: Re: [Users] Live Migration failed oV=
irt 3.3 Nightly<br> ><br> > On Sun, Sep 15, 2013 at 09:57:47PM +1000,=
 Andrew Lau wrote:<br> > > On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenig=
sberg <<a href=3D"mailto:danken@redhat.com" target=3D"_blank" data-mce-h=
ref=3D"mailto:danken@redhat.com">danken@redhat.com</a>> wrote:<br> > =
><br> > > > On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew La=
u wrote:<br> > > > > On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenig=
sberg <<a href=3D"mailto:danken@redhat.com" target=3D"_blank" data-mce-h=
ref=3D"mailto:danken@redhat.com">danken@redhat.com</a>><br> > > &g=
t; wrote:<br> > > > ><br> > > > > > On Sun, Sep =
15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:<br> > > > > >=
; > Hi Dan,<br> > > > > > ><br> > > > > &g=
t; > Certainly, I've uploaded them to fedora's paste bin and tried to<br=
div class=3D"gmail_default"><div class=3D"gmail_default"><span style=3D"fon=
t-family: tahoma,sans-serif;" data-mce-style=3D"font-family: tahoma,sans-se=
rif;" face=3D"tahoma, sans-serif">[root@hv02 ~]# service vdsmd restart</spa=
n></div><div class=3D"gmail_default"><span style=3D"font-family: tahoma,san=
s-serif;" data-mce-style=3D"font-family: tahoma,sans-serif;" face=3D"tahoma=
, sans-serif">Shutting down vdsm daemon: </span></div><div class=3D"gm=
ail_default"><span style=3D"font-family: tahoma,sans-serif;" data-mce-style=
=3D"font-family: tahoma,sans-serif;" face=3D"tahoma, sans-serif">vdsm watch=
dog stop                   &nb=
sp;                     [=
  OK  ]</span></div><div class=3D"gmail_default"><span style=3D"f=
ont-family: tahoma,sans-serif;" data-mce-style=3D"font-family: tahoma,sans-=
serif;" face=3D"tahoma, sans-serif">vdsm stop         &=
nbsp;                    =
                    [ &nb=
sp;OK  ]</span></div><div class=3D"gmail_default"><span style=3D"font-=
family: tahoma,sans-serif;" data-mce-style=3D"font-family: tahoma,sans-seri=
f;" face=3D"tahoma, sans-serif">Starting configure libvirt to VDSM ...</spa=
n></div><div class=3D"gmail_default"><span style=3D"font-family: tahoma,san=
s-serif;" data-mce-style=3D"font-family: tahoma,sans-serif;" face=3D"tahoma=
, sans-serif">libvirt is already configured for vdsm</span></div><div class=
=3D"gmail_default"><span style=3D"font-family: tahoma,sans-serif;" data-mce=
-style=3D"font-family: tahoma,sans-serif;" face=3D"tahoma, sans-serif">=3DD=
one configuring libvirt=3D</span></div><div class=3D"gmail_default"><span s=
tyle=3D"font-family: tahoma,sans-serif;" data-mce-style=3D"font-family: tah=
oma,sans-serif;" face=3D"tahoma, sans-serif">libvir: Network Filter Driver =
error : Requested operation is not valid: nwfilter is in use</span></div><d=
iv class=3D"gmail_default"><span style=3D"font-family: tahoma,sans-serif;" =
data-mce-style=3D"font-family: tahoma,sans-serif;" face=3D"tahoma, sans-ser=
if">Checking conflicts ...</span></div><div class=3D"gmail_default"><span s=
tyle=3D"font-family: tahoma,sans-serif;" data-mce-style=3D"font-family: tah=
oma,sans-serif;" face=3D"tahoma, sans-serif">SUCCESS: ssl configured to tru=
e. No conflicts</span></div><div class=3D"gmail_default"><span style=3D"fon=
t-family: tahoma,sans-serif;" data-mce-style=3D"font-family: tahoma,sans-se=
rif;" face=3D"tahoma, sans-serif">Starting up vdsm daemon: </span></di=
v><div class=3D"gmail_default"><span style=3D"font-family: tahoma,sans-seri=
f;" data-mce-style=3D"font-family: tahoma,sans-serif;" face=3D"tahoma, sans=
-serif">vdsm start                 =
                     =
;           [  OK  ]</span></div><div st=
yle=3D"font-family:tahoma,sans-serif" data-mce-style=3D"font-family: tahoma=
,sans-serif;"><br></div><div style=3D"font-family:tahoma,sans-serif" data-m=
ce-style=3D"font-family: tahoma,sans-serif;"><br></div><div style=3D"font-f=
amily:tahoma,sans-serif" data-mce-style=3D"font-family: tahoma,sans-serif;"=
 libvirtd service itself? not vdsm<br></div></div></body></html>
------=_Part_21768893_1437824270.1379317267664--