2:41 a.m.
------=_Part_21768893_1437824270.1379317267664
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
----- Original Message -----
From: "Andrew Lau" <andrew(a)andrewklau.com>
To: "Omer Frenkel" <ofrenkel(a)redhat.com>
Cc: "Dan Kenigsberg" <danken(a)redhat.com>, libvir-list(a)redhat.com,
"users"
<users(a)ovirt.org>
Sent: Monday, September 16, 2013 1:38:53 AM
Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel <
ofrenkel(a)redhat.com > wrote:
> ----- Original Message -----
> > From: "Dan Kenigsberg" < danken(a)redhat.com >
> > To: "Andrew Lau" < andrew(a)andrewklau.com >
> > Cc: libvir-list(a)redhat.com , "users" < users(a)ovirt.org >
> > Sent: Sunday, September 15, 2013 3:47:03 PM
> > Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
> >
> > On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
> > > On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg < danken(a)redhat.com
>
> > > wrote:
> > >
> > > > On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
> > > > > On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg <
danken(a)redhat.com
> > > > > >
> > > > wrote:
> > > > >
> > > > > > On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau
wrote:
> > > > > > > Hi Dan,
> > > > > > >
> > > > > > > Certainly, I've uploaded them to fedora's
paste bin and tried
> > > > > > > to
> > > > > > > snip
> > > > > > just
> > > > > > > the relevant details.
> > > > > > >
> > > > > > > Sender ( hv01.melb.domain.net ):
> > > > > > > http://paste.fedoraproject.org/39660/92339651/
> > > > > >
> > > > > > This one has
> > > > > >
> > > > > > libvirtError: operation failed: Failed to connect to
remote
> > > > > > libvirt
> > > > > > URI qemu+tls:// hv02.melb.domain.net/system
> > > > > >
> > > > > > which is most often related to firewall issues, and some
time to
> > > > > > key
> > > > > > mismatch.
> > > > > >
> > > > > > Does
> > > > > > virsh -c qemu+tls:// hv02.melb.domain.net/system
capabilities
> > > > > > work when run from the command line of hv01?
> > > > > >
> > > > > > Dan.
> > > > > > > Receiver ( hv02.melb.domain.net ): `
> > > > > > > http://paste.fedoraproject.org/39661/23406913/
> > > > > > >
> > > > > > > VM being transfered is ovirt_guest_vm
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Andrew
> > > > > >
> > > > >
> > > > > virsh -c qemu+tls:// hv02.melb.domain.net/system
> > > > > 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version:
> > > > > 0.10.2,
> > > > > package: 18.el6_4.9 (CentOS BuildSystem <
http://bugs.centos.org >,
> > > > > 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org )
> > > > > 2013-09-15 10:41:10.620+0000: 23994: warning :
> > > > > virNetTLSContextCheckCertificate:1102 : Certificate check
failed
> > > > > Certificate failed validation: The certificate hasn't got a
known
> > > > > issuer.
> > > >
> > > > Would you share your
> > > >
> > > >
> > > > openssl x509 -in
> > > > /etc/pki/vdsm/certs/cacert.pem -text
> > > >
> > > > openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
> > > >
> > > > on both hosts? This content may be sensitive, and may not
> > > > provide an answer why libvirt on src cannot contact libvirtd on the
> > > > other host. So before you do that, would you test if
> > > >
> > > >
> > > > vdsClient -s hv02.melb.domain.net getVdsCapabilities
> > > >
> > > > works when run on hv01? It may be that the certificates are fine,
but
> > > > libvirt is not configured to use the correct ones.
> > > >
> > > > Dan.
> > > >
> > > >
> > > vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
> > >
> > > I did a quick comparison between the files on both hosts, they seem to
> > > have
> > > the right details (host names, authority etc.)
> > > cacert.pem matches
> > >
> > > /etc/libvirt/libvirtd.conf
> > >
> > > ca_file="/etc/pki/vdsm/certs/cacert.pem"
> > > cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
> > > key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
> >
> this sounds a little like
> https://bugzilla.redhat.com/show_bug.cgi?id=996146
> can you try to restart libvirt (on both hosts just to be sure)
and try
> again?
> > Maybe someone on libvir-list could guess why this could be
happening?
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
I did try that already
service vdsmd restart
[root@hv02 ~]# service vdsmd restart
Shutting down vdsm daemon:
vdsm watchdog stop [ OK ]
vdsm stop [ OK ]
Starting configure libvirt to VDSM ...
libvirt is already configured for vdsm
=Done configuring libvirt=
libvir: Network Filter Driver error : Requested operation is not valid:
nwfilter is in use
Checking conflicts ...
SUCCESS: ssl configured to true. No conflicts
Starting up vdsm daemon:
vdsm start [ OK ]
Migration still failed. Keep in mind, when I had oVirt 3.3 on these
nodes
migration was working fine. Only when I upgraded to the nightly and it
picked up the new vdsm packages it started to fail.
can you try to restart the libvirtd service itself? not vdsm
------=_Part_21768893_1437824270.1379317267664
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"font-family: times new roman, new york,
times, se=
rif; font-size: 12pt; color:
#000000"><div><br></div><div><br></div><hr
id=
=3D"zwchr"><blockquote style=3D"border-left:2px solid
#1010FF;margin-left:5=
px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-de=
coration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-=
mce-style=3D"border-left: 2px solid #1010FF; margin-left: 5px; padding-left=
: 5px; color: #000; font-weight: normal; font-style: normal; text-decoratio=
n: none; font-family: Helvetica,Arial,sans-serif; font-size:
12pt;"><b>From=
: </b>"Andrew Lau"
&lt;andrew(a)andrewklau.com&gt;<br><b>To: </b>"Omer Frenke=
l" &lt;ofrenkel(a)redhat.com&gt;<br><b>Cc: </b>"Dan
Kenigsberg" <danken@re=
dhat.com>, libvir-list(a)redhat.com, "users"
&lt;users(a)ovirt.org&gt;<br><b=
Sent: </b>Monday, September 16, 2013 1:38:53
AM<br><b>Subject: </b>Re: [Us=
ers] Live Migration failed oVirt
3.3 Nightly<br><div><br></div><div dir=3D"=
ltr"><div class=3D"gmail_default"
style=3D"font-family:tahoma,sans-serif" d=
ata-mce-style=3D"font-family: tahoma,sans-serif;"><span
style=3D"font-famil=
y:arial" data-mce-style=3D"font-family: arial;">On Sun, Sep 15, 2013 at
11:=
51 PM, Omer Frenkel </span><span dir=3D"ltr"
style=3D"font-family:arial" da=
ta-mce-style=3D"font-family: arial;"><<a
href=3D"mailto:ofrenkel@redhat.=
com" target=3D"_blank"
data-mce-href=3D"mailto:ofrenkel@redhat.com">ofrenke=
l(a)redhat.com</a>&gt;</span><span style=3D"font-family:arial"
data-mce-style=
=3D"font-family: arial;"> wrote:</span><br></div><div
class=3D"gmail_extra"=
<div class=3D"gmail_quote"><blockquote
class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,20=
4);border-left-style:solid;padding-left:1ex" data-mce-style=3D"margin: 0px =
0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-l=
eft-style: solid; padding-left: 1ex;"><div class=3D""><div
class=3D"h5"><br=
<br> ----- Original Message -----<br> > From:
"Dan Kenigsberg" <<a =
href=3D"mailto:danken@redhat.com" target=3D"_blank"
data-mce-href=3D"mailto=
:danken@redhat.com">danken@redhat.com</a>><br> > To:
"Andrew Lau" <=
;<a href=3D"mailto:andrew@andrewklau.com" target=3D"_blank"
data-mce-href=
=3D"mailto:andrew@andrewklau.com">andrew@andrewklau.com</a>><br>
> Cc=
: <a href=3D"mailto:libvir-list@redhat.com" target=3D"_blank"
data-mce-href=
=3D"mailto:libvir-list@redhat.com">libvir-list@redhat.com</a>,
"users" <=
<a href=3D"mailto:users@ovirt.org" target=3D"_blank"
data-mce-href=3D"mailt=
o:users@ovirt.org">users@ovirt.org</a>><br> > Sent:
Sunday, September=
15, 2013 3:47:03 PM<br> > Subject: Re: [Users] Live Migration failed oV=
irt 3.3 Nightly<br> ><br> > On Sun, Sep 15, 2013 at 09:57:47PM
+1000,=
Andrew Lau wrote:<br> > > On Sun, Sep 15, 2013 at 9:34 PM, Dan
Kenig=
sberg <<a href=3D"mailto:danken@redhat.com"
target=3D"_blank" data-mce-h=
ref=3D"mailto:danken@redhat.com">danken@redhat.com</a>>
wrote:<br> > =
><br> > > > On Sun, Sep 15, 2013 at 08:44:18PM +1000,
Andrew La=
u wrote:<br> > > > > On Sun, Sep 15, 2013 at 8:00 PM,
Dan Kenig=
sberg <<a href=3D"mailto:danken@redhat.com"
target=3D"_blank" data-mce-h=
ref=3D"mailto:danken@redhat.com">danken@redhat.com</a>><br>
> > &g=
t; wrote:<br> > > > ><br> > >
> > > On Sun, Sep =
15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:<br> > > >
> >=
; > Hi Dan,<br> > > > > >
><br> > > > > &g=
t; > Certainly, I've uploaded them to fedora's paste bin and tried
to<br=
> > > > > > snip<br>
> > > > > just<br> =
> >
> > > > the relevant details.<br> > >
> >=
> ><br> > > > > > > Sender
(<a href=3D"http://hv01=
.melb.domain.net" target=3D"_blank"
data-mce-href=3D"http://hv01.melb.domai=
n.net">hv01.melb.domain.net</a>):<br> > > >
> > > <a href=
=3D"http://paste.fedoraproject.org/39660/92339651/" target=3D"_blank"
data-=
mce-href=3D"http://paste.fedoraproject.org/39660/92339651/">...
doraproject.org/39660/92339651/</a><br> > > > >
><br> > &=
gt; > > > This one has<br> > > > >
><br> > > =
> > > libvirtError: operation failed: Failed to
conn=
ect to remote<br> > > > > >
libvirt<br> > &=
gt; > > > URI qemu+tls://<a
href=3D"http://hv02.melb.domain.net/sy=
stem" target=3D"_blank"
data-mce-href=3D"http://hv02.melb.domain.net/system=
">hv02.melb.domain.net/system</a><br> > > >
> ><br> > >=
; > > > which is most often related to firewall issues, and some t=
ime to key<br> > > > > > mismatch.<br>
> > > > &=
gt;<br> > > > > > Does<br> >
> > > > &=
nbsp; virsh -c qemu+tls://<a href=3D"http://hv02.melb.domain.net/system" ta=
rget=3D"_blank"
data-mce-href=3D"http://hv02.melb.domain.net/system">hv02.m=
elb.domain.net/system</a> capabilities<br> > > > >
> work wh=
en run from the command line of hv01?<br> > > > >
><br> >=
> > > > Dan.<br> > > > >
> > Receiver (<a hr=
ef=3D"http://hv02.melb.domain.net" target=3D"_blank"
data-mce-href=3D"http:=
//hv02.melb.domain.net">hv02.melb.domain.net</a>): `<br> >
> > >=
; > > <a
href=3D"http://paste.fedoraproject.org/39661/23406913/" targ=
et=3D"_blank"
data-mce-href=3D"http://paste.fedoraproject.org/39661/2340691=
3/">http://paste.fedoraproject.org/39661/23406913/</a><b... >
> > &=
gt; > ><br> > > > > > > VM
being transfered is ovir=
t_guest_vm<br> > > > > > ><br>
> > > > > &=
gt; Thanks,<br> > > > > > >
Andrew<br> > > > >=
; ><br> > > > ><br> > >
> > virsh -c qemu+tls://=
<a href=3D"http://hv02.melb.domain.net/system" target=3D"_blank"
data-mce-h=
ref=3D"http://hv02.melb.domain.net/system">hv02.melb.domain....
br> > > > > 2013-09-15 10:41:10.620+0000: 23994: info :
libvirt=
version: 0.10.2,<br> > > > > package: 18.el6_4.9
(CentOS Build=
System <<a href=3D"http://bugs.centos.org" target=3D"_blank"
data-mce-hr=
ef=3D"http://bugs.centos.org">http://bugs.centos.org</a>>,<br>
> >=
> > 2013-07-02-11:19:29, <a
href=3D"http://c6b8.bsys.dev.centos.org"=
target=3D"_blank"
data-mce-href=3D"http://c6b8.bsys.dev.centos.org">c6b8.b=
sys.dev.centos.org</a>)<br> > > > > 2013-09-15
10:41:10.620+000=
0: 23994: warning :<br> > > > >
virNetTLSContextCheckCertificat=
e:1102 : Certificate check failed<br> > > > >
Certificate faile=
d validation: The certificate hasn't got a known<br> > > >
> is=
suer.<br> > > ><br> > > > Would you
share your<br> > &=
gt; ><br> > > ><br> > > >
openssl x509 -in<br> > &g=
t; > /etc/pki/vdsm/certs/cacert.pem -text<br> > >
><br> > &g=
t; > openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text<br> >
&g=
t; ><br> > > > on both hosts? This content may be
sensitive, an=
d may not<br> > > > provide an answer why libvirt on src cannot
co=
ntact libvirtd on the<br> > > > other host. So before you do
that,=
would you test if<br> > > ><br> > >
><br> > > >=
vdsClient -s <a href=3D"http://hv02.melb.domain.net"
target=3D"_bla=
nk"
data-mce-href=3D"http://hv02.melb.domain.net">hv02.melb.domain.net</a>
=
getVdsCapabilities<br> > > ><br> > >
> works when run on =
hv01? It may be that the certificates are fine, but<br> > > >
libv=
irt is not configured to use the correct ones.<br> > >
><br> > =
> > Dan.<br> > > ><br> > >
><br> > > vdsClien=
t -s <a href=3D"http://hv02.melb.domain.net" target=3D"_blank"
data-mce-hre=
f=3D"http://hv02.melb.domain.net">hv02.melb.domain.net</a>
getVdsCapabiliti=
es runs fine<br> > ><br> > > I did a quick
comparison between t=
he files on both hosts, they seem to have<br> > > the right details
(=
host names, authority etc.)<br> > > cacert.pem matches<br>
> ><=
br> > > /etc/libvirt/libvirtd.conf<br> > ><br>
> > ca_fil=
e=3D"/etc/pki/vdsm/certs/cacert.pem"<br> > >
cert_file=3D"/etc/pki/vd=
sm/certs/vdsmcert.pem"<br> > >
key_file=3D"/etc/pki/vdsm/keys/vdsmkey=
.pem"<br> ><br> <br></div></div>this sounds a
little like<br> <a href=3D=
"https://bugzilla.redhat.com/show_bug.cgi?id=3D996146"
target=3D"_blank" da=
ta-mce-href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D996146&...
//bugzilla.redhat.com/show_bug.cgi?id=3D996146</a><br> <br> can you try
to =
restart libvirt (on both hosts just to be sure) and try again?<br><div clas=
s=3D"im"><br> > Maybe someone on libvir-list could guess why
this could =
be happening?<br></div>>
_______________________________________________=
<br> > Users mailing list<br> > <a
href=3D"mailto:Users@ovirt.org" ta=
rget=3D"_blank"
data-mce-href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a=
<br> > <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users" target=
=3D"_blank"
data-mce-href=3D"http://lists.ovirt.org/mailman/listinfo/users"=
http://lists.ovirt.org/mailman/listinfo/users</a><br>
><br></blockquote=
</div><br></div><div class=3D"gmail_extra"><div
class=3D"gmail_default" st=
yle=3D"font-family:tahoma,sans-serif" data-mce-style=3D"font-family:
tahoma=
,sans-serif;">I did try that already</div><div
class=3D"gmail_default" styl=
e=3D"font-family:tahoma,sans-serif" data-mce-style=3D"font-family:
tahoma,s=
ans-serif;"><br></div><div class=3D"gmail_default"
style=3D"font-family:tah=
oma,sans-serif" data-mce-style=3D"font-family:
tahoma,sans-serif;">service =
vdsmd restart</div><div class=3D"gmail_default"
style=3D"font-family:tahoma=
,sans-serif" data-mce-style=3D"font-family:
tahoma,sans-serif;"><br></div><=
div class=3D"gmail_default"><div
class=3D"gmail_default"><span style=3D"fon=
t-family: tahoma,sans-serif;" data-mce-style=3D"font-family: tahoma,sans-se=
rif;" face=3D"tahoma, sans-serif">[root@hv02 ~]# service vdsmd
restart</spa=
n></div><div class=3D"gmail_default"><span
style=3D"font-family: tahoma,san=
s-serif;" data-mce-style=3D"font-family: tahoma,sans-serif;"
face=3D"tahoma=
, sans-serif">Shutting down vdsm daemon: </span></div><div
class=3D"gm=
ail_default"><span style=3D"font-family: tahoma,sans-serif;"
data-mce-style=
=3D"font-family: tahoma,sans-serif;" face=3D"tahoma,
sans-serif">vdsm watch=
dog stop
&nb=
sp;
[=
OK ]</span></div><div
class=3D"gmail_default"><span style=3D"f=
ont-family: tahoma,sans-serif;" data-mce-style=3D"font-family: tahoma,sans-=
serif;" face=3D"tahoma, sans-serif">vdsm stop
&=
nbsp;
=
[ &nb=
sp;OK ]</span></div><div
class=3D"gmail_default"><span style=3D"font-=
family: tahoma,sans-serif;" data-mce-style=3D"font-family: tahoma,sans-seri=
f;" face=3D"tahoma, sans-serif">Starting configure libvirt to VDSM
...</spa=
n></div><div class=3D"gmail_default"><span
style=3D"font-family: tahoma,san=
s-serif;" data-mce-style=3D"font-family: tahoma,sans-serif;"
face=3D"tahoma=
, sans-serif">libvirt is already configured for
vdsm</span></div><div class=
=3D"gmail_default"><span style=3D"font-family:
tahoma,sans-serif;" data-mce=
-style=3D"font-family: tahoma,sans-serif;" face=3D"tahoma,
sans-serif">=3DD=
one configuring libvirt=3D</span></div><div
class=3D"gmail_default"><span s=
tyle=3D"font-family: tahoma,sans-serif;" data-mce-style=3D"font-family:
tah=
oma,sans-serif;" face=3D"tahoma, sans-serif">libvir: Network Filter
Driver =
error : Requested operation is not valid: nwfilter is in
use</span></div><d=
iv class=3D"gmail_default"><span style=3D"font-family:
tahoma,sans-serif;" =
data-mce-style=3D"font-family: tahoma,sans-serif;" face=3D"tahoma,
sans-ser=
if">Checking conflicts ...</span></div><div
class=3D"gmail_default"><span s=
tyle=3D"font-family: tahoma,sans-serif;" data-mce-style=3D"font-family:
tah=
oma,sans-serif;" face=3D"tahoma, sans-serif">SUCCESS: ssl configured to
tru=
e. No conflicts</span></div><div
class=3D"gmail_default"><span style=3D"fon=
t-family: tahoma,sans-serif;" data-mce-style=3D"font-family: tahoma,sans-se=
rif;" face=3D"tahoma, sans-serif">Starting up vdsm
daemon: </span></di=
v><div class=3D"gmail_default"><span style=3D"font-family:
tahoma,sans-seri=
f;" data-mce-style=3D"font-family: tahoma,sans-serif;" face=3D"tahoma,
sans=
-serif">vdsm start
=
 =
; [ OK
]</span></div><div st=
yle=3D"font-family:tahoma,sans-serif" data-mce-style=3D"font-family:
tahoma=
,sans-serif;"><br></div><div
style=3D"font-family:tahoma,sans-serif" data-m=
ce-style=3D"font-family: tahoma,sans-serif;"><br></div><div
style=3D"font-f=
amily:tahoma,sans-serif" data-mce-style=3D"font-family:
tahoma,sans-serif;"=
Migration still failed. Keep in mind, when I had oVirt 3.3 on these
nodes =
migration was working fine. Only when I upgraded to the nightly and it
pick=
ed up the new vdsm packages it started to fail.</div><div
style=3D"font-fam=
ily:tahoma,sans-serif" data-mce-style=3D"font-family:
tahoma,sans-serif;"><=
br></div></div><br></div></div></blockquote><div>can
you try to restart the=
libvirtd service itself? not
vdsm<br></div></div></body></html>
------=_Part_21768893_1437824270.1379317267664--