6:34 a.m.
On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg
<danken(a)redhat.com> wrote:
> On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:
> > Hi Dan,
> >
> > Certainly, I've uploaded them to fedora's paste bin and tried to snip
> just
> > the relevant details.
> >
> > Sender (hv01.melb.domain.net):
> > http://paste.fedoraproject.org/39660/92339651/
>
> This one has
>
> libvirtError: operation failed: Failed to connect to remote libvirt
> URI qemu+tls://hv02.melb.domain.net/system
>
> which is most often related to firewall issues, and some time to key
> mismatch.
>
> Does
> virsh -c qemu+tls://hv02.melb.domain.net/system capabilities
> work when run from the command line of hv01?
>
> Dan.
> > Receiver (hv02.melb.domain.net): `
> > http://paste.fedoraproject.org/39661/23406913/
> >
> > VM being transfered is ovirt_guest_vm
> >
> > Thanks,
> > Andrew
>
virsh -c qemu+tls://hv02.melb.domain.net/system
2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: 0.10.2,
package: 18.el6_4.9 (CentOS BuildSystem <http://bugs.centos.org>;,
2013-07-02-11:19:29, c6b8.bsys.dev.centos.org)
2013-09-15 10:41:10.620+0000: 23994: warning :
virNetTLSContextCheckCertificate:1102 : Certificate check failed
Certificate failed validation: The certificate hasn't got a known issuer.
Would you share your
openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -text
openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
on both hosts? This content may be sensitive, and may not
provide an answer why libvirt on src cannot contact libvirtd on the
other host. So before you do that, would you test if
vdsClient -s hv02.melb.domain.net getVdsCapabilities
works when run on hv01? It may be that the certificates are fine, but
libvirt is not configured to use the correct ones.
Dan.
6:57 a.m.
New subject: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg <danken(a)redhat.com> wrote:
On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
> On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg <danken(a)redhat.com>
wrote:
>
> > On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:
> > > Hi Dan,
> > >
> > > Certainly, I've uploaded them to fedora's paste bin and tried to
snip
> > just
> > > the relevant details.
> > >
> > > Sender (hv01.melb.domain.net):
> > > http://paste.fedoraproject.org/39660/92339651/
> >
> > This one has
> >
> > libvirtError: operation failed: Failed to connect to remote libvirt
> > URI qemu+tls://hv02.melb.domain.net/system
> >
> > which is most often related to firewall issues, and some time to key
> > mismatch.
> >
> > Does
> > virsh -c qemu+tls://hv02.melb.domain.net/system capabilities
> > work when run from the command line of hv01?
> >
> > Dan.
> > > Receiver (hv02.melb.domain.net): `
> > > http://paste.fedoraproject.org/39661/23406913/
> > >
> > > VM being transfered is ovirt_guest_vm
> > >
> > > Thanks,
> > > Andrew
> >
>
> virsh -c qemu+tls://hv02.melb.domain.net/system
> 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: 0.10.2,
> package: 18.el6_4.9 (CentOS BuildSystem <http://bugs.centos.org>;,
> 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org)
> 2013-09-15 10:41:10.620+0000: 23994: warning :
> virNetTLSContextCheckCertificate:1102 : Certificate check failed
> Certificate failed validation: The certificate hasn't got a known issuer.
Would you share your
openssl x509 -in
/etc/pki/vdsm/certs/cacert.pem -text
openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
on both hosts? This content may be sensitive, and may not
provide an answer why libvirt on src cannot contact libvirtd on the
other host. So before you do that, would you test if
vdsClient -s hv02.melb.domain.net getVdsCapabilities
works when run on hv01? It may be that the certificates are fine, but
libvirt is not configured to use the correct ones.
Dan.
vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
I did a quick comparison between the files on both hosts, they seem to have
the right details (host names, authority etc.)
cacert.pem matches
/etc/libvirt/libvirtd.conf
ca_file="/etc/pki/vdsm/certs/cacert.pem"
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
7:47 a.m.
New subject: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg
<danken(a)redhat.com> wrote:
> On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
> > On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg <danken(a)redhat.com>
> wrote:
> >
> > > On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:
> > > > Hi Dan,
> > > >
> > > > Certainly, I've uploaded them to fedora's paste bin and tried
to snip
> > > just
> > > > the relevant details.
> > > >
> > > > Sender (hv01.melb.domain.net):
> > > > http://paste.fedoraproject.org/39660/92339651/
> > >
> > > This one has
> > >
> > > libvirtError: operation failed: Failed to connect to remote libvirt
> > > URI qemu+tls://hv02.melb.domain.net/system
> > >
> > > which is most often related to firewall issues, and some time to key
> > > mismatch.
> > >
> > > Does
> > > virsh -c qemu+tls://hv02.melb.domain.net/system capabilities
> > > work when run from the command line of hv01?
> > >
> > > Dan.
> > > > Receiver (hv02.melb.domain.net): `
> > > > http://paste.fedoraproject.org/39661/23406913/
> > > >
> > > > VM being transfered is ovirt_guest_vm
> > > >
> > > > Thanks,
> > > > Andrew
> > >
> >
> > virsh -c qemu+tls://hv02.melb.domain.net/system
> > 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: 0.10.2,
> > package: 18.el6_4.9 (CentOS BuildSystem <http://bugs.centos.org>;,
> > 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org)
> > 2013-09-15 10:41:10.620+0000: 23994: warning :
> > virNetTLSContextCheckCertificate:1102 : Certificate check failed
> > Certificate failed validation: The certificate hasn't got a known issuer.
>
> Would you share your
>
>
> openssl x509 -in
> /etc/pki/vdsm/certs/cacert.pem -text
>
> openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
>
> on both hosts? This content may be sensitive, and may not
> provide an answer why libvirt on src cannot contact libvirtd on the
> other host. So before you do that, would you test if
>
>
> vdsClient -s hv02.melb.domain.net getVdsCapabilities
>
> works when run on hv01? It may be that the certificates are fine, but
> libvirt is not configured to use the correct ones.
>
> Dan.
>
>
vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
I did a quick comparison between the files on both hosts, they seem to have
the right details (host names, authority etc.)
cacert.pem matches
/etc/libvirt/libvirtd.conf
ca_file="/etc/pki/vdsm/certs/cacert.pem"
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
Maybe someone on libvir-list could guess why this could be happening?
8:51 a.m.
New subject: [Users] Live Migration failed oVirt 3.3 Nightly
----- Original Message -----
From: "Dan Kenigsberg" <danken(a)redhat.com>
To: "Andrew Lau" <andrew(a)andrewklau.com>
Cc: libvir-list(a)redhat.com, "users" <users(a)ovirt.org>
Sent: Sunday, September 15, 2013 3:47:03 PM
Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
> On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg <danken(a)redhat.com> wrote:
>
> > On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
> > > On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg <danken(a)redhat.com>
> > wrote:
> > >
> > > > On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:
> > > > > Hi Dan,
> > > > >
> > > > > Certainly, I've uploaded them to fedora's paste bin and
tried to
> > > > > snip
> > > > just
> > > > > the relevant details.
> > > > >
> > > > > Sender (hv01.melb.domain.net):
> > > > > http://paste.fedoraproject.org/39660/92339651/
> > > >
> > > > This one has
> > > >
> > > > libvirtError: operation failed: Failed to connect to remote
> > > > libvirt
> > > > URI qemu+tls://hv02.melb.domain.net/system
> > > >
> > > > which is most often related to firewall issues, and some time to key
> > > > mismatch.
> > > >
> > > > Does
> > > > virsh -c qemu+tls://hv02.melb.domain.net/system capabilities
> > > > work when run from the command line of hv01?
> > > >
> > > > Dan.
> > > > > Receiver (hv02.melb.domain.net): `
> > > > > http://paste.fedoraproject.org/39661/23406913/
> > > > >
> > > > > VM being transfered is ovirt_guest_vm
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > >
> > >
> > > virsh -c qemu+tls://hv02.melb.domain.net/system
> > > 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: 0.10.2,
> > > package: 18.el6_4.9 (CentOS BuildSystem <http://bugs.centos.org>;,
> > > 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org)
> > > 2013-09-15 10:41:10.620+0000: 23994: warning :
> > > virNetTLSContextCheckCertificate:1102 : Certificate check failed
> > > Certificate failed validation: The certificate hasn't got a known
> > > issuer.
> >
> > Would you share your
> >
> >
> > openssl x509 -in
> > /etc/pki/vdsm/certs/cacert.pem -text
> >
> > openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
> >
> > on both hosts? This content may be sensitive, and may not
> > provide an answer why libvirt on src cannot contact libvirtd on the
> > other host. So before you do that, would you test if
> >
> >
> > vdsClient -s hv02.melb.domain.net getVdsCapabilities
> >
> > works when run on hv01? It may be that the certificates are fine, but
> > libvirt is not configured to use the correct ones.
> >
> > Dan.
> >
> >
> vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
>
> I did a quick comparison between the files on both hosts, they seem to have
> the right details (host names, authority etc.)
> cacert.pem matches
>
> /etc/libvirt/libvirtd.conf
>
> ca_file="/etc/pki/vdsm/certs/cacert.pem"
> cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
> key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
this sounds a little like
https://bugzilla.redhat.com/show_bug.cgi?id=996146
can you try to restart libvirt (on both hosts just to be sure) and try again?
Maybe someone on libvir-list could guess why this could be
happening?
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
5:38 p.m.
New subject: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel <ofrenkel(a)redhat.com> wrote:
----- Original Message -----
> From: "Dan Kenigsberg" <danken(a)redhat.com>
> To: "Andrew Lau" <andrew(a)andrewklau.com>
> Cc: libvir-list(a)redhat.com, "users" <users(a)ovirt.org>
> Sent: Sunday, September 15, 2013 3:47:03 PM
> Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
>
> On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
> > On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg <danken(a)redhat.com>
wrote:
> >
> > > On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
> > > > On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg
<danken(a)redhat.com
>
> > > wrote:
> > > >
> > > > > On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:
> > > > > > Hi Dan,
> > > > > >
> > > > > > Certainly, I've uploaded them to fedora's paste bin
and tried
to
> > > > > > snip
> > > > > just
> > > > > > the relevant details.
> > > > > >
> > > > > > Sender (hv01.melb.domain.net):
> > > > > > http://paste.fedoraproject.org/39660/92339651/
> > > > >
> > > > > This one has
> > > > >
> > > > > libvirtError: operation failed: Failed to connect to remote
> > > > > libvirt
> > > > > URI qemu+tls://hv02.melb.domain.net/system
> > > > >
> > > > > which is most often related to firewall issues, and some time
to
key
> > > > > mismatch.
> > > > >
> > > > > Does
> > > > > virsh -c qemu+tls://hv02.melb.domain.net/system
capabilities
> > > > > work when run from the command line of hv01?
> > > > >
> > > > > Dan.
> > > > > > Receiver (hv02.melb.domain.net): `
> > > > > > http://paste.fedoraproject.org/39661/23406913/
> > > > > >
> > > > > > VM being transfered is ovirt_guest_vm
> > > > > >
> > > > > > Thanks,
> > > > > > Andrew
> > > > >
> > > >
> > > > virsh -c qemu+tls://hv02.melb.domain.net/system
> > > > 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version:
0.10.2,
> > > > package: 18.el6_4.9 (CentOS BuildSystem
<http://bugs.centos.org>;,
> > > > 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org)
> > > > 2013-09-15 10:41:10.620+0000: 23994: warning :
> > > > virNetTLSContextCheckCertificate:1102 : Certificate check failed
> > > > Certificate failed validation: The certificate hasn't got a
known
> > > > issuer.
> > >
> > > Would you share your
> > >
> > >
> > > openssl x509 -in
> > > /etc/pki/vdsm/certs/cacert.pem -text
> > >
> > > openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
> > >
> > > on both hosts? This content may be sensitive, and may not
> > > provide an answer why libvirt on src cannot contact libvirtd on the
> > > other host. So before you do that, would you test if
> > >
> > >
> > > vdsClient -s hv02.melb.domain.net getVdsCapabilities
> > >
> > > works when run on hv01? It may be that the certificates are fine, but
> > > libvirt is not configured to use the correct ones.
> > >
> > > Dan.
> > >
> > >
> > vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
> >
> > I did a quick comparison between the files on both hosts, they seem to
have
> > the right details (host names, authority etc.)
> > cacert.pem matches
> >
> > /etc/libvirt/libvirtd.conf
> >
> > ca_file="/etc/pki/vdsm/certs/cacert.pem"
> > cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
> > key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
>
this sounds a little like
https://bugzilla.redhat.com/show_bug.cgi?id=996146
can you try to restart libvirt (on both hosts just to be sure) and try
again?
> Maybe someone on libvir-list could guess why this could be happening?
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
I did try that already
service vdsmd restart
[root@hv02 ~]# service vdsmd restart
Shutting down vdsm daemon:
vdsm watchdog stop [ OK ]
vdsm stop [ OK ]
Starting configure libvirt to VDSM ...
libvirt is already configured for vdsm
=Done configuring libvirt=
libvir: Network Filter Driver error : Requested operation is not valid:
nwfilter is in use
Checking conflicts ...
SUCCESS: ssl configured to true. No conflicts
Starting up vdsm daemon:
vdsm start [ OK ]
Migration still failed. Keep in mind, when I had oVirt 3.3 on these nodes
migration was working fine. Only when I upgraded to the nightly and it
picked up the new vdsm packages it started to fail.
2:41 a.m.
New subject: [Users] Live Migration failed oVirt 3.3 Nightly
------=_Part_21768893_1437824270.1379317267664
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
----- Original Message -----
From: "Andrew Lau" <andrew(a)andrewklau.com>
To: "Omer Frenkel" <ofrenkel(a)redhat.com>
Cc: "Dan Kenigsberg" <danken(a)redhat.com>, libvir-list(a)redhat.com,
"users"
<users(a)ovirt.org>
Sent: Monday, September 16, 2013 1:38:53 AM
Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel <
ofrenkel(a)redhat.com > wrote:
> ----- Original Message -----
> > From: "Dan Kenigsberg" < danken(a)redhat.com >
> > To: "Andrew Lau" < andrew(a)andrewklau.com >
> > Cc: libvir-list(a)redhat.com , "users" < users(a)ovirt.org >
> > Sent: Sunday, September 15, 2013 3:47:03 PM
> > Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
> >
> > On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
> > > On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg < danken(a)redhat.com
>
> > > wrote:
> > >
> > > > On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
> > > > > On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg <
danken(a)redhat.com
> > > > > >
> > > > wrote:
> > > > >
> > > > > > On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau
wrote:
> > > > > > > Hi Dan,
> > > > > > >
> > > > > > > Certainly, I've uploaded them to fedora's
paste bin and tried
> > > > > > > to
> > > > > > > snip
> > > > > > just
> > > > > > > the relevant details.
> > > > > > >
> > > > > > > Sender ( hv01.melb.domain.net ):
> > > > > > > http://paste.fedoraproject.org/39660/92339651/
> > > > > >
> > > > > > This one has
> > > > > >
> > > > > > libvirtError: operation failed: Failed to connect to
remote
> > > > > > libvirt
> > > > > > URI qemu+tls:// hv02.melb.domain.net/system
> > > > > >
> > > > > > which is most often related to firewall issues, and some
time to
> > > > > > key
> > > > > > mismatch.
> > > > > >
> > > > > > Does
> > > > > > virsh -c qemu+tls:// hv02.melb.domain.net/system
capabilities
> > > > > > work when run from the command line of hv01?
> > > > > >
> > > > > > Dan.
> > > > > > > Receiver ( hv02.melb.domain.net ): `
> > > > > > > http://paste.fedoraproject.org/39661/23406913/
> > > > > > >
> > > > > > > VM being transfered is ovirt_guest_vm
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Andrew
> > > > > >
> > > > >
> > > > > virsh -c qemu+tls:// hv02.melb.domain.net/system
> > > > > 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version:
> > > > > 0.10.2,
> > > > > package: 18.el6_4.9 (CentOS BuildSystem <
http://bugs.centos.org >,
> > > > > 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org )
> > > > > 2013-09-15 10:41:10.620+0000: 23994: warning :
> > > > > virNetTLSContextCheckCertificate:1102 : Certificate check
failed
> > > > > Certificate failed validation: The certificate hasn't got a
known
> > > > > issuer.
> > > >
> > > > Would you share your
> > > >
> > > >
> > > > openssl x509 -in
> > > > /etc/pki/vdsm/certs/cacert.pem -text
> > > >
> > > > openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
> > > >
> > > > on both hosts? This content may be sensitive, and may not
> > > > provide an answer why libvirt on src cannot contact libvirtd on the
> > > > other host. So before you do that, would you test if
> > > >
> > > >
> > > > vdsClient -s hv02.melb.domain.net getVdsCapabilities
> > > >
> > > > works when run on hv01? It may be that the certificates are fine,
but
> > > > libvirt is not configured to use the correct ones.
> > > >
> > > > Dan.
> > > >
> > > >
> > > vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
> > >
> > > I did a quick comparison between the files on both hosts, they seem to
> > > have
> > > the right details (host names, authority etc.)
> > > cacert.pem matches
> > >
> > > /etc/libvirt/libvirtd.conf
> > >
> > > ca_file="/etc/pki/vdsm/certs/cacert.pem"
> > > cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
> > > key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
> >
> this sounds a little like
> https://bugzilla.redhat.com/show_bug.cgi?id=996146
> can you try to restart libvirt (on both hosts just to be sure)
and try
> again?
> > Maybe someone on libvir-list could guess why this could be
happening?
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
I did try that already
service vdsmd restart
[root@hv02 ~]# service vdsmd restart
Shutting down vdsm daemon:
vdsm watchdog stop [ OK ]
vdsm stop [ OK ]
Starting configure libvirt to VDSM ...
libvirt is already configured for vdsm
=Done configuring libvirt=
libvir: Network Filter Driver error : Requested operation is not valid:
nwfilter is in use
Checking conflicts ...
SUCCESS: ssl configured to true. No conflicts
Starting up vdsm daemon:
vdsm start [ OK ]
Migration still failed. Keep in mind, when I had oVirt 3.3 on these
nodes
migration was working fine. Only when I upgraded to the nightly and it
picked up the new vdsm packages it started to fail.
can you try to restart the libvirtd service itself? not vdsm
------=_Part_21768893_1437824270.1379317267664
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"font-family: times new roman, new york,
times, se=
rif; font-size: 12pt; color:
#000000"><div><br></div><div><br></div><hr
id=
=3D"zwchr"><blockquote style=3D"border-left:2px solid
#1010FF;margin-left:5=
px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-de=
coration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-=
mce-style=3D"border-left: 2px solid #1010FF; margin-left: 5px; padding-left=
: 5px; color: #000; font-weight: normal; font-style: normal; text-decoratio=
n: none; font-family: Helvetica,Arial,sans-serif; font-size:
12pt;"><b>From=
: </b>"Andrew Lau"
&lt;andrew(a)andrewklau.com&gt;<br><b>To: </b>"Omer Frenke=
l" &lt;ofrenkel(a)redhat.com&gt;<br><b>Cc: </b>"Dan
Kenigsberg" <danken@re=
dhat.com>, libvir-list(a)redhat.com, "users"
&lt;users(a)ovirt.org&gt;<br><b=
Sent: </b>Monday, September 16, 2013 1:38:53
AM<br><b>Subject: </b>Re: [Us=
ers] Live Migration failed oVirt
3.3 Nightly<br><div><br></div><div dir=3D"=
ltr"><div class=3D"gmail_default"
style=3D"font-family:tahoma,sans-serif" d=
ata-mce-style=3D"font-family: tahoma,sans-serif;"><span
style=3D"font-famil=
y:arial" data-mce-style=3D"font-family: arial;">On Sun, Sep 15, 2013 at
11:=
51 PM, Omer Frenkel </span><span dir=3D"ltr"
style=3D"font-family:arial" da=
ta-mce-style=3D"font-family: arial;"><<a
href=3D"mailto:ofrenkel@redhat.=
com" target=3D"_blank"
data-mce-href=3D"mailto:ofrenkel@redhat.com">ofrenke=
l(a)redhat.com</a>&gt;</span><span style=3D"font-family:arial"
data-mce-style=
=3D"font-family: arial;"> wrote:</span><br></div><div
class=3D"gmail_extra"=
<div class=3D"gmail_quote"><blockquote
class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,20=
4);border-left-style:solid;padding-left:1ex" data-mce-style=3D"margin: 0px =
0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-l=
eft-style: solid; padding-left: 1ex;"><div class=3D""><div
class=3D"h5"><br=
<br> ----- Original Message -----<br> > From:
"Dan Kenigsberg" <<a =
href=3D"mailto:danken@redhat.com" target=3D"_blank"
data-mce-href=3D"mailto=
:danken@redhat.com">danken@redhat.com</a>><br> > To:
"Andrew Lau" <=
;<a href=3D"mailto:andrew@andrewklau.com" target=3D"_blank"
data-mce-href=
=3D"mailto:andrew@andrewklau.com">andrew@andrewklau.com</a>><br>
> Cc=
: <a href=3D"mailto:libvir-list@redhat.com" target=3D"_blank"
data-mce-href=
=3D"mailto:libvir-list@redhat.com">libvir-list@redhat.com</a>,
"users" <=
<a href=3D"mailto:users@ovirt.org" target=3D"_blank"
data-mce-href=3D"mailt=
o:users@ovirt.org">users@ovirt.org</a>><br> > Sent:
Sunday, September=
15, 2013 3:47:03 PM<br> > Subject: Re: [Users] Live Migration failed oV=
irt 3.3 Nightly<br> ><br> > On Sun, Sep 15, 2013 at 09:57:47PM
+1000,=
Andrew Lau wrote:<br> > > On Sun, Sep 15, 2013 at 9:34 PM, Dan
Kenig=
sberg <<a href=3D"mailto:danken@redhat.com"
target=3D"_blank" data-mce-h=
ref=3D"mailto:danken@redhat.com">danken@redhat.com</a>>
wrote:<br> > =
><br> > > > On Sun, Sep 15, 2013 at 08:44:18PM +1000,
Andrew La=
u wrote:<br> > > > > On Sun, Sep 15, 2013 at 8:00 PM,
Dan Kenig=
sberg <<a href=3D"mailto:danken@redhat.com"
target=3D"_blank" data-mce-h=
ref=3D"mailto:danken@redhat.com">danken@redhat.com</a>><br>
> > &g=
t; wrote:<br> > > > ><br> > >
> > > On Sun, Sep =
15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:<br> > > >
> >=
; > Hi Dan,<br> > > > > >
><br> > > > > &g=
t; > Certainly, I've uploaded them to fedora's paste bin and tried
to<br=
> > > > > > snip<br>
> > > > > just<br> =
> >
> > > > the relevant details.<br> > >
> >=
> ><br> > > > > > > Sender
(<a href=3D"http://hv01=
.melb.domain.net" target=3D"_blank"
data-mce-href=3D"http://hv01.melb.domai=
n.net">hv01.melb.domain.net</a>):<br> > > >
> > > <a href=
=3D"http://paste.fedoraproject.org/39660/92339651/" target=3D"_blank"
data-=
mce-href=3D"http://paste.fedoraproject.org/39660/92339651/">...
doraproject.org/39660/92339651/</a><br> > > > >
><br> > &=
gt; > > > This one has<br> > > > >
><br> > > =
> > > libvirtError: operation failed: Failed to
conn=
ect to remote<br> > > > > >
libvirt<br> > &=
gt; > > > URI qemu+tls://<a
href=3D"http://hv02.melb.domain.net/sy=
stem" target=3D"_blank"
data-mce-href=3D"http://hv02.melb.domain.net/system=
">hv02.melb.domain.net/system</a><br> > > >
> ><br> > >=
; > > > which is most often related to firewall issues, and some t=
ime to key<br> > > > > > mismatch.<br>
> > > > &=
gt;<br> > > > > > Does<br> >
> > > > &=
nbsp; virsh -c qemu+tls://<a href=3D"http://hv02.melb.domain.net/system" ta=
rget=3D"_blank"
data-mce-href=3D"http://hv02.melb.domain.net/system">hv02.m=
elb.domain.net/system</a> capabilities<br> > > > >
> work wh=
en run from the command line of hv01?<br> > > > >
><br> >=
> > > > Dan.<br> > > > >
> > Receiver (<a hr=
ef=3D"http://hv02.melb.domain.net" target=3D"_blank"
data-mce-href=3D"http:=
//hv02.melb.domain.net">hv02.melb.domain.net</a>): `<br> >
> > >=
; > > <a
href=3D"http://paste.fedoraproject.org/39661/23406913/" targ=
et=3D"_blank"
data-mce-href=3D"http://paste.fedoraproject.org/39661/2340691=
3/">http://paste.fedoraproject.org/39661/23406913/</a><b... >
> > &=
gt; > ><br> > > > > > > VM
being transfered is ovir=
t_guest_vm<br> > > > > > ><br>
> > > > > &=
gt; Thanks,<br> > > > > > >
Andrew<br> > > > >=
; ><br> > > > ><br> > >
> > virsh -c qemu+tls://=
<a href=3D"http://hv02.melb.domain.net/system" target=3D"_blank"
data-mce-h=
ref=3D"http://hv02.melb.domain.net/system">hv02.melb.domain....
br> > > > > 2013-09-15 10:41:10.620+0000: 23994: info :
libvirt=
version: 0.10.2,<br> > > > > package: 18.el6_4.9
(CentOS Build=
System <<a href=3D"http://bugs.centos.org" target=3D"_blank"
data-mce-hr=
ef=3D"http://bugs.centos.org">http://bugs.centos.org</a>>,<br>
> >=
> > 2013-07-02-11:19:29, <a
href=3D"http://c6b8.bsys.dev.centos.org"=
target=3D"_blank"
data-mce-href=3D"http://c6b8.bsys.dev.centos.org">c6b8.b=
sys.dev.centos.org</a>)<br> > > > > 2013-09-15
10:41:10.620+000=
0: 23994: warning :<br> > > > >
virNetTLSContextCheckCertificat=
e:1102 : Certificate check failed<br> > > > >
Certificate faile=
d validation: The certificate hasn't got a known<br> > > >
> is=
suer.<br> > > ><br> > > > Would you
share your<br> > &=
gt; ><br> > > ><br> > > >
openssl x509 -in<br> > &g=
t; > /etc/pki/vdsm/certs/cacert.pem -text<br> > >
><br> > &g=
t; > openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text<br> >
&g=
t; ><br> > > > on both hosts? This content may be
sensitive, an=
d may not<br> > > > provide an answer why libvirt on src cannot
co=
ntact libvirtd on the<br> > > > other host. So before you do
that,=
would you test if<br> > > ><br> > >
><br> > > >=
vdsClient -s <a href=3D"http://hv02.melb.domain.net"
target=3D"_bla=
nk"
data-mce-href=3D"http://hv02.melb.domain.net">hv02.melb.domain.net</a>
=
getVdsCapabilities<br> > > ><br> > >
> works when run on =
hv01? It may be that the certificates are fine, but<br> > > >
libv=
irt is not configured to use the correct ones.<br> > >
><br> > =
> > Dan.<br> > > ><br> > >
><br> > > vdsClien=
t -s <a href=3D"http://hv02.melb.domain.net" target=3D"_blank"
data-mce-hre=
f=3D"http://hv02.melb.domain.net">hv02.melb.domain.net</a>
getVdsCapabiliti=
es runs fine<br> > ><br> > > I did a quick
comparison between t=
he files on both hosts, they seem to have<br> > > the right details
(=
host names, authority etc.)<br> > > cacert.pem matches<br>
> ><=
br> > > /etc/libvirt/libvirtd.conf<br> > ><br>
> > ca_fil=
e=3D"/etc/pki/vdsm/certs/cacert.pem"<br> > >
cert_file=3D"/etc/pki/vd=
sm/certs/vdsmcert.pem"<br> > >
key_file=3D"/etc/pki/vdsm/keys/vdsmkey=
.pem"<br> ><br> <br></div></div>this sounds a
little like<br> <a href=3D=
"https://bugzilla.redhat.com/show_bug.cgi?id=3D996146"
target=3D"_blank" da=
ta-mce-href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D996146&...
//bugzilla.redhat.com/show_bug.cgi?id=3D996146</a><br> <br> can you try
to =
restart libvirt (on both hosts just to be sure) and try again?<br><div clas=
s=3D"im"><br> > Maybe someone on libvir-list could guess why
this could =
be happening?<br></div>>
_______________________________________________=
<br> > Users mailing list<br> > <a
href=3D"mailto:Users@ovirt.org" ta=
rget=3D"_blank"
data-mce-href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a=
<br> > <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users" target=
=3D"_blank"
data-mce-href=3D"http://lists.ovirt.org/mailman/listinfo/users"=
http://lists.ovirt.org/mailman/listinfo/users</a><br>
><br></blockquote=
</div><br></div><div class=3D"gmail_extra"><div
class=3D"gmail_default" st=
yle=3D"font-family:tahoma,sans-serif" data-mce-style=3D"font-family:
tahoma=
,sans-serif;">I did try that already</div><div
class=3D"gmail_default" styl=
e=3D"font-family:tahoma,sans-serif" data-mce-style=3D"font-family:
tahoma,s=
ans-serif;"><br></div><div class=3D"gmail_default"
style=3D"font-family:tah=
oma,sans-serif" data-mce-style=3D"font-family:
tahoma,sans-serif;">service =
vdsmd restart</div><div class=3D"gmail_default"
style=3D"font-family:tahoma=
,sans-serif" data-mce-style=3D"font-family:
tahoma,sans-serif;"><br></div><=
div class=3D"gmail_default"><div
class=3D"gmail_default"><span style=3D"fon=
t-family: tahoma,sans-serif;" data-mce-style=3D"font-family: tahoma,sans-se=
rif;" face=3D"tahoma, sans-serif">[root@hv02 ~]# service vdsmd
restart</spa=
n></div><div class=3D"gmail_default"><span
style=3D"font-family: tahoma,san=
s-serif;" data-mce-style=3D"font-family: tahoma,sans-serif;"
face=3D"tahoma=
, sans-serif">Shutting down vdsm daemon: </span></div><div
class=3D"gm=
ail_default"><span style=3D"font-family: tahoma,sans-serif;"
data-mce-style=
=3D"font-family: tahoma,sans-serif;" face=3D"tahoma,
sans-serif">vdsm watch=
dog stop
&nb=
sp;
[=
OK ]</span></div><div
class=3D"gmail_default"><span style=3D"f=
ont-family: tahoma,sans-serif;" data-mce-style=3D"font-family: tahoma,sans-=
serif;" face=3D"tahoma, sans-serif">vdsm stop
&=
nbsp;
=
[ &nb=
sp;OK ]</span></div><div
class=3D"gmail_default"><span style=3D"font-=
family: tahoma,sans-serif;" data-mce-style=3D"font-family: tahoma,sans-seri=
f;" face=3D"tahoma, sans-serif">Starting configure libvirt to VDSM
...</spa=
n></div><div class=3D"gmail_default"><span
style=3D"font-family: tahoma,san=
s-serif;" data-mce-style=3D"font-family: tahoma,sans-serif;"
face=3D"tahoma=
, sans-serif">libvirt is already configured for
vdsm</span></div><div class=
=3D"gmail_default"><span style=3D"font-family:
tahoma,sans-serif;" data-mce=
-style=3D"font-family: tahoma,sans-serif;" face=3D"tahoma,
sans-serif">=3DD=
one configuring libvirt=3D</span></div><div
class=3D"gmail_default"><span s=
tyle=3D"font-family: tahoma,sans-serif;" data-mce-style=3D"font-family:
tah=
oma,sans-serif;" face=3D"tahoma, sans-serif">libvir: Network Filter
Driver =
error : Requested operation is not valid: nwfilter is in
use</span></div><d=
iv class=3D"gmail_default"><span style=3D"font-family:
tahoma,sans-serif;" =
data-mce-style=3D"font-family: tahoma,sans-serif;" face=3D"tahoma,
sans-ser=
if">Checking conflicts ...</span></div><div
class=3D"gmail_default"><span s=
tyle=3D"font-family: tahoma,sans-serif;" data-mce-style=3D"font-family:
tah=
oma,sans-serif;" face=3D"tahoma, sans-serif">SUCCESS: ssl configured to
tru=
e. No conflicts</span></div><div
class=3D"gmail_default"><span style=3D"fon=
t-family: tahoma,sans-serif;" data-mce-style=3D"font-family: tahoma,sans-se=
rif;" face=3D"tahoma, sans-serif">Starting up vdsm
daemon: </span></di=
v><div class=3D"gmail_default"><span style=3D"font-family:
tahoma,sans-seri=
f;" data-mce-style=3D"font-family: tahoma,sans-serif;" face=3D"tahoma,
sans=
-serif">vdsm start
=
 =
; [ OK
]</span></div><div st=
yle=3D"font-family:tahoma,sans-serif" data-mce-style=3D"font-family:
tahoma=
,sans-serif;"><br></div><div
style=3D"font-family:tahoma,sans-serif" data-m=
ce-style=3D"font-family: tahoma,sans-serif;"><br></div><div
style=3D"font-f=
amily:tahoma,sans-serif" data-mce-style=3D"font-family:
tahoma,sans-serif;"=
Migration still failed. Keep in mind, when I had oVirt 3.3 on these
nodes =
migration was working fine. Only when I upgraded to the nightly and it
pick=
ed up the new vdsm packages it started to fail.</div><div
style=3D"font-fam=
ily:tahoma,sans-serif" data-mce-style=3D"font-family:
tahoma,sans-serif;"><=
br></div></div><br></div></div></blockquote><div>can
you try to restart the=
libvirtd service itself? not
vdsm<br></div></div></body></html>
------=_Part_21768893_1437824270.1379317267664--
2:47 a.m.
New subject: [Users] Live Migration failed oVirt 3.3 Nightly
On Mon, Sep 16, 2013 at 5:41 PM, Omer Frenkel <ofrenkel(a)redhat.com> wrote:
------------------------------
*From: *"Andrew Lau" <andrew(a)andrewklau.com>
*To: *"Omer Frenkel" <ofrenkel(a)redhat.com>
*Cc: *"Dan Kenigsberg" <danken(a)redhat.com>, libvir-list(a)redhat.com,
"users" <users(a)ovirt.org>
*Sent: *Monday, September 16, 2013 1:38:53 AM
*Subject: *Re: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel <ofrenkel(a)redhat.com>wrote:
>
>
> ----- Original Message -----
> > From: "Dan Kenigsberg" <danken(a)redhat.com>
> > To: "Andrew Lau" <andrew(a)andrewklau.com>
> > Cc: libvir-list(a)redhat.com, "users" <users(a)ovirt.org>
> > Sent: Sunday, September 15, 2013 3:47:03 PM
> > Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
> >
> > On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
> > > On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg <danken(a)redhat.com>
> wrote:
> > >
> > > > On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
> > > > > On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg <
> danken(a)redhat.com>
> > > > wrote:
> > > > >
> > > > > > On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:
> > > > > > > Hi Dan,
> > > > > > >
> > > > > > > Certainly, I've uploaded them to fedora's paste
bin and tried
> to
> > > > > > > snip
> > > > > > just
> > > > > > > the relevant details.
> > > > > > >
> > > > > > > Sender (hv01.melb.domain.net):
> > > > > > > http://paste.fedoraproject.org/39660/92339651/
> > > > > >
> > > > > > This one has
> > > > > >
> > > > > > libvirtError: operation failed: Failed to connect to
remote
> > > > > > libvirt
> > > > > > URI qemu+tls://hv02.melb.domain.net/system
> > > > > >
> > > > > > which is most often related to firewall issues, and some
time
> to key
> > > > > > mismatch.
> > > > > >
> > > > > > Does
> > > > > > virsh -c
qemu+tls://hv02.melb.domain.net/systemcapabilities
> > > > > > work when run from the command line of hv01?
> > > > > >
> > > > > > Dan.
> > > > > > > Receiver (hv02.melb.domain.net): `
> > > > > > > http://paste.fedoraproject.org/39661/23406913/
> > > > > > >
> > > > > > > VM being transfered is ovirt_guest_vm
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Andrew
> > > > > >
> > > > >
> > > > > virsh -c qemu+tls://hv02.melb.domain.net/system
> > > > > 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version:
> 0.10.2,
> > > > > package: 18.el6_4.9 (CentOS BuildSystem
<http://bugs.centos.org>;,
> > > > > 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org)
> > > > > 2013-09-15 10:41:10.620+0000: 23994: warning :
> > > > > virNetTLSContextCheckCertificate:1102 : Certificate check failed
> > > > > Certificate failed validation: The certificate hasn't got a
known
> > > > > issuer.
> > > >
> > > > Would you share your
> > > >
> > > >
> > > > openssl x509 -in
> > > > /etc/pki/vdsm/certs/cacert.pem -text
> > > >
> > > > openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
> > > >
> > > > on both hosts? This content may be sensitive, and may not
> > > > provide an answer why libvirt on src cannot contact libvirtd on the
> > > > other host. So before you do that, would you test if
> > > >
> > > >
> > > > vdsClient -s hv02.melb.domain.net getVdsCapabilities
> > > >
> > > > works when run on hv01? It may be that the certificates are fine,
> but
> > > > libvirt is not configured to use the correct ones.
> > > >
> > > > Dan.
> > > >
> > > >
> > > vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
> > >
> > > I did a quick comparison between the files on both hosts, they seem
> to have
> > > the right details (host names, authority etc.)
> > > cacert.pem matches
> > >
> > > /etc/libvirt/libvirtd.conf
> > >
> > > ca_file="/etc/pki/vdsm/certs/cacert.pem"
> > > cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
> > > key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
> >
>
> this sounds a little like
> https://bugzilla.redhat.com/show_bug.cgi?id=996146
>
> can you try to restart libvirt (on both hosts just to be sure) and try
> again?
>
> > Maybe someone on libvir-list could guess why this could be happening?
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
I did try that already
service vdsmd restart
[root@hv02 ~]# service vdsmd restart
Shutting down vdsm daemon:
vdsm watchdog stop [ OK ]
vdsm stop [ OK ]
Starting configure libvirt to VDSM ...
libvirt is already configured for vdsm
=Done configuring libvirt=
libvir: Network Filter Driver error : Requested operation is not valid:
nwfilter is in use
Checking conflicts ...
SUCCESS: ssl configured to true. No conflicts
Starting up vdsm daemon:
vdsm start [ OK ]
Migration still failed. Keep in mind, when I had oVirt 3.3 on these nodes
migration was working fine. Only when I upgraded to the nightly and it
picked up the new vdsm packages it started to fail.
can you try to restart the libvirtd service itself? not vdsm
[root@hv01 ~]# service libvirtd restart
Stopping libvirtd daemon: libvirtd: libvirtd is managed by upstart and
started, use initctl instead
[root@hv01 ~]# initctl restart libvirtd
libvirtd start/running, process 4538
Migration was successful, thanks!
3881
days inactive
3882
days old
6 comments
3 participants
participants (3)
-
Andrew Lau -
Dan Kenigsberg -
Omer Frenkel