9 a.m.
No, it doesn't succeed neither.
I've also noticed that in the Gluster Swift secction it appears to me
everything as Not installed, except the memcached that appears as Down,
could this has something to do with all these?
The private chat was my bad I think, I've only answered to you in some
point, sorry!
Thanks!
2013/9/25 Dan Kenigsberg <danken(a)redhat.com>
On Wed, Sep 25, 2013 at 10:40:30AM -0300, emitor(a)gmail.com wrote:
> I've created a new debian VM. I've restarted the vdsm on the source and
> destination host and after try the migration now I received the following
> message:
>
> Migration failed due to Error: Fatal error during migration (VM:
> debian-test2, Source: ovirt1, Destination: ovirt2).
>
> Do you think that with a clean instalation of the host could this be
solved?
No.
We have a real bug on re-connecting to VMs on startup.
But why are we in a private chat? Please report it as a reply to the
mailing list thread.
But let me verify: if you start a VM and then migrate (no vdsm restart),
does migration succede?
Dan.
--
*Emiliano Tortorella*
+598 98941176
emitor(a)gmail.com
9:21 a.m.
New subject: [Users] Migration issues with ovirt 3.3
I've just tried to do a migration without vdsm restart and it works with
the debian-test2 VM.
I've also tried with the pfsense1 VM but it doesn't, might it be because
this vm has an IDE type disk?
Here goes the logs!
2013/9/25 emitor(a)gmail.com <emitor(a)gmail.com>
No, it doesn't succeed neither.
I've also noticed that in the Gluster Swift secction it appears to me
everything as Not installed, except the memcached that appears as Down,
could this has something to do with all these?
The private chat was my bad I think, I've only answered to you in some
point, sorry!
Thanks!
2013/9/25 Dan Kenigsberg <danken(a)redhat.com>
> On Wed, Sep 25, 2013 at 10:40:30AM -0300, emitor(a)gmail.com wrote:
> > I've created a new debian VM. I've restarted the vdsm on the source and
> > destination host and after try the migration now I received the
> following
> > message:
> >
> > Migration failed due to Error: Fatal error during migration (VM:
> > debian-test2, Source: ovirt1, Destination: ovirt2).
> >
> > Do you think that with a clean instalation of the host could this be
> solved?
> No.
>
> We have a real bug on re-connecting to VMs on startup.
>
> But why are we in a private chat? Please report it as a reply to the
> mailing list thread.
>
> But let me verify: if you start a VM and then migrate (no vdsm restart),
> does migration succede?
>
> Dan.
>
--
*Emiliano Tortorella*
+598 98941176
emitor(a)gmail.com
--
*Emiliano Tortorella*
+598 98941176
emitor(a)gmail.com
1:36 p.m.
New subject: [Users] Migration issues with ovirt 3.3
So Dan, could I do some other test for you?, The live migration it's only
working with a Debian install, I've just finished to install a windows
server 2K8 and it doesn't work neither, gives the following error "VM W2K8
is down. Exit message: 'iface'."
Regards!
2013/9/25 emitor(a)gmail.com <emitor(a)gmail.com>
I've just tried to do a migration without vdsm restart and it
works with
the debian-test2 VM.
I've also tried with the pfsense1 VM but it doesn't, might it be because
this vm has an IDE type disk?
Here goes the logs!
2013/9/25 emitor(a)gmail.com <emitor(a)gmail.com>
No, it doesn't succeed neither.
>
> I've also noticed that in the Gluster Swift secction it appears to me
> everything as Not installed, except the memcached that appears as Down,
> could this has something to do with all these?
>
> The private chat was my bad I think, I've only answered to you in some
> point, sorry!
>
> Thanks!
>
>
>
> 2013/9/25 Dan Kenigsberg <danken(a)redhat.com>
>
>> On Wed, Sep 25, 2013 at 10:40:30AM -0300, emitor(a)gmail.com wrote:
>> > I've created a new debian VM. I've restarted the vdsm on the source
and
>> > destination host and after try the migration now I received the
>> following
>> > message:
>> >
>> > Migration failed due to Error: Fatal error during migration (VM:
>> > debian-test2, Source: ovirt1, Destination: ovirt2).
>> >
>> > Do you think that with a clean instalation of the host could this be
>> solved?
>> No.
>>
>> We have a real bug on re-connecting to VMs on startup.
>>
>> But why are we in a private chat? Please report it as a reply to the
>> mailing list thread.
>>
>> But let me verify: if you start a VM and then migrate (no vdsm restart),
>> does migration succede?
>>
>> Dan.
>>
>
>
>
> --
> *Emiliano Tortorella*
> +598 98941176
> emitor(a)gmail.com
>
--
*Emiliano Tortorella*
+598 98941176
emitor(a)gmail.com
--
*Emiliano Tortorella*
+598 98941176
emitor(a)gmail.com
7:15 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On Fri, Sep 27, 2013 at 03:36:22PM -0300, emitor(a)gmail.com wrote:
So Dan, could I do some other test for you?, The live migration
it's only
working with a Debian install, I've just finished to install a windows
server 2K8 and it doesn't work neither, gives the following error "VM W2K8
is down. Exit message: 'iface'."
Frankly, I am out of ideas of what could be causing this very important
bug :-(
It is unrelated to hotplugging, and probably not to migration -
something bad happens when the VM parameters are persisted to disk, or
loaded from it.
I do not see how the type of the guest could matter. Could you tell if
there is any other difference betrween your W2K8 VM and your Debian one?
Could you post here the first vmCreate command of the two VMs, and their
libvirt domxml?
Regards,
Dan.
9:19 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On Sep 29, 2013, at 14:15 , Dan Kenigsberg <danken(a)redhat.com> wrote:
On Fri, Sep 27, 2013 at 03:36:22PM -0300, emitor(a)gmail.com wrote:
> So Dan, could I do some other test for you?, The live migration it's only
> working with a Debian install, I've just finished to install a windows
> server 2K8 and it doesn't work neither, gives the following error "VM W2K8
> is down. Exit message: 'iface'."
Do you have any non-gluster setup to give it a try?
We have this bug 1007980 which also seems to demonstrate in gluster-setup only….don't
know why yet, but perhaps this can be related….
Thanks,
michal
Frankly, I am out of ideas of what could be causing this very important
bug :-(
It is unrelated to hotplugging, and probably not to migration -
something bad happens when the VM parameters are persisted to disk, or
loaded from it.
I do not see how the type of the guest could matter. Could you tell if
there is any other difference betrween your W2K8 VM and your Debian one?
Could you post here the first vmCreate command of the two VMs, and their
libvirt domxml?
Regards,
Dan.
12:35 p.m.
New subject: [Users] Migration issues with ovirt 3.3
Dan, which are the path to those files that you're asking to me?
Now the debian vm it's not working. After try the migration of the Windows
vm, the live migration it's not working in the debian.
Do you think that this issue it's related to the hosts or with the manager?
i really would like to get this working and to other user this is working
fine. Do you think if that i make a whole new install maybe this start to
work?
I'm going to test what Michal say right now with the same host, but i think
that's going to happen the same thing.
Regards!
2013/9/30 Michal Skrivanek <mskrivan(a)redhat.com>
On Sep 29, 2013, at 14:15 , Dan Kenigsberg <danken(a)redhat.com> wrote:
> On Fri, Sep 27, 2013 at 03:36:22PM -0300, emitor(a)gmail.com wrote:
>> So Dan, could I do some other test for you?, The live migration it's
only
>> working with a Debian install, I've just finished to install a windows
>> server 2K8 and it doesn't work neither, gives the following error "VM
W2K8
>> is down. Exit message: 'iface'."
Do you have any non-gluster setup to give it a try?
We have this bug 1007980 which also seems to demonstrate in gluster-setup
only….don't know why yet, but perhaps this can be related….
Thanks,
michal
>
> Frankly, I am out of ideas of what could be causing this very important
> bug :-(
>
> It is unrelated to hotplugging, and probably not to migration -
> something bad happens when the VM parameters are persisted to disk, or
> loaded from it.
>
> I do not see how the type of the guest could matter. Could you tell if
> there is any other difference betrween your W2K8 VM and your Debian one?
>
> Could you post here the first vmCreate command of the two VMs, and their
> libvirt domxml?
>
> Regards,
> Dan.
>
>
--
*Emiliano Tortorella*
+598 98941176
emitor(a)gmail.com
3:54 p.m.
New subject: [Users] Migration issues with ovirt 3.3
On Tue, Oct 01, 2013 at 02:35:34PM -0300, emitor(a)gmail.com wrote:
Dan, which are the path to those files that you're asking to me?
Now the debian vm it's not working. After try the migration of the Windows
vm, the live migration it's not working in the debian.
The fact that now both VMs behave the same makes comparing them
fruitless. Still it may be intersting to see your vdsm.log (from source
and destination) since the first time a faulty VM was started until the
'iface' exception.
Do you think that this issue it's related to the hosts or with the manager?
It smells very much Vdsm-specific, but I'm only guessing.
i really would like to get this working and to other user this is
working
fine. Do you think if that i make a whole new install maybe this start to
work?
I do not suppose this would help, but I'm not in the position to exclude
exploration routes.
I'm going to test what Michal say right now with the same host, but i think
that's going to happen the same thing.
Regards!
5:07 p.m.
New subject: [Users] Migration issues with ovirt 3.3
I'm having this issue on my ovirt 3.3 setup (two node, one is AIO,
GlusterFS storage, both on F19) as well.
Jason
On Tue, 2013-10-01 at 21:54 +0100, Dan Kenigsberg wrote:
On Tue, Oct 01, 2013 at 02:35:34PM -0300, emitor(a)gmail.com wrote:
> Dan, which are the path to those files that you're asking to me?
>
> Now the debian vm it's not working. After try the migration of the Windows
> vm, the live migration it's not working in the debian.
The fact that now both VMs behave the same makes comparing them
fruitless. Still it may be intersting to see your vdsm.log (from source
and destination) since the first time a faulty VM was started until the
'iface' exception.
>
> Do you think that this issue it's related to the hosts or with the manager?
It smells very much Vdsm-specific, but I'm only guessing.
> i really would like to get this working and to other user this is working
> fine. Do you think if that i make a whole new install maybe this start to
> work?
I do not suppose this would help, but I'm not in the position to exclude
exploration routes.
>
> I'm going to test what Michal say right now with the same host, but i think
> that's going to happen the same thing.
>
> Regards!
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
11:36 a.m.
New subject: [Users] Migration issues with ovirt 3.3
I've configured the host as a NFS cluster. I've imported the VMs, the
debian now it's migrating without problems, the pfSense now gives me an
error (Migration failed due to Error: Desktop does not exist) and the
Windows VM doesn't start in the new cluster after import it.
I've attached the logs of the vdsm
The other thing that's not working for me is the HA, even of the Debian VM,
I don't know if ti has something to do with all of this.
Now I'm going to reinstall the host and the engine using the stable
repositories, I hope to get some more stability and get the HA working.
Regards!
2013/10/1 Jason Brooks <jbrooks(a)redhat.com>
I'm having this issue on my ovirt 3.3 setup (two node, one is
AIO,
GlusterFS storage, both on F19) as well.
Jason
On Tue, 2013-10-01 at 21:54 +0100, Dan Kenigsberg wrote:
> On Tue, Oct 01, 2013 at 02:35:34PM -0300, emitor(a)gmail.com wrote:
> > Dan, which are the path to those files that you're asking to me?
> >
> > Now the debian vm it's not working. After try the migration of the
Windows
> > vm, the live migration it's not working in the debian.
>
> The fact that now both VMs behave the same makes comparing them
> fruitless. Still it may be intersting to see your vdsm.log (from source
> and destination) since the first time a faulty VM was started until the
> 'iface' exception.
>
> >
> > Do you think that this issue it's related to the hosts or with the
manager?
>
> It smells very much Vdsm-specific, but I'm only guessing.
>
> > i really would like to get this working and to other user this is
working
> > fine. Do you think if that i make a whole new install maybe this start
to
> > work?
>
> I do not suppose this would help, but I'm not in the position to exclude
> exploration routes.
>
> >
> > I'm going to test what Michal say right now with the same host, but i
think
> > that's going to happen the same thing.
> >
> > Regards!
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
*Emiliano Tortorella*
+598 98941176
emitor(a)gmail.com
12:28 p.m.
New subject: [Users] Migration issues with ovirt 3.3
On Wed, Oct 02, 2013 at 01:36:56PM -0300, emitor(a)gmail.com wrote:
I've configured the host as a NFS cluster. I've imported the
VMs, the
debian now it's migrating without problems, the pfSense now gives me an
error (Migration failed due to Error: Desktop does not exist) and the
Windows VM doesn't start in the new cluster after import it.
I've attached the logs of the vdsm
Thanks for providing your, but they begin too late.
The bug is still a mystery to me. Something garbles Vdsm's internal
representation of virtual disks. And it's bad.
I would appreciate if you could dig backward in your logs and find the
last successfull startup of a VM, say W2K8_cluster1
(86f10137-d917-4508-a07a-3f83e6f24a95). Then, browse forward to find the
first failure (be it on next startup of VM or during migration).
Best regards,
Dan.
12:55 p.m.
New subject: [Users] Migration issues with ovirt 3.3
Dan, I've made a whole new installation of oVirt 3.3 using the stable
repos, I've updated libvirt to 1.1.2-1 and configured a glusterFS cluster.
I've just tried the live migration of the Windows Host, here i send you the
vdsm, libvir and qemu logs
2013/10/4 Dan Kenigsberg <danken(a)redhat.com>
On Wed, Oct 02, 2013 at 01:36:56PM -0300, emitor(a)gmail.com wrote:
> I've configured the host as a NFS cluster. I've imported the VMs, the
> debian now it's migrating without problems, the pfSense now gives me an
> error (Migration failed due to Error: Desktop does not exist) and the
> Windows VM doesn't start in the new cluster after import it.
>
> I've attached the logs of the vdsm
Thanks for providing your, but they begin too late.
The bug is still a mystery to me. Something garbles Vdsm's internal
representation of virtual disks. And it's bad.
I would appreciate if you could dig backward in your logs and find the
last successfull startup of a VM, say W2K8_cluster1
(86f10137-d917-4508-a07a-3f83e6f24a95). Then, browse forward to find the
first failure (be it on next startup of VM or during migration).
Best regards,
Dan.
--
*Emiliano Tortorella*
+598 98941176
emitor(a)gmail.com
2:30 p.m.
New subject: [Users] Migration issues with ovirt 3.3
--Apple-Mail=_80349546-76AA-4918-9B45-6194CB54F8F2
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=windows-1252
On 4 Oct 2013, at 19:55, emitor(a)gmail.com wrote:
Dan, I've made a whole new installation of oVirt 3.3 using the
stable =
repos, I've updated libvirt to 1.1.2-1 and configured a glusterFS =
cluster.
=20
I've just tried the live migration of the Windows Host, here i send =
you the
vdsm, libvir and qemu logs
hmm=85I see both PCI address for CDROM as we've been seeing on gluster =
setups. So far no update from gluster AFAIK=85it seems to be broken on =
their side somewhere=85
and I also see error on migrations dst which seems to be =
another/different disk device mess up
The W2K8 create VM call in the ovirt1's log is already wrong, can you =
capture/include the initial create call? In the previous cases it was =
always the case that the VM was created(from engine) with IDE address =
but then after the first run it somehow changed to PCI on the first =
status update from vdsm
Thanks,
michal
=20
=20
2013/10/4 Dan Kenigsberg <danken(a)redhat.com>
On Wed, Oct 02, 2013 at 01:36:56PM -0300, emitor(a)gmail.com wrote:
> I've configured the host as a NFS cluster. I've imported the VMs, =
the
> debian now it's migrating without problems, the pfSense now
gives me =
an
> error (Migration failed due to Error: Desktop does not exist)
and =
the
> Windows VM doesn't start in the new cluster after import
it.
>
> I've attached the logs of the vdsm
=20
Thanks for providing your, but they begin too late.
=20
The bug is still a mystery to me. Something garbles Vdsm's internal
representation of virtual disks. And it's bad.
=20
I would appreciate if you could dig backward in your logs and find the
last successfull startup of a VM, say W2K8_cluster1
(86f10137-d917-4508-a07a-3f83e6f24a95). Then, browse forward to find =
the
first failure (be it on next startup of VM or during migration).
=20
Best regards,
Dan.
=20
=20
=20
--=20
Emiliano Tortorella
+598 98941176
emitor(a)gmail.com
=
<ovirt_logs_clusterGFS.tar.gz>____________________________________________=
___
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_80349546-76AA-4918-9B45-6194CB54F8F2
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=windows-1252
<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><br><div><div>On 4 Oct 2013, at 19:55, <a =
href=3D"mailto:emitor@gmail.com">emitor@gmail.com</a>
wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote
type=3D"cite"><div =
dir=3D"ltr">Dan, I've made a whole new installation of oVirt 3.3 using =
the stable repos, I've updated libvirt to 1.1.2-1 and configured a =
glusterFS cluster.<div><br></div><div>I've just tried the live
migration =
of the Windows Host, here i send you the vdsm, libvir and qemu =
logs</div></div></blockquote><div><br></div>hmm=85I
see both PCI address =
for CDROM as we've been seeing on gluster setups. So far no update from =
gluster AFAIK=85it seems to be broken on their side =
somewhere=85</div><div>and I also see error on migrations dst which =
seems to be another/different disk device mess =
up</div><div><div><br></div><div>The W2K8 create VM
call in the ovirt1's =
log is already wrong, can you capture/include the initial create call? =
In the previous cases it was always the case that the VM was =
created(from engine) with IDE address but then after the first run it =
somehow changed to PCI on the first status update from =
vdsm</div><div><br></div><div>Thanks,</div><div>michal</div><div><br></div=
<blockquote type=3D"cite"><div
dir=3D"ltr">
</div><div
class=3D"gmail_extra"><br><br><div =
class=3D"gmail_quote">2013/10/4 Dan Kenigsberg <span
dir=3D"ltr"><<a =
href=3D"mailto:danken@redhat.com" =
target=3D"_blank">danken(a)redhat.com</a>&gt;</span><br><blockquote
=
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc =
solid;padding-left:1ex">
<div class=3D"im">On Wed, Oct 02, 2013 at 01:36:56PM -0300, <a =
href=3D"mailto:emitor@gmail.com">emitor@gmail.com</a> wrote:<br>
> I've configured the host as a NFS cluster. I've imported the VMs, =
the<br>
> debian now it's migrating without problems, the pfSense now gives =
me an<br>
> error (Migration failed due to Error: Desktop does not exist) and =
the<br>
> Windows VM doesn't start in the new cluster after import it.<br>
><br>
> I've attached the logs of the vdsm<br>
<br>
</div>Thanks for providing your, but they begin too late.<br>
<br>
The bug is still a mystery to me. Something garbles Vdsm's internal<br>
representation of virtual disks. And it's bad.<br>
<br>
I would appreciate if you could dig backward in your logs and find =
the<br>
last successfull startup of a VM, say W2K8_cluster1<br>
(86f10137-d917-4508-a07a-3f83e6f24a95). Then, browse forward to find =
the<br>
first failure (be it on next startup of VM or during migration).<br>
<br>
Best regards,<br>
Dan.<br>
</blockquote></div><br><br
clear=3D"all"><div><br></div>-- <br><div =
dir=3D"ltr"><b>Emiliano Tortorella</b><br>+598
98941176<br><a =
href=3D"mailto:emitor@gmail.com" =
target=3D"_blank">emitor(a)gmail.com</a></div>
</div>
=
<span><ovirt_logs_clusterGFS.tar.gz></span>_________________________=
______________________<br>Users mailing list<br><a =
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br>http://lists.ovirt.=
org/mailman/listinfo/users<br></blockquote></div><br></body></html>=
--Apple-Mail=_80349546-76AA-4918-9B45-6194CB54F8F2--
5:25 p.m.
New subject: [Users] Migration issues with ovirt 3.3
I've done some further tests and comparisons
In this F19 based + ovirt stable repo the difference in qemu command
line between the working run once with cd enabled of this CentOS 6.4
x86_64 and the normal run that doesn't proceed is:
run once
-drive
file=/rhev/data-center/mnt/f18engine.mydomain:_var_lib_exports_iso/cc790a86-72e2-4aa2-a0b6-700756c399c3/images/11111111-1111-1111-1111-111111111111/CentOS-6.4-x86_64-bin-DVD1.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=2
run
-drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
A working normal run of a WinXP 32bit VM in another oVirt 3.2.3 environment is
-drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
so it is the same as in 3.3.
Possible problems:
- 64bit OS
I can try tomorrow to create a CentOS 6.4 32 bit and compare results
- qemu default emulated pc version
Some comparisons
a) f18 host (AIO) with oVirt 3.2.3
qemu-kvm-1.2.2-14.fc18.x86_64
[root@tekkaman polimi]# qemu-kvm -machine ?
Supported machines are:
none empty machine
pc Standard PC (alias of pc-1.2)
pc-1.2 Standard PC (default)
pc-1.1 Standard PC
pc-1.0 Standard PC
pc-0.15 Standard PC
pc-0.14 Standard PC
pc-0.13 Standard PC
pc-0.12 Standard PC
pc-0.11 Standard PC, qemu 0.11
pc-0.10 Standard PC, qemu 0.10
isapc ISA-only PC
When I start WinXP (32bit of course) I see this command line:
/usr/bin/qemu-kvm -name winxp -S -M pc-0.14 -cpu Opteron_G3 ....
b) f19 host in oVirt 3.3
qemu-kvm-1.4.2-11.fc19.x86_64
[root@f18ovn03 ~]# qemu-kvm -machine ?
Supported machines are:
none empty machine
pc Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-1.4)
pc-i440fx-1.4 Standard PC (i440FX + PIIX, 1996) (default)
pc-1.3 Standard PC
pc-1.2 Standard PC
pc-1.1 Standard PC
pc-1.0 Standard PC
pc-0.15 Standard PC
pc-0.14 Standard PC
pc-0.13 Standard PC
pc-0.12 Standard PC
pc-0.11 Standard PC, qemu 0.11
pc-0.10 Standard PC, qemu 0.10
isapc ISA-only PC
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-1.4)
pc-q35-1.4 Standard PC (Q35 + ICH9, 2009)
qemu command line for a CentOS 6.4 64bit vm is:
/usr/bin/qemu-system-x86_64 -machine accel=kvm -name c6s -S -machine
pc-1.0,accel=kvm,usb=off -cpu Opteron_G2
both /usr/bin/qemu-kvm and /usr/bin/qemu-system-x86_64 are in
qemu-system-x86 rpm package.
Questions:
- given a version of oVirt, pc version in qemu generated command line
is the same for every defined VM or is there any logic depending on OS
chosen and/or bit (32 vs 64)?
- if answer to previous question is fixed pc version, is there any
particular reason to use pc-0.14 in oVirt 3.2.3 and pc-1.0 in oVirt
3.3 and not the latest available (so pc-1.2 in 3.2.3 and pc-1.3 or
pc-i440fx-1.4 in 3.3)?
- where can I find documentation about difference between the
different pc-x.y versions?
- can I use a wrapper in oVirt 3.3 to start VM with a different pc-x.y
(eg pc-0.14) and see if anything changes helping debug and solve the
problem?
Thanks,
Gianluca
5:47 p.m.
New subject: [Users] Migration issues with ovirt 3.3
On Mon, Oct 07, 2013 at 12:25:30AM +0200, Gianluca Cecchi wrote:
I've done some further tests and comparisons
Is this related to emitor's bug? For emitor, when a VM does not start it
happens when libvirt recongnizes that its devices have bad addresses. It
never gets to qemu command line phase.
In this F19 based + ovirt stable repo the difference in qemu command
line between the working run once with cd enabled of this CentOS 6.4
x86_64 and the normal run that doesn't proceed is:
run once
-drive
file=/rhev/data-center/mnt/f18engine.mydomain:_var_lib_exports_iso/cc790a86-72e2-4aa2-a0b6-700756c399c3/images/11111111-1111-1111-1111-111111111111/CentOS-6.4-x86_64-bin-DVD1.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=2
run
-drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
A working normal run of a WinXP 32bit VM in another oVirt 3.2.3 environment is
-drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
so it is the same as in 3.3.
Possible problems:
- 64bit OS
I can try tomorrow to create a CentOS 6.4 32 bit and compare results
- qemu default emulated pc version
Some comparisons
a) f18 host (AIO) with oVirt 3.2.3
qemu-kvm-1.2.2-14.fc18.x86_64
[root@tekkaman polimi]# qemu-kvm -machine ?
Supported machines are:
none empty machine
pc Standard PC (alias of pc-1.2)
pc-1.2 Standard PC (default)
pc-1.1 Standard PC
pc-1.0 Standard PC
pc-0.15 Standard PC
pc-0.14 Standard PC
pc-0.13 Standard PC
pc-0.12 Standard PC
pc-0.11 Standard PC, qemu 0.11
pc-0.10 Standard PC, qemu 0.10
isapc ISA-only PC
When I start WinXP (32bit of course) I see this command line:
/usr/bin/qemu-kvm -name winxp -S -M pc-0.14 -cpu Opteron_G3 ....
b) f19 host in oVirt 3.3
qemu-kvm-1.4.2-11.fc19.x86_64
[root@f18ovn03 ~]# qemu-kvm -machine ?
Supported machines are:
none empty machine
pc Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-1.4)
pc-i440fx-1.4 Standard PC (i440FX + PIIX, 1996) (default)
pc-1.3 Standard PC
pc-1.2 Standard PC
pc-1.1 Standard PC
pc-1.0 Standard PC
pc-0.15 Standard PC
pc-0.14 Standard PC
pc-0.13 Standard PC
pc-0.12 Standard PC
pc-0.11 Standard PC, qemu 0.11
pc-0.10 Standard PC, qemu 0.10
isapc ISA-only PC
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-1.4)
pc-q35-1.4 Standard PC (Q35 + ICH9, 2009)
qemu command line for a CentOS 6.4 64bit vm is:
/usr/bin/qemu-system-x86_64 -machine accel=kvm -name c6s -S -machine
pc-1.0,accel=kvm,usb=off -cpu Opteron_G2
both /usr/bin/qemu-kvm and /usr/bin/qemu-system-x86_64 are in
qemu-system-x86 rpm package.
Questions:
- given a version of oVirt, pc version in qemu generated command line
is the same for every defined VM or is there any logic depending on OS
chosen and/or bit (32 vs 64)?
emulatedMachine is set by the effective
clusterLevel.
- if answer to previous question is fixed pc version, is there any
particular reason to use pc-0.14 in oVirt 3.2.3 and pc-1.0 in oVirt
3.3 and not the latest available (so pc-1.2 in 3.2.3 and pc-1.3 or
pc-i440fx-1.4 in 3.3)?
We promise inter-cluster migration of VMs. Hence, 3.2.3 may use only the
newest machine available at 3.2.0.
- where can I find documentation about difference between the
different pc-x.y versions?
I am not aware of anything beyond the qemu source code and release
notes.
- can I use a wrapper in oVirt 3.3 to start VM with a different
pc-x.y
(eg pc-0.14) and see if anything changes helping debug and solve the
problem?
Yes, you may write a vdsm before_vm_start hook. It should search for
the 'machine' attribute and set it to whatever value you please.
Dan.
7:59 p.m.
New subject: [Users] Migration issues with ovirt 3.3
On Fri, Oct 04, 2013 at 02:55:42PM -0300, emitor(a)gmail.com wrote:
Dan, I've made a whole new installation of oVirt 3.3 using the
stable
repos, I've updated libvirt to 1.1.2-1 and configured a glusterFS cluster.
I've just tried the live migration of the Windows Host, here i send you the
vdsm, libvir and qemu logs
Here goes.
vm.Vm._getUnderlyingDriveInfo() matches the list of disks provided by
Engine with the list of disks reported back by libvirt. After a match
has been found, the libvirt-assigned address of each disk is reported to
Engine.
_getUnderlyingDriveInfo is awfully naive. If the cdrom is empty, the
function matches it with the gluster volume, and overides the cdrom's
address with the gluster volume's.
Would you please test if http://gerrit.ovirt.org/19906 solves the
issue? (I haven't. Too late at night.)
Regards,
Dan.
6:44 p.m.
New subject: [Users] Migration issues with ovirt 3.3
On Mon, Oct 7, 2013 at 2:59 AM, Dan Kenigsberg wrote:
Would you please test if http://gerrit.ovirt.org/19906 solves the
issue? (I haven't. Too late at night.)
Regards,
Dan.
I can confirm that it resolves
https://bugzilla.redhat.com/show_bug.cgi?id=1007980
so now I'm able to start VM without having to select run once and
attaching a cd iso
(note that is only valid for newly created VMs though)
But migration still fails
On engine:
2013-10-08 01:20:51,459 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-15) START, GlusterV
olumesListVDSCommand(HostName = f18ovn03, HostId =
b67bcfd4-f868-49d5-8704-4936ee922249), log id: 4425a532
2013-10-08 01:20:51,547 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-15) FINISH, Gluster
VolumesListVDSCommand, return:
{ed71a4c2-6205-4aad-9aab-85da086d5ba3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8defb13f
,
c8cbcac7-1d40-4cee-837d-bb97467fb2bd=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@91be2b8a},
log id: 4425a532
2013-10-08 01:20:51,697 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-39) RefreshVmList vm id d54660a2-45
ed-41ae-ab99-a6f93ebbdbb1 is migrating to vds f18ovn01 ignoring it in
the refresh until migration is done
2013-10-08 01:20:54,765 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand]
(pool-6-thread-38) START, GlusterServicesListVDSC
ommand(HostName = f18ovn01, HostId =
80188ccc-83b2-4bc8-9385-8d07f7458a3c), log id: 41ed75a4
2013-10-08 01:20:54,766 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand]
(pool-6-thread-36) START, GlusterServicesListVDSC
ommand(HostName = f18ovn03, HostId =
b67bcfd4-f868-49d5-8704-4936ee922249), log id: 7eea38bc
2013-10-08 01:20:56,618 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-2) START, GlusterVo
lumesListVDSCommand(HostName = f18ovn01, HostId =
80188ccc-83b2-4bc8-9385-8d07f7458a3c), log id: 34d0827b
2013-10-08 01:20:56,722 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand]
(DefaultQuartzScheduler_Worker-84) Command ListVDS execution failed.
Exception: VDSNetworkException: java.net.SocketException: Connection
reset
2013-10-08 01:20:56,725 ERROR
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand]
(pool-6-thread-36) Command GlusterServicesListVDS execution failed.
Exception: VDSNetworkException: java.net.SocketException: Connection
reset
2013-10-08 01:20:56,725 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand]
(pool-6-thread-36) FINISH, GlusterServicesListVDSCommand, log id:
7eea38bc
2013-10-08 01:20:56,727 WARN
[org.ovirt.engine.core.vdsbroker.VdsManager]
(DefaultQuartzScheduler_Worker-84) Failed to refresh VDS , vds =
b67bcfd4-f868-49d5-8704-4936ee922249 : f18ovn03, VDS Network Error,
continuing.
java.net.SocketException: Connection reset
2013-10-08 01:20:58,583 INFO
[org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand]
(DefaultQuartzScheduler_Worker-48) Running command:
SetStoragePoolStatusCommand internal: true. Entities affected : ID:
eb679feb-4da2-4fd0-a185-abbe459ffa70 Type: StoragePool
2013-10-08 01:20:58,603 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-48) Correlation ID: 16f779a3, Call
Stack: null, Custom Event ID: -1, Message: Invalid status on Data
Center Gluster. Setting status to Non Responsive.
2013-10-08 01:20:58,701 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-48) Command
org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand
return value
TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654,
mMessage=Not SPM: ()]]
2013-10-08 01:20:58,714 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-48) HostName = f18ovn03
2013-10-08 01:20:58,714 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-48) Command HSMGetAllTasksStatusesVDS
execution failed. Exception: IRSNonOperationalException:
IRSGenericException: IRSErrorException: IRSNonOperationalException:
Not SPM: ()
2013-10-08 01:20:59,356 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand]
(DefaultQuartzScheduler_Worker-41) Command ListVDS execution failed.
Exception: VDSNetworkException: java.net.SocketException: Connection
reset
On host where VM was running vdsm.log contains:
Thread-4643::DEBUG::2013-10-08
01:20:53,700::BindingXMLRPC::979::vds::(wrapper) client
[10.4.4.60]::call vmMigrate with ({'tunneled': 'false', 'dst':
'10.4.4.58:54321', 'vmId':
'd54660a2-45ed-41ae-ab99-a6f93ebbdbb1',
'method': 'online', 'src': '10.4.4.59'},) {} flowID
[7c55a4a9]
Thread-4643::DEBUG::2013-10-08 01:20:53,700::API::462::vds::(migrate)
{'tunneled': 'false', 'dst': '10.4.4.58:54321',
'vmId':
'd54660a2-45ed-41ae-ab99-a6f93ebbdbb1', 'method': 'online',
'src':
'10.4.4.59'}
Thread-4643::DEBUG::2013-10-08
01:20:53,701::BindingXMLRPC::986::vds::(wrapper) return vmMigrate with
{'status': {'message': 'Migration in progress', 'code':
0},
'progress': 0}
Thread-4644::DEBUG::2013-10-08
01:20:53,702::vm::182::vm.Vm::(_setupVdsConnection)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Destination server is:
10.4.4.58:54321
Thread-4644::DEBUG::2013-10-08
01:20:53,703::vm::184::vm.Vm::(_setupVdsConnection)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Initiating connection
with destination
Thread-4644::DEBUG::2013-10-08
01:20:53,718::vm::238::vm.Vm::(_prepareGuest)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Migration started
Thread-4644::DEBUG::2013-10-08 01:20:53,739::vm::302::vm.Vm::(run)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration semaphore
acquired
Thread-4644::DEBUG::2013-10-08
01:20:53,933::vm::360::vm.Vm::(_startUnderlyingMigration)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::starting migration to
qemu+tls://10.4.4.58/system with miguri tcp://10.4.4.58
Thread-4645::DEBUG::2013-10-08 01:20:53,934::vm::718::vm.Vm::(run)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration downtime thread
started
Thread-4646::DEBUG::2013-10-08 01:20:53,935::vm::756::vm.Vm::(run)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::starting migration
monitor thread
Thread-4648::DEBUG::2013-10-08
01:20:54,321::BindingXMLRPC::979::vds::(wrapper) client
[10.4.4.60]::call volumesList with () {}
Thread-4648::DEBUG::2013-10-08
01:20:54,349::BindingXMLRPC::986::vds::(wrapper) return volumesList
with {'status': {'message': 'Done', 'code': 0},
'volumes': {'gviso':
{'transportType': ['TCP'], 'uuid':
'c8cbcac7-1d40-4cee-837d-bb97467fb2bd', 'bricks':
['f18ovn01.mydomain:/gluster/ISO_GLUSTER/brick1',
'f18ovn03.mydomain:/gluster/ISO_GLUSTER/brick1'], 'volumeName':
'gviso', 'volumeType': 'REPLICATE', 'replicaCount':
'2', 'brickCount':
'2', 'distCount': '2', 'volumeStatus': 'ONLINE',
'stripeCount': '1',
'options': {'storage.owner-gid': '36',
'storage.owner-uid': '36',
'server.allow-insecure': 'on'}}, 'gvdata':
{'transportType': ['TCP'],
'uuid': 'ed71a4c2-6205-4aad-9aab-85da086d5ba3', 'bricks':
['f18ovn01.mydomain:/gluster/DATA_GLUSTER/brick1',
'f18ovn03.mydomain:/gluster/DATA_GLUSTER/brick1'], 'volumeName':
'gvdata', 'volumeType': 'REPLICATE', 'replicaCount':
'2',
'brickCount': '2', 'distCount': '2',
'volumeStatus': 'ONLINE',
'stripeCount': '1', 'options': {'server.allow-insecure':
'on',
'storage.owner-uid': '36', 'storage.owner-gid': '36'}}}}
Thread-4644::ERROR::2013-10-08
01:20:54,873::libvirtconnection::94::libvirtconnection::(wrapper)
connection to libvirt broken. ecode: 38 edom: 7
Thread-4644::ERROR::2013-10-08
01:20:54,873::libvirtconnection::96::libvirtconnection::(wrapper)
taking calling process down.
MainThread::DEBUG::2013-10-08
01:20:54,874::vdsm::45::vds::(sigtermHandler) Received signal 15
Thread-4644::DEBUG::2013-10-08 01:20:54,874::vm::733::vm.Vm::(cancel)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::canceling migration
downtime thread
Thread-4644::DEBUG::2013-10-08 01:20:54,875::vm::803::vm.Vm::(stop)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::stopping migration
monitor thread
Thread-4645::DEBUG::2013-10-08 01:20:54,875::vm::730::vm.Vm::(run)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration downtime thread
exiting
Thread-4644::ERROR::2013-10-08
01:20:54,875::vm::244::vm.Vm::(_recover)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Cannot recv data:
Input/output error
Thread-4644::ERROR::2013-10-08 01:20:55,008::vm::324::vm.Vm::(run)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Failed to migrate
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 311, in run
self._startUnderlyingMigration()
File "/usr/share/vdsm/vm.py", line 388, in _startUnderlyingMigration
None, maxBandwidth)
File "/usr/share/vdsm/vm.py", line 826, in f
ret = attr(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.py",
line 76, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1253, in
migrateToURI2
if ret == -1: raise libvirtError ('virDomainMigrateToURI2()
failed', dom=self)
libvirtError: Cannot recv data: Input/output error
MainThread::INFO::2013-10-08
01:20:55,321::vmChannels::183::vds::(stop) VM channels listener was
stopped.
MainThread::INFO::2013-10-08 01:20:55,322::momIF::81::MOM::(stop)
Shutting down MOM
MainThread::DEBUG::2013-10-08
01:20:55,322::task::579::TaskManager.Task::(_updateState)
Task=`23df8b46-c916-4c52-98a9-3e6540171467`::moving from state init ->
state preparing
MainThread::INFO::2013-10-08
01:20:55,323::logUtils::44::dispatcher::(wrapper) Run and protect:
prepareForShutdown(options=None)
Thread-11::DEBUG::2013-10-08
01:20:55,323::storageServer::698::ConnectionMonitor::(_monitorConnections)
Monitoring stopped
VM Channels Listener::INFO::2013-10-08
01:20:55,672::vmChannels::178::vds::(run) VM channels listener thread
has ended.
Thread-23::DEBUG::2013-10-08
01:20:55,929::domainMonitor::161::Storage.DomainMonitorThread::(_monitorLoop)
Stopping domain monitor for cc790a86-72e2-4aa2-a0b6-700756c399c3
Thread-24::DEBUG::2013-10-08
01:20:55,930::domainMonitor::161::Storage.DomainMonitorThread::(_monitorLoop)
Stopping domain monitor for d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291
Thread-24::INFO::2013-10-08
01:20:55,930::clusterlock::197::SANLock::(releaseHostId) Releasing
host id for domain d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291 (id: 2)
Thread-24::DEBUG::2013-10-08
01:20:55,930::domainMonitor::170::Storage.DomainMonitorThread::(_monitorLoop)
Unable to release the host id 2 for domain
d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 167, in _monitorLoop
self.domain.releaseHostId(self.hostId, unused=True)
File "/usr/share/vdsm/storage/sd.py", line 461, in releaseHostId
self._clusterLock.releaseHostId(hostId, async, unused)
File "/usr/share/vdsm/storage/clusterlock.py", line 204, in releaseHostId
raise se.ReleaseHostIdFailure(self._sdUUID, e)
ReleaseHostIdFailure: Cannot release host id:
('d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', SanlockException(16, 'Sanlock
lockspace remove failure', 'Device or resource busy'))
MainThread::DEBUG::2013-10-08
01:20:55,932::taskManager::80::TaskManager::(prepareForShutdown)
Request to stop all tasks
MainThread::INFO::2013-10-08
01:20:55,932::logUtils::47::dispatcher::(wrapper) Run and protect:
prepareForShutdown, Return response: None
MainThread::DEBUG::2013-10-08
01:20:55,932::task::1168::TaskManager.Task::(prepare)
Task=`23df8b46-c916-4c52-98a9-3e6540171467`::finished: None
MainThread::DEBUG::2013-10-08
01:20:55,933::task::579::TaskManager.Task::(_updateState)
Task=`23df8b46-c916-4c52-98a9-3e6540171467`::moving from state
preparing -> state finished
On target host vdsm.log:
Thread-3393::DEBUG::2013-10-08
01:20:53,900::resourceManager::634::ResourceManager::(releaseResource)
Released resource 'Storage.d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291' (0
active users)
Thread-3393::DEBUG::2013-10-08
01:20:53,900::resourceManager::640::ResourceManager::(releaseResource)
Resource 'Storage.d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291' is free,
finding out if anyone is waiting for it.
Thread-3393::DEBUG::2013-10-08
01:20:53,900::resourceManager::648::ResourceManager::(releaseResource)
No one is waiting for resource
'Storage.d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', Clearing records.
Thread-3393::DEBUG::2013-10-08
01:20:53,901::task::974::TaskManager.Task::(_decref)
Task=`be76585d-3de3-4520-873e-9eb501e12311`::ref 0 aborting False
Thread-3393::INFO::2013-10-08
01:20:53,901::clientIF::325::vds::(prepareVolumePath) prepared volume
path:
/rhev/data-center/eb679feb-4da2-4fd0-a185-abbe459ffa70/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49
Thread-3393::INFO::2013-10-08
01:20:53,901::clientIF::325::vds::(prepareVolumePath) prepared volume
path:
gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49
Thread-3393::DEBUG::2013-10-08
01:20:53,907::vm::2036::vm.Vm::(_startUnderlyingVm)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::_ongoingCreations
released
Thread-3393::DEBUG::2013-10-08
01:20:53,908::vm::3347::vm.Vm::(_waitForIncomingMigrationFinish)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Waiting 300 seconds for
end of migration
Thread-3392::ERROR::2013-10-08
01:20:53,907::vm::4789::vm.Vm::(_updateDevicesDomxmlCache)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Alias not found for
device type disk during migration at destination host
Thread-3392::DEBUG::2013-10-08
01:20:53,918::API::510::vds::(migrationCreate) Destination VM creation
succeeded
Thread-3392::DEBUG::2013-10-08
01:20:53,918::BindingXMLRPC::986::vds::(wrapper) return
vmMigrationCreate with {'status': {'message': 'Done',
'code': 0},
'migrationPort': 0, 'params': {'guestFQDN': '',
'acpiEnable': 'true',
'emulatedMachine': 'pc-1.0', 'afterMigrationStatus':
'Up',
'spiceSecureChannels':
'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard',
'pid': '0', 'memGuaranteedSize': 1024,
'transparentHugePages': 'true',
'displaySecurePort': '-1', 'timeOffset': '-114',
'cpuType':
'Opteron_G2', 'smp': '1', 'pauseCode': 'NOERR',
'migrationDest':
'libvirt', 'custom':
{'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104bedevice_1459cc81-2f88-45dd-ae3c-0c030a084d2e':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=1459cc81-2f88-45dd-ae3c-0c030a084d2e, device=unix,
type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0,
controller=0, type=virtio-serial}, managed=false, plugged=true,
readOnly=false, deviceAlias=channel0, customProperties={}}',
'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104bedevice_1459cc81-2f88-45dd-ae3c-0c030a084d2edevice_5bea8f60-7de6-4c93-99ff-af67635e2af6device_8d086706-28b6-40fa-a86d-3733c9ebe5c1':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=8d086706-28b6-40fa-a86d-3733c9ebe5c1, device=spicevmc,
type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0,
controller=0, type=virtio-serial}, managed=false, plugged=true,
readOnly=false, deviceAlias=channel2, customProperties={}}',
'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19e':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=db19ff20-8f6d-488c-bad8-19e33b01f19e, device=ide,
type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00,
domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false,
plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}}',
'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104be':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=9c0a4779-62bc-414b-abb5-a2fab36104be, device=virtio-serial,
type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00,
domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false,
plugged=true, readOnly=false, deviceAlias=virtio-serial0,
customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=a5fa5a6d-f191-42d2-be55-d6227499c658, device=scsi,
type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00,
domain=0x0000, type=pci, slot=0x04, function=0x0}, managed=false,
plugged=true, readOnly=false, deviceAlias=scsi0,
customProperties={}}',
'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104bedevice_1459cc81-2f88-45dd-ae3c-0c030a084d2edevice_5bea8f60-7de6-4c93-99ff-af67635e2af6':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=5bea8f60-7de6-4c93-99ff-af67635e2af6, device=unix,
type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0,
controller=0, type=virtio-serial}, managed=false, plugged=true,
readOnly=false, deviceAlias=channel1, customProperties={}}',
'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=bd099ee2-faee-41c3-9e01-0a01f458abe9, device=disk, type=DISK,
bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000,
type=pci, slot=0x06, function=0x0}, managed=false, plugged=true,
readOnly=false, deviceAlias=virtio-disk0, customProperties={}}'},
'vmType': 'kvm', 'spiceSslCipherSuite': 'DEFAULT',
'memSize': 1024,
'vmName': 'c8again32', 'nice': '0', 'username':
'Unknown', 'vmId':
'd54660a2-45ed-41ae-ab99-a6f93ebbdbb1', 'displayIp': '0',
'keyboardLayout': 'en-us', 'displayPort': '-1',
'smartcardEnable':
'false', 'guestIPs': '', 'nicModel':
'rtl8139,pv',
'smpCoresPerSocket': '1', 'kvmEnable': 'true',
'pitReinjection':
'false', 'devices': [{'device': 'unix', 'alias':
'channel0', 'type':
'channel', 'address': {'bus': '0', 'controller':
'0', 'type':
'virtio-serial', 'port': '1'}}, {'device': 'unix',
'alias':
'channel1', 'type': 'channel', 'address': {'bus':
'0', 'controller':
'0', 'type': 'virtio-serial', 'port': '2'}},
{'device': 'spicevmc',
'alias': 'channel2', 'type': 'channel', 'address':
{'bus': '0',
'controller': '0', 'type': 'virtio-serial',
'port': '3'}}, {'device':
'scsi', 'alias': 'scsi0', 'model': 'virtio-scsi',
'type':
'controller', 'address': {'slot': '0x04', 'bus':
'0x00', 'domain':
'0x0000', 'type': 'pci', 'function': '0x0'}},
{'device': 'usb',
'alias': 'usb0', 'type': 'controller', 'address':
{'slot': '0x01',
'bus': '0x00', 'domain': '0x0000', 'type':
'pci', 'function': '0x2'}},
{'device': 'ide', 'alias': 'ide0', 'type':
'controller', 'address':
{'slot': '0x01', 'bus': '0x00', 'domain':
'0x0000', 'type': 'pci',
'function': '0x1'}}, {'device': 'virtio-serial',
'alias':
'virtio-serial0', 'type': 'controller', 'address':
{'slot': '0x05',
'bus': '0x00', 'domain': '0x0000', 'type':
'pci', 'function': '0x0'}},
{'specParams': {'vram': '65536'}, 'alias':
'video0', 'deviceId':
'c4763c67-c69b-4b7c-b0cd-ac5beca82669', 'address': {'slot':
'0x02',
'bus': '0x00', 'domain': '0x0000', 'type':
'pci', 'function': '0x0'},
'device': 'qxl', 'type': 'video'}, {'nicModel':
'pv', 'macAddr':
'00:1a:4a:8f:04:f7', 'linkActive': True, 'network':
'ovirtmgmt',
'specParams': {}, 'custom': {}, 'filter':
'vdsm-no-mac-spoofing',
'alias': 'net0', 'deviceId':
'4478616d-d9d4-4284-8e6b-2cce0d330504',
'address': {'slot': '0x03', 'bus': '0x00',
'domain': '0x0000', 'type':
'pci', 'function': '0x0'}, 'device': 'bridge',
'type': 'interface',
'name': 'vnet0'}, {'index': '2', 'iface':
'ide', 'name': 'hdc',
'alias': 'ide0-1-0', 'shared': 'false',
'specParams': {'path': ''},
'readonly': 'True', 'deviceId':
'a3d3a41d-7f47-4a64-b3fe-10f33312f929', 'address': {'bus':
'1',
'controller': '0', 'type': 'drive', 'target':
'0', 'unit': '0'},
'device': 'cdrom', 'path': '', 'type':
'disk'}, {'index': 0, 'iface':
'virtio', 'volumeInfo': {'volPort': '0',
'volType': 'network',
'volfileServer': 'f18ovn01.mydomain', 'volTransport':
'tcp',
'protocol': 'gluster', 'path':
'gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49'},
'format': 'raw', 'shared': 'false', 'optional':
'false', 'volumeID':
'a123801a-0a4d-4a47-a426-99d8480d2e49', 'propagateErrors': 'off',
'imageID': '15f9ca1c-c435-4892-9eb7-0c84583b2a7d', 'specParams':
{},
'readonly': 'false', 'domainID':
'd0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', 'deviceId':
'15f9ca1c-c435-4892-9eb7-0c84583b2a7d', 'truesize': '4337233920',
'poolID': 'eb679feb-4da2-4fd0-a185-abbe459ffa70', 'device':
'disk',
'reqsize': '0', 'path':
'/rhev/data-center/eb679feb-4da2-4fd0-a185-abbe459ffa70/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49',
'apparentsize': '10737418240', 'type': 'disk',
'volumeChain':
[{'path':
'/rhev/data-center/eb679feb-4da2-4fd0-a185-abbe459ffa70/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49',
'domainID': 'd0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', 'vmVolInfo':
{'volPort': '0', 'volType': 'network',
'volfileServer':
'f18ovn01.mydomain', 'volTransport': 'tcp', 'protocol':
'gluster',
'path':
'gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49'},
'volumeID': 'a123801a-0a4d-4a47-a426-99d8480d2e49', 'imageID':
'15f9ca1c-c435-4892-9eb7-0c84583b2a7d'}]}, {'index': '0',
'iface':
'pci', 'name': 'vda', 'alias': 'virtio-disk0',
'readonly': 'False',
'address': {'slot': '0x06', 'bus': '0x00',
'domain': '0x0000', 'type':
'pci', 'function': '0x0'}, 'device': 'disk',
'path':
'gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49',
'type': 'disk'}, {'target': 1048576, 'specParams':
{'model':
'virtio'}, 'alias': 'balloon0', 'deviceId':
'216acfca-9672-4390-91b0-feb7aa201484', 'address': {'slot':
'0x07',
'bus': '0x00', 'domain': '0x0000', 'type':
'pci', 'function': '0x0'},
'device': 'memballoon', 'type': 'balloon'}],
'status': 'Migration
Destination', 'clientIp': '', 'display': 'qxl'}}
Thread-3396::DEBUG::2013-10-08
01:20:54,554::BindingXMLRPC::979::vds::(wrapper) client
[10.4.4.60]::call vmGetStats with
('d54660a2-45ed-41ae-ab99-a6f93ebbdbb1',) {}
Thread-3396::DEBUG::2013-10-08
01:20:54,555::BindingXMLRPC::986::vds::(wrapper) return vmGetStats
with {'status': {'message': 'Done', 'code': 0},
'statsList':
[{'status': 'Migration Destination', 'hash': '0',
'acpiEnable':
'true', 'pid': '0', 'displayIp': '0',
'displayPort': '-1',
'displaySecurePort': '-1', 'timeOffset': '-114',
'pauseCode': 'NOERR',
'kvmEnable': 'true', 'network': {}, 'vmId':
'd54660a2-45ed-41ae-ab99-a6f93ebbdbb1', 'displayType': 'qxl',
'disks':
{}, 'monitorResponse': '0', 'elapsedTime': '178',
'vmType': 'kvm',
'clientIp': ''}]}
Thread-3397::DEBUG::2013-10-08
01:20:54,885::BindingXMLRPC::979::vds::(wrapper) client
[10.4.4.59]::call vmDestroy with
('d54660a2-45ed-41ae-ab99-a6f93ebbdbb1',) {}
Thread-3397::INFO::2013-10-08 01:20:54,886::API::317::vds::(destroy)
vmContainerLock acquired by vm d54660a2-45ed-41ae-ab99-a6f93ebbdbb1
Thread-3397::DEBUG::2013-10-08
01:20:54,886::vm::4258::vm.Vm::(destroy)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::destroy Called
Thread-3397::INFO::2013-10-08
01:20:54,887::vm::4204::vm.Vm::(releaseVm)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Release VM resources
Thread-3397::WARNING::2013-10-08
01:20:54,887::clientIF::337::vds::(teardownVolumePath) Drive is not a
vdsm image: VOLWM_CHUNK_MB:1024 VOLWM_CHUNK_REPLICATE_MULT:2
VOLWM_FREE_PCT:50 _blockDev:False _checkIoTuneCategories:<bound method
Drive._checkIoTuneCategories of <vm.Drive object at 0x7fae4415bc90>>
_customize:<bound method Drive._customize of <vm.Drive object at
0x7fae4415bc90>> _deviceXML:<disk device="cdrom"
type="file">
<driver name="qemu" type="raw"/>
<source startupPolicy="optional"/>
<target bus="ide" dev="hdc"/>
<readonly/>
<serial/>
<alias name="ide0-1-0"/>
<address bus="1" controller="0" target="0"
type="drive" unit="0"/>
</disk> _makeName:<bound method Drive._makeName of <vm.Drive
object at 0x7fae4415bc90>> _validateIoTuneParams:<bound method
Drive._validateIoTuneParams of <vm.Drive object at 0x7fae4415bc90>>
address:{'bus': '1', 'controller': '0', 'type':
'drive', 'target':
'0', 'unit': '0'} alias:ide0-1-0 apparentsize:0 blockDev:False
cache:none conf:{'guestFQDN': '', 'acpiEnable': 'true',
'emulatedMachine': 'pc-1.0', 'afterMigrationStatus':
'Up',
'spiceSecureChannels':
'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard',
'pid': '0', 'memGuaranteedSize': 1024,
'transparentHugePages': 'true',
'displaySecurePort': '-1', 'timeOffset': '-114',
'cpuType':
'Opteron_G2', 'smp': '1', 'pauseCode': 'NOERR',
'migrationDest':
'libvirt', 'custom':
{'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104bedevice_1459cc81-2f88-45dd-ae3c-0c030a084d2e':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=1459cc81-2f88-45dd-ae3c-0c030a084d2e, device=unix,
type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0,
controller=0, type=virtio-serial}, managed=false, plugged=true,
readOnly=false, deviceAlias=channel0, customProperties={}}',
'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104bedevice_1459cc81-2f88-45dd-ae3c-0c030a084d2edevice_5bea8f60-7de6-4c93-99ff-af67635e2af6device_8d086706-28b6-40fa-a86d-3733c9ebe5c1':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=8d086706-28b6-40fa-a86d-3733c9ebe5c1, device=spicevmc,
type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0,
controller=0, type=virtio-serial}, managed=false, plugged=true,
readOnly=false, deviceAlias=channel2, customProperties={}}',
'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19e':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=db19ff20-8f6d-488c-bad8-19e33b01f19e, device=ide,
type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00,
domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false,
plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}}',
'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104be':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=9c0a4779-62bc-414b-abb5-a2fab36104be, device=virtio-serial,
type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00,
domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false,
plugged=true, readOnly=false, deviceAlias=virtio-serial0,
customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=a5fa5a6d-f191-42d2-be55-d6227499c658, device=scsi,
type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00,
domain=0x0000, type=pci, slot=0x04, function=0x0}, managed=false,
plugged=true, readOnly=false, deviceAlias=scsi0,
customProperties={}}',
'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104bedevice_1459cc81-2f88-45dd-ae3c-0c030a084d2edevice_5bea8f60-7de6-4c93-99ff-af67635e2af6':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=5bea8f60-7de6-4c93-99ff-af67635e2af6, device=unix,
type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0,
controller=0, type=virtio-serial}, managed=false, plugged=true,
readOnly=false, deviceAlias=channel1, customProperties={}}',
'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9':
'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1,
deviceId=bd099ee2-faee-41c3-9e01-0a01f458abe9, device=disk, type=DISK,
bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000,
type=pci, slot=0x06, function=0x0}, managed=false, plugged=true,
readOnly=false, deviceAlias=virtio-disk0, customProperties={}}'},
'vmType': 'kvm', 'spiceSslCipherSuite': 'DEFAULT',
'memSize': 1024,
'vmName': 'c8again32', 'nice': '0', 'username':
'Unknown', 'vmId':
'd54660a2-45ed-41ae-ab99-a6f93ebbdbb1', 'displayIp': '0',
'keyboardLayout': 'en-us', 'displayPort': '-1',
'smartcardEnable':
'false', 'guestIPs': '', 'nicModel':
'rtl8139,pv',
'smpCoresPerSocket': '1', 'kvmEnable': 'true',
'pitReinjection':
'false', 'devices': [{'device': 'unix', 'alias':
'channel0', 'type':
'channel', 'address': {'bus': '0', 'controller':
'0', 'type':
'virtio-serial', 'port': '1'}}, {'device': 'unix',
'alias':
'channel1', 'type': 'channel', 'address': {'bus':
'0', 'controller':
'0', 'type': 'virtio-serial', 'port': '2'}},
{'device': 'spicevmc',
'alias': 'channel2', 'type': 'channel', 'address':
{'bus': '0',
'controller': '0', 'type': 'virtio-serial',
'port': '3'}}, {'device':
'scsi', 'alias': 'scsi0', 'model': 'virtio-scsi',
'type':
'controller', 'address': {'slot': '0x04', 'bus':
'0x00', 'domain':
'0x0000', 'type': 'pci', 'function': '0x0'}},
{'device': 'usb',
'alias': 'usb0', 'type': 'controller', 'address':
{'slot': '0x01',
'bus': '0x00', 'domain': '0x0000', 'type':
'pci', 'function': '0x2'}},
{'device': 'ide', 'alias': 'ide0', 'type':
'controller', 'address':
{'slot': '0x01', 'bus': '0x00', 'domain':
'0x0000', 'type': 'pci',
'function': '0x1'}}, {'device': 'virtio-serial',
'alias':
'virtio-serial0', 'type': 'controller', 'address':
{'slot': '0x05',
'bus': '0x00', 'domain': '0x0000', 'type':
'pci', 'function': '0x0'}},
{'specParams': {'vram': '65536'}, 'alias':
'video0', 'deviceId':
'c4763c67-c69b-4b7c-b0cd-ac5beca82669', 'address': {'slot':
'0x02',
'bus': '0x00', 'domain': '0x0000', 'type':
'pci', 'function': '0x0'},
'device': 'qxl', 'type': 'video'}, {'nicModel':
'pv', 'macAddr':
'00:1a:4a:8f:04:f7', 'linkActive': True, 'network':
'ovirtmgmt',
'specParams': {}, 'custom': {}, 'filter':
'vdsm-no-mac-spoofing',
'alias': 'net0', 'deviceId':
'4478616d-d9d4-4284-8e6b-2cce0d330504',
'address': {'slot': '0x03', 'bus': '0x00',
'domain': '0x0000', 'type':
'pci', 'function': '0x0'}, 'device': 'bridge',
'type': 'interface',
'name': 'vnet0'}, {'index': '2', 'iface':
'ide', 'name': 'hdc',
'alias': 'ide0-1-0', 'shared': 'false',
'specParams': {'path': ''},
'readonly': 'True', 'deviceId':
'a3d3a41d-7f47-4a64-b3fe-10f33312f929', 'address': {'bus':
'1',
'controller': '0', 'type': 'drive', 'target':
'0', 'unit': '0'},
'device': 'cdrom', 'path': '', 'type':
'disk'}, {'index': 0, 'iface':
'virtio', 'volumeInfo': {'volPort': '0',
'volType': 'network',
'volfileServer': 'f18ovn01.mydomain', 'volTransport':
'tcp',
'protocol': 'gluster', 'path':
'gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49'},
'format': 'raw', 'shared': 'false', 'optional':
'false', 'volumeID':
'a123801a-0a4d-4a47-a426-99d8480d2e49', 'propagateErrors': 'off',
'imageID': '15f9ca1c-c435-4892-9eb7-0c84583b2a7d', 'specParams':
{},
'readonly': 'false', 'domainID':
'd0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', 'deviceId':
'15f9ca1c-c435-4892-9eb7-0c84583b2a7d', 'truesize': '4337233920',
'poolID': 'eb679feb-4da2-4fd0-a185-abbe459ffa70', 'device':
'disk',
'reqsize': '0', 'path':
'/rhev/data-center/eb679feb-4da2-4fd0-a185-abbe459ffa70/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49',
'apparentsize': '10737418240', 'type': 'disk',
'volumeChain':
[{'path':
'/rhev/data-center/eb679feb-4da2-4fd0-a185-abbe459ffa70/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49',
'domainID': 'd0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', 'vmVolInfo':
{'volPort': '0', 'volType': 'network',
'volfileServer':
'f18ovn01.mydomain', 'volTransport': 'tcp', 'protocol':
'gluster',
'path':
'gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49'},
'volumeID': 'a123801a-0a4d-4a47-a426-99d8480d2e49', 'imageID':
'15f9ca1c-c435-4892-9eb7-0c84583b2a7d'}]}, {'index': '0',
'iface':
'pci', 'name': 'vda', 'alias': 'virtio-disk0',
'readonly': 'False',
'address': {'slot': '0x06', 'bus': '0x00',
'domain': '0x0000', 'type':
'pci', 'function': '0x0'}, 'device': 'disk',
'path':
'gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49',
'type': 'disk'}, {'target': 1048576, 'specParams':
{'model':
'virtio'}, 'alias': 'balloon0', 'deviceId':
'216acfca-9672-4390-91b0-feb7aa201484', 'address': {'slot':
'0x07',
'bus': '0x00', 'domain': '0x0000', 'type':
'pci', 'function': '0x0'},
'device': 'memballoon', 'type': 'balloon'}],
'status': 'Migration
Destination', 'clientIp': '', 'display': 'qxl'}
createXmlElem:<bound
method Drive.createXmlElem of <vm.Drive object at 0x7fae4415bc90>>
device:cdrom deviceId:a3d3a41d-7f47-4a64-b3fe-10f33312f929
getNextVolumeSize:<bound method Drive.getNextVolumeSize of <vm.Drive
object at 0x7fae4415bc90>> getXML:<bound method Drive.getXML of
<vm.Drive object at 0x7fae4415bc90>> iface:ide index:2
isDiskReplicationInProgress:<bound method
Drive.isDiskReplicationInProgress of <vm.Drive object at
0x7fae4415bc90>> isVdsmImage:<bound method Drive.isVdsmImage of
<vm.Drive object at 0x7fae4415bc90>> log:<logUtils.SimpleLogAdapter
object at 0x7fae2058f1d0> name:hdc networkDev:False path:
readonly:True reqsize:0 serial: shared:false specParams:{'path': ''}
truesize:0 type:disk volExtensionChunk:1024 watermarkLimit:536870912
Traceback (most recent call last):
File "/usr/share/vdsm/clientIF.py", line 331, in teardownVolumePath
res = self.irs.teardownImage(drive['domainID'],
File "/usr/share/vdsm/vm.py", line 1344, in __getitem__
raise KeyError(key)
KeyError: 'domainID'
Thread-3393::ERROR::2013-10-08
01:20:54,888::libvirtconnection::94::libvirtconnection::(wrapper)
connection to libvirt broken. ecode: 1 edom: 7
Thread-3393::ERROR::2013-10-08
01:20:54,892::libvirtconnection::96::libvirtconnection::(wrapper)
taking calling process down.
firewall on destination seems ok:
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 192.168.3.3 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:54321
...
3:40 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On Tue, Oct 08, 2013 at 01:44:47AM +0200, Gianluca Cecchi wrote:
On Mon, Oct 7, 2013 at 2:59 AM, Dan Kenigsberg wrote:
>
> Would you please test if http://gerrit.ovirt.org/19906 solves the
> issue? (I haven't. Too late at night.)
>
> Regards,
> Dan.
I can confirm that it resolves
https://bugzilla.redhat.com/show_bug.cgi?id=1007980
so now I'm able to start VM without having to select run once and
attaching a cd iso
(note that is only valid for newly created VMs though)
Yes. Old VMs are trashed with the bogus address reproted by the buggy
Vdsm. Can someone from engine supply a script to clear all device
addresses from the VM database table?
But migration still fails
It seems like an unrelated failure. I do not know what's blocking
migration traffic. Could you see if libvirtd.log and qemu logs at source
and destinaiton have clues?
Thread-4644::DEBUG::2013-10-08
01:20:53,933::vm::360::vm.Vm::(_startUnderlyingMigration)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::starting migration to
qemu+tls://10.4.4.58/system with miguri tcp://10.4.4.58
Thread-4645::DEBUG::2013-10-08 01:20:53,934::vm::718::vm.Vm::(run)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration downtime thread
started
Thread-4646::DEBUG::2013-10-08 01:20:53,935::vm::756::vm.Vm::(run)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::starting migration
monitor thread
Thread-4648::DEBUG::2013-10-08
01:20:54,321::BindingXMLRPC::979::vds::(wrapper) client
[10.4.4.60]::call volumesList with () {}
Thread-4648::DEBUG::2013-10-08
01:20:54,349::BindingXMLRPC::986::vds::(wrapper) return volumesList
with {'status': {'message': 'Done', 'code': 0},
'volumes': {'gviso':
{'transportType': ['TCP'], 'uuid':
'c8cbcac7-1d40-4cee-837d-bb97467fb2bd', 'bricks':
['f18ovn01.mydomain:/gluster/ISO_GLUSTER/brick1',
'f18ovn03.mydomain:/gluster/ISO_GLUSTER/brick1'], 'volumeName':
'gviso', 'volumeType': 'REPLICATE', 'replicaCount':
'2', 'brickCount':
'2', 'distCount': '2', 'volumeStatus': 'ONLINE',
'stripeCount': '1',
'options': {'storage.owner-gid': '36',
'storage.owner-uid': '36',
'server.allow-insecure': 'on'}}, 'gvdata':
{'transportType': ['TCP'],
'uuid': 'ed71a4c2-6205-4aad-9aab-85da086d5ba3', 'bricks':
['f18ovn01.mydomain:/gluster/DATA_GLUSTER/brick1',
'f18ovn03.mydomain:/gluster/DATA_GLUSTER/brick1'], 'volumeName':
'gvdata', 'volumeType': 'REPLICATE', 'replicaCount':
'2',
'brickCount': '2', 'distCount': '2',
'volumeStatus': 'ONLINE',
'stripeCount': '1', 'options': {'server.allow-insecure':
'on',
'storage.owner-uid': '36', 'storage.owner-gid': '36'}}}}
Thread-4644::ERROR::2013-10-08
01:20:54,873::libvirtconnection::94::libvirtconnection::(wrapper)
connection to libvirt broken. ecode: 38 edom: 7
Thread-4644::ERROR::2013-10-08
01:20:54,873::libvirtconnection::96::libvirtconnection::(wrapper)
taking calling process down.
MainThread::DEBUG::2013-10-08
01:20:54,874::vdsm::45::vds::(sigtermHandler) Received signal 15
Thread-4644::DEBUG::2013-10-08 01:20:54,874::vm::733::vm.Vm::(cancel)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::canceling migration
downtime thread
Thread-4644::DEBUG::2013-10-08 01:20:54,875::vm::803::vm.Vm::(stop)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::stopping migration
monitor thread
Thread-4645::DEBUG::2013-10-08 01:20:54,875::vm::730::vm.Vm::(run)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration downtime thread
exiting
Thread-4644::ERROR::2013-10-08
01:20:54,875::vm::244::vm.Vm::(_recover)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Cannot recv data:
Input/output error
Thread-4644::ERROR::2013-10-08 01:20:55,008::vm::324::vm.Vm::(run)
vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Failed to migrate
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 311, in run
self._startUnderlyingMigration()
File "/usr/share/vdsm/vm.py", line 388, in _startUnderlyingMigration
None, maxBandwidth)
File "/usr/share/vdsm/vm.py", line 826, in f
ret = attr(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.py",
line 76, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1253, in
migrateToURI2
if ret == -1: raise libvirtError ('virDomainMigrateToURI2()
failed', dom=self)
libvirtError: Cannot recv data: Input/output error
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 192.168.3.3 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:54321
...
5:27 a.m.
New subject: [Users] Migration issues with ovirt 3.3
----- Original Message -----
From: "Dan Kenigsberg" <danken(a)redhat.com>
To: "Gianluca Cecchi" <gianluca.cecchi(a)gmail.com>
Cc: "users" <users(a)ovirt.org>, "Michal Skrivanek"
<mskrivan(a)redhat.com>
Sent: Tuesday, October 8, 2013 11:40:25 AM
Subject: Re: [Users] Migration issues with ovirt 3.3
On Tue, Oct 08, 2013 at 01:44:47AM +0200, Gianluca Cecchi wrote:
> On Mon, Oct 7, 2013 at 2:59 AM, Dan Kenigsberg wrote:
>
> >
> > Would you please test if http://gerrit.ovirt.org/19906 solves the
> > issue? (I haven't. Too late at night.)
> >
> > Regards,
> > Dan.
>
> I can confirm that it resolves
> https://bugzilla.redhat.com/show_bug.cgi?id=1007980
>
> so now I'm able to start VM without having to select run once and
> attaching a cd iso
> (note that is only valid for newly created VMs though)
Yes. Old VMs are trashed with the bogus address reproted by the buggy
Vdsm. Can someone from engine supply a script to clear all device
addresses from the VM database table?
you can use this line (assuming engine as db and user), make sure only 'bad' vms
return:
psql -U engine -d engine -c "select distinct vm_name from vm_static, vm_device where
vm_guid=vm_id and device='cdrom' and address ilike '%pci%';"
if so, you can run this to clear the address field for them, so they could run again:
psql -U engine -d engine -c "update vm_device set address='' where
device='cdrom' and address ilike '%pci%';"
>
> But migration still fails
>
It seems like an unrelated failure. I do not know what's blocking
migration traffic. Could you see if libvirtd.log and qemu logs at source
and destinaiton have clues?
> Thread-4644::DEBUG::2013-10-08
> 01:20:53,933::vm::360::vm.Vm::(_startUnderlyingMigration)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::starting migration to
> qemu+tls://10.4.4.58/system with miguri tcp://10.4.4.58
> Thread-4645::DEBUG::2013-10-08 01:20:53,934::vm::718::vm.Vm::(run)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration downtime thread
> started
> Thread-4646::DEBUG::2013-10-08 01:20:53,935::vm::756::vm.Vm::(run)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::starting migration
> monitor thread
> Thread-4648::DEBUG::2013-10-08
> 01:20:54,321::BindingXMLRPC::979::vds::(wrapper) client
> [10.4.4.60]::call volumesList with () {}
> Thread-4648::DEBUG::2013-10-08
> 01:20:54,349::BindingXMLRPC::986::vds::(wrapper) return volumesList
> with {'status': {'message': 'Done', 'code': 0},
'volumes': {'gviso':
> {'transportType': ['TCP'], 'uuid':
> 'c8cbcac7-1d40-4cee-837d-bb97467fb2bd', 'bricks':
> ['f18ovn01.mydomain:/gluster/ISO_GLUSTER/brick1',
> 'f18ovn03.mydomain:/gluster/ISO_GLUSTER/brick1'], 'volumeName':
> 'gviso', 'volumeType': 'REPLICATE', 'replicaCount':
'2', 'brickCount':
> '2', 'distCount': '2', 'volumeStatus':
'ONLINE', 'stripeCount': '1',
> 'options': {'storage.owner-gid': '36',
'storage.owner-uid': '36',
> 'server.allow-insecure': 'on'}}, 'gvdata':
{'transportType': ['TCP'],
> 'uuid': 'ed71a4c2-6205-4aad-9aab-85da086d5ba3', 'bricks':
> ['f18ovn01.mydomain:/gluster/DATA_GLUSTER/brick1',
> 'f18ovn03.mydomain:/gluster/DATA_GLUSTER/brick1'], 'volumeName':
> 'gvdata', 'volumeType': 'REPLICATE', 'replicaCount':
'2',
> 'brickCount': '2', 'distCount': '2',
'volumeStatus': 'ONLINE',
> 'stripeCount': '1', 'options':
{'server.allow-insecure': 'on',
> 'storage.owner-uid': '36', 'storage.owner-gid':
'36'}}}}
> Thread-4644::ERROR::2013-10-08
> 01:20:54,873::libvirtconnection::94::libvirtconnection::(wrapper)
> connection to libvirt broken. ecode: 38 edom: 7
> Thread-4644::ERROR::2013-10-08
> 01:20:54,873::libvirtconnection::96::libvirtconnection::(wrapper)
> taking calling process down.
> MainThread::DEBUG::2013-10-08
> 01:20:54,874::vdsm::45::vds::(sigtermHandler) Received signal 15
> Thread-4644::DEBUG::2013-10-08 01:20:54,874::vm::733::vm.Vm::(cancel)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::canceling migration
> downtime thread
> Thread-4644::DEBUG::2013-10-08 01:20:54,875::vm::803::vm.Vm::(stop)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::stopping migration
> monitor thread
> Thread-4645::DEBUG::2013-10-08 01:20:54,875::vm::730::vm.Vm::(run)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration downtime thread
> exiting
> Thread-4644::ERROR::2013-10-08
> 01:20:54,875::vm::244::vm.Vm::(_recover)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Cannot recv data:
> Input/output error
> Thread-4644::ERROR::2013-10-08 01:20:55,008::vm::324::vm.Vm::(run)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Failed to migrate
> Traceback (most recent call last):
> File "/usr/share/vdsm/vm.py", line 311, in run
> self._startUnderlyingMigration()
> File "/usr/share/vdsm/vm.py", line 388, in _startUnderlyingMigration
> None, maxBandwidth)
> File "/usr/share/vdsm/vm.py", line 826, in f
> ret = attr(*args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.py",
> line 76, in wrapper
> ret = f(*args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1253, in
> migrateToURI2
> if ret == -1: raise libvirtError ('virDomainMigrateToURI2()
> failed', dom=self)
> libvirtError: Cannot recv data: Input/output error
>
> Chain INPUT (policy ACCEPT)
> target prot opt source destination
> ACCEPT all -- 192.168.3.3 0.0.0.0/0
> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
> RELATED,ESTABLISHED
> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:54321
> ...
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
7:27 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On Tue, Oct 8, 2013 at 12:27 PM, Omer Frenkel wrote:
> >
> > so now I'm able to start VM without having to select run once and
> > attaching a cd iso
> > (note that is only valid for newly created VMs though)
>
> Yes. Old VMs are trashed with the bogus address reproted by the buggy
> Vdsm. Can someone from engine supply a script to clear all device
> addresses from the VM database table?
>
you can use this line (assuming engine as db and user), make sure only 'bad' vms
return:
psql -U engine -d engine -c "select distinct vm_name from vm_static, vm_device where
vm_guid=vm_id and device='cdrom' and address ilike '%pci%';"
if so, you can run this to clear the address field for them, so they could run again:
psql -U engine -d engine -c "update vm_device set address='' where
device='cdrom' and address ilike '%pci%';"
I wanted to test this but for some reason it seems actually it solved itself.
I first ran the query and already had no value:
engine=# select distinct vm_name from vm_static, vm_device where
vm_guid=vm_id and device='cdrom' and address ilike '%pci%';
vm_name
---------
(0 rows)
(overall my VMs are
engine=# select distinct vm_name from vm_static;
vm_name
-----------
Blank
c6s
c8again32
(3 rows)
)
Then I tried to start the powered off VM that gave problems and it started ok.
Tried to run shutdown inside it and power on again and it worked too.
As soon as I updated the bugzilla with my comment, I restarted vdsmd
on both nodes but I wasn't able to run it...
strange... any other thing that could have resolved by itself.
Things done in sequence related to that VM were:
modify vm.py on both nodes and restart vdsmd while VM was powered off
verify that power on gave the error
verify that run once worked as before
shutdown and power off VM
after two days (no activity on my side at all) I wanted to try the DB
workaround to solve this VM status but apparently the column was
already empty and VM able to start normally....
Gianluca
8:35 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On Wed, Oct 09, 2013 at 02:27:16PM +0200, Gianluca Cecchi wrote:
On Tue, Oct 8, 2013 at 12:27 PM, Omer Frenkel wrote:
>> >
>> > so now I'm able to start VM without having to select run once and
>> > attaching a cd iso
>> > (note that is only valid for newly created VMs though)
>>
>> Yes. Old VMs are trashed with the bogus address reproted by the buggy
>> Vdsm. Can someone from engine supply a script to clear all device
>> addresses from the VM database table?
>>
>
>
> you can use this line (assuming engine as db and user), make sure only 'bad'
vms return:
> psql -U engine -d engine -c "select distinct vm_name from vm_static, vm_device
where vm_guid=vm_id and device='cdrom' and address ilike '%pci%';"
>
> if so, you can run this to clear the address field for them, so they could run
again:
> psql -U engine -d engine -c "update vm_device set address='' where
device='cdrom' and address ilike '%pci%';"
>
I wanted to test this but for some reason it seems actually it solved itself.
I first ran the query and already had no value:
engine=# select distinct vm_name from vm_static, vm_device where
vm_guid=vm_id and device='cdrom' and address ilike '%pci%';
vm_name
---------
(0 rows)
(overall my VMs are
engine=# select distinct vm_name from vm_static;
vm_name
-----------
Blank
c6s
c8again32
(3 rows)
Which of these 3 is the one that was started up with an empty cdrom on a
vanilla ovirt-3.3.0 vdsm? The script is expect to show only those.
8:39 a.m.
New subject: [Users] Migration issues with ovirt 3.3
Il 09/ott/2013 15:35 "Dan Kenigsberg" <danken(a)redhat.com> ha scritto:
On Wed, Oct 09, 2013 at 02:27:16PM +0200, Gianluca Cecchi wrote:
> On Tue, Oct 8, 2013 at 12:27 PM, Omer Frenkel wrote:
>
> >> >
> >> > so now I'm able to start VM without having to select run once and
> >> > attaching a cd iso
> >> > (note that is only valid for newly created VMs though)
> >>
> >> Yes. Old VMs are trashed with the bogus address reproted by the buggy
> >> Vdsm. Can someone from engine supply a script to clear all device
> >> addresses from the VM database table?
> >>
> >
> >
> > you can use this line (assuming engine as db and user), make sure
only
'bad' vms return:
> > psql -U engine -d engine -c "select distinct vm_name
from vm_static,
vm_device where vm_guid=vm_id and device='cdrom' and address
ilike '%pci%';"
> >
> > if so, you can run this to clear the address field for them, so they
could run again:
> > psql -U engine -d engine -c "update vm_device set
address='' where
device='cdrom' and address ilike
'%pci%';"
> >
>
> I wanted to test this but for some reason it seems actually it solved
itself.
> I first ran the query and already had no value:
> engine=# select distinct vm_name from vm_static, vm_device where
> vm_guid=vm_id and device='cdrom' and address ilike '%pci%';
> vm_name
> ---------
> (0 rows)
>
> (overall my VMs are
> engine=# select distinct vm_name from vm_static;
> vm_name
> -----------
> Blank
> c6s
> c8again32
> (3 rows)
Which of these 3 is the one that was started up with an empty cdrom on a
vanilla ovirt-3.3.0 vdsm? The script is expect to show only those.
It is c6s
8:41 a.m.
New subject: [Users] Migration issues with ovirt 3.3
Il 09/ott/2013 15:39 "Gianluca Cecchi" <gianluca.cecchi(a)gmail.com> ha
scritto:
Il 09/ott/2013 15:35 "Dan Kenigsberg" <danken(a)redhat.com> ha scritto:
>
> On Wed, Oct 09, 2013 at 02:27:16PM +0200, Gianluca Cecchi wrote:
> > On Tue, Oct 8, 2013 at 12:27 PM, Omer Frenkel wrote:
> >
> > >> >
> > >> > so now I'm able to start VM without having to select run once
and
> > >> > attaching a cd iso
> > >> > (note that is only valid for newly created VMs though)
> > >>
> > >> Yes. Old VMs are trashed with the bogus address reproted by the
buggy
> > >> Vdsm. Can someone from engine supply a script to
clear all device
> > >> addresses from the VM database table?
> > >>
> > >
> > >
> > > you can use this line (assuming engine as db and user), make sure
only 'bad' vms return:
> > > psql -U engine -d engine -c "select distinct
vm_name from
vm_static, vm_device where vm_guid=vm_id and device='cdrom' and
address
ilike '%pci%';"
> > >
> > > if so, you can run this to clear the address field for them, so
they
could run again:
> > > psql -U engine -d engine -c "update vm_device set
address='' where
device='cdrom' and address ilike
'%pci%';"
> > >
> >
> > I wanted to test this but for some reason it seems actually it solved
itself.
> > I first ran the query and already had no value:
> > engine=# select distinct vm_name from vm_static, vm_device where
> > vm_guid=vm_id and device='cdrom' and address ilike '%pci%';
> > vm_name
> > ---------
> > (0 rows)
> >
> > (overall my VMs are
> > engine=# select distinct vm_name from vm_static;
> > vm_name
> > -----------
> > Blank
> > c6s
> > c8again32
> > (3 rows)
>
> Which of these 3 is the one that was started up with an empty cdrom on a
> vanilla ovirt-3.3.0 vdsm? The script is expect to show only those.
It is c6s
Note that the query suggested by Omer had 0 rows.
My further query with 3 results was to show you all rows from vm_static at
the moment
9:19 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On Wed, Oct 09, 2013 at 03:41:58PM +0200, Gianluca Cecchi wrote:
Il 09/ott/2013 15:39 "Gianluca Cecchi"
<gianluca.cecchi(a)gmail.com> ha
scritto:
>
>
> Il 09/ott/2013 15:35 "Dan Kenigsberg" <danken(a)redhat.com> ha
scritto:
>
> >
> > On Wed, Oct 09, 2013 at 02:27:16PM +0200, Gianluca Cecchi wrote:
> > > On Tue, Oct 8, 2013 at 12:27 PM, Omer Frenkel wrote:
> > >
> > > >> >
> > > >> > so now I'm able to start VM without having to select run
once and
> > > >> > attaching a cd iso
> > > >> > (note that is only valid for newly created VMs though)
> > > >>
> > > >> Yes. Old VMs are trashed with the bogus address reproted by the
buggy
> > > >> Vdsm. Can someone from engine supply a script to clear all
device
> > > >> addresses from the VM database table?
> > > >>
> > > >
> > > >
> > > > you can use this line (assuming engine as db and user), make sure
only 'bad' vms return:
> > > > psql -U engine -d engine -c "select distinct vm_name from
vm_static, vm_device where vm_guid=vm_id and device='cdrom' and address
ilike '%pci%';"
> > > >
> > > > if so, you can run this to clear the address field for them, so
they could run again:
> > > > psql -U engine -d engine -c "update vm_device set
address='' where
device='cdrom' and address ilike '%pci%';"
> > > >
> > >
> > > I wanted to test this but for some reason it seems actually it solved
itself.
> > > I first ran the query and already had no value:
> > > engine=# select distinct vm_name from vm_static, vm_device where
> > > vm_guid=vm_id and device='cdrom' and address ilike
'%pci%';
> > > vm_name
> > > ---------
> > > (0 rows)
> > >
> > > (overall my VMs are
> > > engine=# select distinct vm_name from vm_static;
> > > vm_name
> > > -----------
> > > Blank
> > > c6s
> > > c8again32
> > > (3 rows)
> >
> > Which of these 3 is the one that was started up with an empty cdrom on a
> > vanilla ovirt-3.3.0 vdsm? The script is expect to show only those.
>
> It is c6s
Note that the query suggested by Omer had 0 rows.
My further query with 3 results was to show you all rows from vm_static at
the moment
Yeah, understood. I have no idea how your c6s cleaned itslef up.
7:42 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote:
>
> But migration still fails
>
It seems like an unrelated failure. I do not know what's blocking
migration traffic. Could you see if libvirtd.log and qemu logs at source
and destinaiton have clues?
It seems that on VM.log under qemu of desdt host I have:
...
-incoming tcp:[::]:49153: Failed to bind socket: Address already in use
See all:
- In libvirtd.log of source host
2013-10-07 23:20:54.471+0000: 1209: debug :
qemuMonitorOpenInternal:751 : QEMU_MONITOR_NEW: mon=0x7fc66412e820
refs=2 fd=30
2013-10-07 23:20:54.472+0000: 1209: warning :
qemuDomainObjEnterMonitorInternal:1136 : This thread seems to be the
async job owner; entering monitor without asking for a nested job is
dangerous
2013-10-07 23:20:54.472+0000: 1209: debug :
qemuMonitorSetCapabilities:1145 : mon=0x7fc66412e820
2013-10-07 23:20:54.472+0000: 1209: debug : qemuMonitorSend:887 :
QEMU_MONITOR_SEND_MSG: mon=0x7fc66412e820
msg={"execute":"qmp_capabilities","id":"libvirt-1"}
fd=-1
2013-10-07 23:20:54.769+0000: 1199: error : qemuMonitorIORead:505 :
Unable to read from monitor: Connection reset by peer
2013-10-07 23:20:54.769+0000: 1199: debug : qemuMonitorIO:638 : Error
on monitor Unable to read from monitor: Connection reset by peer
2013-10-07 23:20:54.769+0000: 1199: debug : qemuMonitorIO:672 :
Triggering error callback
2013-10-07 23:20:54.769+0000: 1199: debug :
qemuProcessHandleMonitorError:351 : Received error on 0x7fc664124fb0
'c8again32'
2013-10-07 23:20:54.769+0000: 1209: debug : qemuMonitorSend:899 : Send
command resulted in error Unable to read from monitor: Connection
reset by peer
2013-10-07 23:20:54.770+0000: 1199: debug : qemuMonitorIO:638 : Error
on monitor Unable to read from monitor: Connection reset by peer
2013-10-07 23:20:54.770+0000: 1209: debug : virFileMakePathHelper:1283
: path=/var/run/libvirt/qemu mode=0777
2013-10-07 23:20:54.770+0000: 1199: debug : qemuMonitorIO:661 :
Triggering EOF callback
2013-10-07 23:20:54.770+0000: 1199: debug :
qemuProcessHandleMonitorEOF:294 : Received EOF on 0x7fc664124fb0
'c8again32'
2013-10-07 23:20:54.770+0000: 1209: debug : qemuProcessStop:3992 :
Shutting down VM 'c8again32' pid=18053 flags=0
2013-10-07 23:20:54.771+0000: 1209: error :
virNWFilterDHCPSnoopEnd:2135 : internal error ifname "vnet0" not in
key map
2013-10-07 23:20:54.782+0000: 1209: debug : virCommandRunAsync:2251 :
About to run /bin/sh -c 'IPT="/usr/sbin/iptables"
$IPT -D libvirt-out -m physdev --physdev-is-bridged --physdev-out
vnet0 -g FO-vnet0
$IPT -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
$IPT -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0
$IPT -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0
$IPT -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT
$IPT -F FO-vnet0
$IPT -X FO-vnet0
$IPT -F FI-vnet0
$IPT -X FI-vnet0
$IPT -F HI-vnet0
$IPT -X HI-vnet0
IPT="/usr/sbin/ip6tables"
$IPT -D libvirt-out -m physdev --physdev-is-bridged --physdev-out
vnet0 -g FO-vnet0
$IPT -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
$IPT -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0
$IPT -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0
$IPT -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT
$IPT -F FO-vnet0
$IPT -X FO-vnet0
$IPT -F FI-vnet0
$IPT -X FI-vnet0
$IPT -F HI-vnet0
$IPT -X HI-vnet0
EBT="/usr/sbin/ebtables"
$EBT -t nat -D PREROUTING -i vnet0 -j libvirt-I-vnet0
$EBT -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0
EBT="/usr/sbin/ebtables"
collect_chains()
{
for tmp2 in $*; do
for tmp in $($EBT -t nat -L $tmp2 | \
sed -n "/Bridge chain/,\$ s/.*-j \\([IO]-.*\\)/\\1/p");
do
echo $tmp
collect_chains $tmp
done
done
}
rm_chains()
{
for tmp in $*; do $EBT -t nat -F $tmp; done
for tmp in $*; do $EBT -t nat -X $tmp; done
}
tmp='\''
'\''
IFS='\'' '\'''\'' '\''$tmp
chains="$(collect_chains libvirt-I-vnet0 libvirt-O-vnet0)"
$EBT -t nat -F libvirt-I-vnet0
$EBT -t nat -F libvirt-O-vnet0
rm_chains $chains
$EBT -t nat -F libvirt-I-vnet0
$EBT -t nat -X libvirt-I-vnet0
$EBT -t nat -F libvirt-O-vnet0
$EBT -t nat -X libvirt-O-vnet0
'
2013-10-07 23:20:54.784+0000: 1209: debug : virCommandRunAsync:2256 :
Command result 0, with PID 18076
2013-10-07 23:20:54.863+0000: 1209: debug : virCommandRun:2125 :
Result exit status 255, stdout: '' stderr: 'iptables v1.4.18: goto
'FO-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.18: goto 'FO-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.18: goto 'FI-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.18: goto 'HI-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables: Bad rule (does a matching rule exist in that chain?).
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
ip6tables v1.4.18: goto 'FO-vnet0' is not a chain
Try `ip6tables -h' or 'ip6tables --help' for more information.
ip6tables v1.4.18: goto 'FO-vnet0' is not a chain
Try `ip6tables -h' or 'ip6tables --help' for more information.
ip6tables v1.4.18: goto 'FI-vnet0' is not a chain
Try `ip6tables -h' or 'ip6tables --help' for more information.
ip6tables v1.4.18: goto 'HI-vnet0' is not a chain
Try `ip6tables -h' or 'ip6tables --help' for more information.
ip6tables: Bad rule (does a matching rule exist in that chain?).
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
Illegal target name 'libvirt-O-vnet0'.
Chain 'libvirt-O-vnet0' doesn't exist.
Chain 'libvirt-O-vnet0' doesn't exist.
Chain 'libvirt-O-vnet0' doesn't exist.
Chain 'libvirt-O-vnet0' doesn't exist.
'
2013-10-07 23:20:54.863+0000: 1209: debug : qemuMonitorClose:821 :
QEMU_MONITOR_CLOSE: mon=0x7fc66412e820 refs=2
2013-10-07 23:20:54.863+0000: 1209: debug : qemuProcessKill:3951 :
vm=c8again32 pid=18053 flags=5
2013-10-07 23:20:54.863+0000: 1209: debug :
virProcessKillPainfully:269 : vpid=18053 force=1
2013-10-07 23:20:54.863+0000: 1209: debug : qemuDomainCleanupRun:2132
: driver=0x7fc664024cd0, vm=c8again32
2013-10-07 23:20:54.863+0000: 1209: debug :
qemuProcessAutoDestroyRemove:4504 : vm=c8again32
2013-10-07 23:20:54.863+0000: 1209: debug :
virQEMUCloseCallbacksUnset:744 : vm=c8again32,
uuid=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, cb=0x7fc66b6fe570
2013-10-07 23:20:54.864+0000: 1209: error :
virPortAllocatorRelease:174 : port 0 must be in range (5900, 65535)
2013-10-07 23:20:54.865+0000: 1209: debug : qemuDomainObjEndJob:1070 :
Stopping job: none (async=migration in)
2013-10-07 23:20:54.865+0000: 1209: debug :
qemuDomainObjEndAsyncJob:1088 : Stopping async job: migration in
2013-10-07 23:20:54.865+0000: 1199: debug :
qemuProcessHandleMonitorEOF:306 : Domain 0x7fc664124fb0 is not active,
ignoring EOF
Caught Segmentation violation dumping internal log buffer:
====== start of log =====
- In destitnation host libvirtd.log
2013-10-07 23:20:53.970+0000: 1206: debug :
virQEMUCloseCallbacksGet:786 : cb=(nil)
2013-10-07 23:20:53.970+0000: 1206: debug :
qemuMigrationEatCookie:1058 : cookielen=0 cookie='(null)'
2013-10-07 23:20:53.971+0000: 1206: debug :
qemuMigrationBakeCookie:1035 : cookielen=250 cookie=<qemu-migration>
<name>c8again32</name>
<uuid>d54660a2-45ed-41ae-ab99-a6f93ebbdbb1</uuid>
<hostname>f18ovn03.mydomain</hostname>
<hostuuid>4045b69a-5dae-4218-ad2b-7911a2c368d9</hostuuid>
<feature name='lockstate'/>
</qemu-migration>
2013-10-07 23:20:53.975+0000: 1206: debug :
qemuDomainDefFormatBuf:1353 : Removing default USB controller from
domain 'c8again32' for migration compatibility
2013-10-07 23:20:53.975+0000: 1206: debug :
qemuDomainDefFormatBuf:1374 : Removing default 'pci-root' from domain
'c8again32' for migration compatibility
2013-10-07 23:20:53.975+0000: 1206: debug : doPeer2PeerMigrate3:3321 :
Prepare3 0x7f41940015d0
2013-10-07 23:20:54.870+0000: 1206: error : virNetSocketReadWire:1373
: Cannot recv data: Input/output error
2013-10-07 23:20:54.870+0000: 1206: debug : virConnectClose:1494 :
conn=0x7f41940015d0
2013-10-07 23:20:54.871+0000: 1206: error :
virNetClientSendInternal:1965 : internal error client socket is closed
2013-10-07 23:20:54.871+0000: 1206: debug :
qemuDomainObjEndAsyncJob:1088 : Stopping async job: migration out
2013-10-07 23:20:54.872+0000: 1206: debug : virFileMakePathHelper:1283
: path=/var/run/libvirt/qemu mode=0777
2013-10-07 23:20:54.872+0000: 1206: debug : virDomainFree:2326 :
dom=0x7f4194190c10, (VM: name=c8again32,
uuid=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1)
2013-10-07 23:20:54.876+0000: 1204: debug : virDomainGetJobInfo:17503
: dom=0x7f418c009530, (VM: name=c8again32,
uuid=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1), info=0x7f41b0fe4b10
2013-10-07 23:20:54.876+0000: 1204: debug : virDomainFree:2326 :
dom=0x7f418c009530, (VM: name=c8again32,
uuid=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1)
2013-10-07 23:20:54.992+0000: 1202: debug : virDomainGetXMLDesc:4384 :
dom=0x7f41a03a7840, (VM: name=c8again32,
uuid=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1), flags=0
2013-10-07 23:20:54.992+0000: 1202: debug : virDomainFree:2326 :
dom=0x7f41a03a7840, (VM: name=c8again32,
uuid=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1)
2013-10-07 23:20:59.574+0000: 1192: error : virNetSocketReadWire:1381
: End of file while reading data: Input/output error
2013-10-07 23:20:59.575+0000: 1192: debug : virConnectClose:1494 :
conn=0x7f41a012eba0
2013-10-07 23:20:59.576+0000: 1192: debug :
virQEMUCloseCallbacksRun:882 : conn=0x7f41a012eba0
2013-10-07 23:20:59.591+0000: 1192: error : virNetSocketReadWire:1381
: End of file while reading data: Input/output error
2013-10-07 23:20:59.592+0000: 1192: debug :
virConnectDomainEventDeregisterAny:17972 : conn=0x7f41a0231a70,
callbackID=14
2013-10-07 23:20:59.592+0000: 1192: debug :
virConnectDomainEventDeregisterAny:17972 : conn=0x7f41a0231a70,
callbackID=15
2013-10-07 23:20:59.592+0000: 1192: debug :
virConnectDomainEventDeregisterAny:17972 : conn=0x7f41a0231a70,
callbackID=16
2013-10-07 23:20:59.592+0000: 1192: debug :
virConnectDomainEventDeregisterAny:17972 : conn=0x7f41a0231a70,
callbackID=20
2013-10-07 23:20:59.592+0000: 1192: debug :
virConnectDomainEventDeregisterAny:17972 : conn=0x7f41a0231a70,
callbackID=18
2013-10-07 23:20:59.592+0000: 1192: debug :
virConnectDomainEventDeregisterAny:17972 : conn=0x7f41a0231a70,
callbackID=17
2013-10-07 23:20:59.592+0000: 1192: debug :
virConnectDomainEventDeregisterAny:17972 : conn=0x7f41a0231a70,
callbackID=19
2013-10-07 23:20:59.592+0000: 1192: debug : virConnectClose:1494 :
conn=0x7f41a0231a70
2013-10-07 23:20:59.593+0000: 1192: debug :
virQEMUCloseCallbacksRun:882 : conn=0x7f41a0231a70
2013-10-07 23:21:00.587+0000: 1203: debug : virIdentitySetAttr:241 :
ident=0x7f41980620d0 attribute=0 value=vdsm
2013-10-07 23:21:00.587+0000: 1203: debug : virIdentitySetAttr:241 :
ident=0x7f41980620d0 attribute=1 value=kvm
2013-10-07 23:21:00.587+0000: 1203: debug : virIdentitySetAttr:241 :
ident=0x7f41980620d0 attribute=2 value=12891
2013-10-07 23:21:00.587+0000: 1203: debug : virIdentitySetAttr:241 :
ident=0x7f41980620d0 attribute=3 value=11193333
2013-10-07 23:21:00.587+0000: 1203: debug : virIdentitySetAttr:241 :
ident=0x7f41980620d0 attribute=6 value=system_u:system_r:initrc_t:s0
2013-10-07 23:21:00.602+0000: 1207: debug : virConnectOpen:1376 :
name=qemu:///system
2013-10-07 23:21:00.602+0000: 1207: debug :
virConnectGetConfigFile:991 : Loading config file
'/etc/libvirt/libvirt.conf'
2013-10-07 23:21:00.602+0000: 1207: debug : virConfReadFile:767 :
filename=/etc/libvirt/libvirt.conf
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1165 : name
"qemu:///system" to URI components:
scheme qemu
server (null)
user (null)
port 0
path /system
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1211 : trying
driver 0 (Test) ...
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1217 : driver 0
Test returned DECLINED
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1211 : trying
driver 1 (OPENVZ) ...
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1217 : driver 1
OPENVZ returned DECLINED
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1211 : trying
driver 2 (VMWARE) ...
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1217 : driver 2
VMWARE returned DECLINED
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1211 : trying
driver 3 (PHYP) ...
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1217 : driver 3
PHYP returned DECLINED
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1211 : trying
driver 4 (VBOX) ...
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1217 : driver 4
VBOX returned DECLINED
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1211 : trying
driver 5 (ESX) ...
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1217 : driver 5
ESX returned DECLINED
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1211 : trying
driver 6 (Hyper-V) ...
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1217 : driver 6
Hyper-V returned DECLINED
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1211 : trying
driver 7 (remote) ...
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1217 : driver 7
remote returned DECLINED
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1211 : trying
driver 8 (Xen) ...
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1217 : driver 8
Xen returned DECLINED
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1211 : trying
driver 9 (xenlight) ...
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1217 : driver 9
xenlight returned DECLINED
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1211 : trying
driver 10 (QEMU) ...
2013-10-07 23:21:00.602+0000: 1207: debug : do_open:1217 : driver 10
QEMU returned SUCCESS
...
- In vm.log under qemu of source host
2013-10-07 23:17:56.249+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -name c8again32 -S -machine
pc-1.0,accel=kvm,usb=off -cpu Opteron_G2 -m 1024 -smp
1,sockets=1,cores=1,threads=1 -uuid
d54660a2-45ed-41ae-ab99-a6f93ebbdbb1 -smbios
type=1,manufacturer=oVirt,product=oVirt
Node,version=19-4,serial=34353439-3036-435A-4A38-303330393338,uuid=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1
-no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/c8again32.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=2013-10-07T23:16:01,driftfix=slew -no-shutdown -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive
file=gluster://f18ovn01.mydomain/gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49,if=none,id=drive-virtio-disk0,format=raw,serial=15f9ca1c-c435-4892-9eb7-0c84583b2a7d,cache=none,werror=stop,rerror=stop,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:8f:04:f7,bus=pci.0,addr=0x3
-chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/d54660a2-45ed-41ae-ab99-a6f93ebbdbb1.com.redhat.rhevm.vdsm,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
-chardev
socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/d54660a2-45ed-41ae-ab99-a6f93ebbdbb1.org.qemu.guest_agent.0,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
-chardev spicevmc,id=charchannel2,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
-spice
tls-port=5900,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on
-k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global
qxl-vga.vram_size=67108864 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
main_channel_link: add main channel client
main_channel_handle_parsed: net test: latency 2.694000 ms, bitrate
521517697 bps (497.358033 Mbps)
red_dispatcher_set_cursor_peer:
inputs_connect: inputs channel client create
main_channel_handle_parsed: agent start
red_channel_client_disconnect: rcc=0x7f2c44243c40
(channel=0x7f2c4421f360 type=2 id=0)
red_channel_client_disconnect: rcc=0x7f2cbad97110
(channel=0x7f2cbad13130 type=3 id=0)
red_channel_client_disconnect: rcc=0x7f2cbad92d80
(channel=0x7f2cbad07f30 type=1 id=0)
main_channel_client_on_disconnect: rcc=0x7f2cbad92d80
red_client_destroy: destroy client 0x7f2cbad8e1c0 with #channels=4
red_dispatcher_disconnect_cursor_peer:
red_channel_client_disconnect: rcc=0x7f2c442a0f00
(channel=0x7f2c4421f920 type=4 id=0)
red_dispatcher_disconnect_display_peer:
2013-10-07 23:53:29.384+0000: shutting down
- In vm.log under qemu of dest host
2013-10-07 23:20:54.260+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -name c8again32 -S -machine
pc-1.0,accel=kvm,usb=off -cpu Opteron_G2 -m 1024 -smp
1,sockets=1,cores=1,threads=1 -uuid
d54660a2-45ed-41ae-ab99-a6f93ebbdbb1 -smbios
type=1,manufacturer=oVirt,product=oVirt
Node,version=19-4,serial=34353439-3036-435A-4A38-303330393338,uuid=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1
-no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/c8again32.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=2013-10-07T23:18:59,driftfix=slew -no-shutdown -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive
file=gluster://f18ovn01.mydomain/gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49,if=none,id=drive-virtio-disk0,format=raw,serial=15f9ca1c-c435-4892-9eb7-0c84583b2a7d,cache=none,werror=stop,rerror=stop,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=31 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:8f:04:f7,bus=pci.0,addr=0x3
-chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/d54660a2-45ed-41ae-ab99-a6f93ebbdbb1.com.redhat.rhevm.vdsm,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
-chardev
socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/d54660a2-45ed-41ae-ab99-a6f93ebbdbb1.org.qemu.guest_agent.0,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
-chardev spicevmc,id=charchannel2,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
-spice
tls-port=5900,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on
-k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global
qxl-vga.vram_size=67108864 -incoming tcp:[::]:49153 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
-incoming tcp:[::]:49153: Failed to bind socket: Address already in use
2013-10-07 23:20:54.771+0000: shutting down
Thanks
Gianluca
8:43 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On Wed, Oct 09, 2013 at 02:42:22PM +0200, Gianluca Cecchi wrote:
On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote:
>
>>
>> But migration still fails
>>
>
> It seems like an unrelated failure. I do not know what's blocking
> migration traffic. Could you see if libvirtd.log and qemu logs at source
> and destinaiton have clues?
>
It seems that on VM.log under qemu of desdt host I have:
...
-incoming tcp:[::]:49153: Failed to bind socket: Address already in use
Is that port really taken (`ss -ntp` should tell by whom)?
See all:
- In libvirtd.log of source host
2013-10-07 23:20:54.471+0000: 1209: debug :
qemuMonitorOpenInternal:751 : QEMU_MONITOR_NEW: mon=0x7fc66412e820
refs=2 fd=30
2013-10-07 23:20:54.472+0000: 1209: warning :
qemuDomainObjEnterMonitorInternal:1136 : This thread seems to be the
async job owner; entering monitor without asking for a nested job is
dangerous
2013-10-07 23:20:54.472+0000: 1209: debug :
qemuMonitorSetCapabilities:1145 : mon=0x7fc66412e820
2013-10-07 23:20:54.472+0000: 1209: debug : qemuMonitorSend:887 :
QEMU_MONITOR_SEND_MSG: mon=0x7fc66412e820
msg={"execute":"qmp_capabilities","id":"libvirt-1"}
fd=-1
2013-10-07 23:20:54.769+0000: 1199: error : qemuMonitorIORead:505 :
Unable to read from monitor: Connection reset by peer
2013-10-07 23:20:54.769+0000: 1199: debug : qemuMonitorIO:638 : Error
on monitor Unable to read from monitor: Connection reset by peer
2013-10-07 23:20:54.769+0000: 1199: debug : qemuMonitorIO:672 :
Triggering error callback
2013-10-07 23:20:54.769+0000: 1199: debug :
qemuProcessHandleMonitorError:351 : Received error on 0x7fc664124fb0
'c8again32'
2013-10-07 23:20:54.769+0000: 1209: debug : qemuMonitorSend:899 : Send
command resulted in error Unable to read from monitor: Connection
reset by peer
2013-10-07 23:20:54.770+0000: 1199: debug : qemuMonitorIO:638 : Error
on monitor Unable to read from monitor: Connection reset by peer
2013-10-07 23:20:54.770+0000: 1209: debug : virFileMakePathHelper:1283
: path=/var/run/libvirt/qemu mode=0777
2013-10-07 23:20:54.770+0000: 1199: debug : qemuMonitorIO:661 :
Triggering EOF callback
2013-10-07 23:20:54.770+0000: 1199: debug :
qemuProcessHandleMonitorEOF:294 : Received EOF on 0x7fc664124fb0
'c8again32'
2013-10-07 23:20:54.770+0000: 1209: debug : qemuProcessStop:3992 :
Shutting down VM 'c8again32' pid=18053 flags=0
2013-10-07 23:20:54.771+0000: 1209: error :
virNWFilterDHCPSnoopEnd:2135 : internal error ifname "vnet0" not in
key map
2013-10-07 23:20:54.782+0000: 1209: debug : virCommandRunAsync:2251 :
About to run /bin/sh -c 'IPT="/usr/sbin/iptables"
$IPT -D libvirt-out -m physdev --physdev-is-bridged --physdev-out
vnet0 -g FO-vnet0
$IPT -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
$IPT -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0
$IPT -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0
$IPT -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT
$IPT -F FO-vnet0
$IPT -X FO-vnet0
$IPT -F FI-vnet0
$IPT -X FI-vnet0
$IPT -F HI-vnet0
$IPT -X HI-vnet0
IPT="/usr/sbin/ip6tables"
$IPT -D libvirt-out -m physdev --physdev-is-bridged --physdev-out
vnet0 -g FO-vnet0
$IPT -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
$IPT -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0
$IPT -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0
$IPT -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT
$IPT -F FO-vnet0
$IPT -X FO-vnet0
$IPT -F FI-vnet0
$IPT -X FI-vnet0
$IPT -F HI-vnet0
$IPT -X HI-vnet0
EBT="/usr/sbin/ebtables"
$EBT -t nat -D PREROUTING -i vnet0 -j libvirt-I-vnet0
$EBT -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0
EBT="/usr/sbin/ebtables"
collect_chains()
{
for tmp2 in $*; do
for tmp in $($EBT -t nat -L $tmp2 | \
sed -n "/Bridge chain/,\$ s/.*-j \\([IO]-.*\\)/\\1/p");
do
echo $tmp
collect_chains $tmp
done
done
}
rm_chains()
{
for tmp in $*; do $EBT -t nat -F $tmp; done
for tmp in $*; do $EBT -t nat -X $tmp; done
}
tmp='\''
'\''
IFS='\'' '\'''\'' '\''$tmp
chains="$(collect_chains libvirt-I-vnet0 libvirt-O-vnet0)"
$EBT -t nat -F libvirt-I-vnet0
$EBT -t nat -F libvirt-O-vnet0
rm_chains $chains
$EBT -t nat -F libvirt-I-vnet0
$EBT -t nat -X libvirt-I-vnet0
$EBT -t nat -F libvirt-O-vnet0
$EBT -t nat -X libvirt-O-vnet0
'
2013-10-07 23:20:54.784+0000: 1209: debug : virCommandRunAsync:2256 :
Command result 0, with PID 18076
2013-10-07 23:20:54.863+0000: 1209: debug : virCommandRun:2125 :
Result exit status 255, stdout: '' stderr: 'iptables v1.4.18: goto
'FO-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.18: goto 'FO-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.18: goto 'FI-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.18: goto 'HI-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables: Bad rule (does a matching rule exist in that chain?).
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
ip6tables v1.4.18: goto 'FO-vnet0' is not a chain
Try `ip6tables -h' or 'ip6tables --help' for more information.
ip6tables v1.4.18: goto 'FO-vnet0' is not a chain
Try `ip6tables -h' or 'ip6tables --help' for more information.
ip6tables v1.4.18: goto 'FI-vnet0' is not a chain
Try `ip6tables -h' or 'ip6tables --help' for more information.
ip6tables v1.4.18: goto 'HI-vnet0' is not a chain
Try `ip6tables -h' or 'ip6tables --help' for more information.
ip6tables: Bad rule (does a matching rule exist in that chain?).
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
Illegal target name 'libvirt-O-vnet0'.
Chain 'libvirt-O-vnet0' doesn't exist.
Chain 'libvirt-O-vnet0' doesn't exist.
Chain 'libvirt-O-vnet0' doesn't exist.
Chain 'libvirt-O-vnet0' doesn't exist.
'
2013-10-07 23:20:54.863+0000: 1209: debug : qemuMonitorClose:821 :
QEMU_MONITOR_CLOSE: mon=0x7fc66412e820 refs=2
2013-10-07 23:20:54.863+0000: 1209: debug : qemuProcessKill:3951 :
vm=c8again32 pid=18053 flags=5
2013-10-07 23:20:54.863+0000: 1209: debug :
virProcessKillPainfully:269 : vpid=18053 force=1
2013-10-07 23:20:54.863+0000: 1209: debug : qemuDomainCleanupRun:2132
: driver=0x7fc664024cd0, vm=c8again32
2013-10-07 23:20:54.863+0000: 1209: debug :
qemuProcessAutoDestroyRemove:4504 : vm=c8again32
2013-10-07 23:20:54.863+0000: 1209: debug :
virQEMUCloseCallbacksUnset:744 : vm=c8again32,
uuid=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, cb=0x7fc66b6fe570
2013-10-07 23:20:54.864+0000: 1209: error :
virPortAllocatorRelease:174 : port 0 must be in range (5900, 65535)
2013-10-07 23:20:54.865+0000: 1209: debug : qemuDomainObjEndJob:1070 :
Stopping job: none (async=migration in)
2013-10-07 23:20:54.865+0000: 1209: debug :
qemuDomainObjEndAsyncJob:1088 : Stopping async job: migration in
2013-10-07 23:20:54.865+0000: 1199: debug :
qemuProcessHandleMonitorEOF:306 : Domain 0x7fc664124fb0 is not active,
ignoring EOF
Caught Segmentation violation dumping internal log buffer:
This line one seems ominous. Can libvir-list help with it?
9:52 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On Wed, Oct 9, 2013 at 3:43 PM, Dan Kenigsberg wrote:
On Wed, Oct 09, 2013 at 02:42:22PM +0200, Gianluca Cecchi wrote:
> On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote:
>
> >
> >>
> >> But migration still fails
> >>
> >
> > It seems like an unrelated failure. I do not know what's blocking
> > migration traffic. Could you see if libvirtd.log and qemu logs at source
> > and destinaiton have clues?
> >
>
> It seems that on VM.log under qemu of desdt host I have:
> ...
> -incoming tcp:[::]:49153: Failed to bind socket: Address already in use
Is that port really taken (`ss -ntp` should tell by whom)?
yeah !
It seems gluster uses it on both sides
On destination
[root@f18ovn01 qemu]# ss -ntp |egrep "State|49153"
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.3.1:975
192.168.3.1:49153 users:(("glusterfs",31166,7))
ESTAB 0 0 192.168.3.1:49153
192.168.3.3:972 users:(("glusterfsd",18615,14))
ESTAB 0 0 192.168.3.1:49153
192.168.3.1:965 users:(("glusterfsd",18615,13))
ESTAB 0 0 192.168.3.1:963
192.168.3.3:49153 users:(("glusterfs",31152,17))
ESTAB 0 0 192.168.3.1:49153
192.168.3.1:975 users:(("glusterfsd",18615,9))
ESTAB 0 0 192.168.3.1:49153
192.168.3.3:966 users:(("glusterfsd",18615,15))
ESTAB 0 0 192.168.3.1:965
192.168.3.1:49153 users:(("glusterfs",31152,7))
ESTAB 0 0 192.168.3.1:960
192.168.3.3:49153 users:(("glusterfs",31166,11))
...
[root@f18ovn01 qemu]# ps -ef|grep 31166
root 14950 10958 0 16:50 pts/0 00:00:00 grep --color=auto 31166
root 31166 1 0 Oct07 ? 00:00:04 /usr/sbin/glusterfs -s
localhost --volfile-id gluster/glustershd -p
/var/lib/glusterd/glustershd/run/glustershd.pid -l
/var/log/glusterfs/glustershd.log -S
/var/run/626066f6d74e376808c27ad679a1e85c.socket --xlator-option
*replicate*.node-uuid=ebaf2f1a-65a8-409a-b911-6e631a5f182f
[root@f18ovn01 qemu]# lsof -Pp 31166|grep 49153
glusterfs 31166 root 7u IPv4 4703891 0t0
TCP f18ovn01.mydomain:975->f18ovn01.mydomain:49153 (ESTABLISHED)
glusterfs 31166 root 11u IPv4 4780563 0t0
TCP f18ovn01.mydomain:960->f18ovn03.mydomain:49153 (ESTABLISHED)
not so good indeed ;-)
10:18 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On Wed, Oct 09, 2013 at 04:52:20PM +0200, Gianluca Cecchi wrote:
On Wed, Oct 9, 2013 at 3:43 PM, Dan Kenigsberg wrote:
> On Wed, Oct 09, 2013 at 02:42:22PM +0200, Gianluca Cecchi wrote:
>> On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote:
>>
>> >
>> >>
>> >> But migration still fails
>> >>
>> >
>> > It seems like an unrelated failure. I do not know what's blocking
>> > migration traffic. Could you see if libvirtd.log and qemu logs at source
>> > and destinaiton have clues?
>> >
>>
>> It seems that on VM.log under qemu of desdt host I have:
>> ...
>> -incoming tcp:[::]:49153: Failed to bind socket: Address already in use
>
> Is that port really taken (`ss -ntp` should tell by whom)?
yeah !
It seems gluster uses it on both sides
Since libvirt has been using this port range first, would you open a
bug on gluster to avoid it?
Dan. (prays that Vdsm is not expected to edit libvirt's migration port
range on installation)
10:52 a.m.
New subject: [Users] [libvirt] Migration issues with ovirt 3.3
Le mercredi 09 octobre 2013 à 16:18 +0100, Dan Kenigsberg a écrit :
Since libvirt has been using this port range first, would you open a
bug on gluster to avoid it?
Already done: https://bugzilla.redhat.com/show_bug.cgi?id=987555
(not using ovirt, but the problem is easy to trigger as soon as you use
GlusterFS and libvirt on the same boxes)
Regards, Daniel
Dan. (prays that Vdsm is not expected to edit libvirt's migration port
range on installation)
--
Daniel Berteaud
FIREWALL-SERVICES SARL.
Société de Services en Logiciels Libres
Technopôle Montesquieu
33650 MARTILLAC
Tel : 05 56 64 15 32
Fax : 05 56 64 15 32
Web : http://www.firewall-services.com
6:20 p.m.
New subject: [Users] [libvirt] Migration issues with ovirt 3.3
On Wed, Oct 9, 2013 at 5:52 PM, Daniel Berteaud wrote:
Le mercredi 09 octobre 2013 à 16:18 +0100, Dan Kenigsberg a écrit :
>
> Since libvirt has been using this port range first, would you open a
> bug on gluster to avoid it?
Already done: https://bugzilla.redhat.com/show_bug.cgi?id=987555
(not using ovirt, but the problem is easy to trigger as soon as you use
GlusterFS and libvirt on the same boxes)
Regards, Daniel
>
> Dan. (prays that Vdsm is not expected to edit libvirt's migration port
> range on installation)
I added my point to the bugzilla and everyone on oVirt should do so in
my opinion... very frustrating.. ;-(
And it seems not so easy to change gluster range.
I read in past threads for previous versions that the range 24009+ was
written inside the code.
In 3.4 it should be an option such as
--volfile-server-port=PORT
from a client point of view... but I tried some things and not able to
arrive at a significant point... always 49152 and 49153 used for my
two bricks
Furthermore, as a side note, I wanted to try migration reattempt it
several times as each time it chooses next port.
I have two bricks so that the first two attempts fail; in VM.log I get
...
-incoming tcp:[::]:49152: Failed to bind socket: Address already in use
...
-incoming tcp:[::]:49153: Failed to bind socket: Address already in use
Unfortunately, tested three times with same result, after the second
attempt the dest node goes into a loop of failing, recoverying from
crash, non reposnive.
I easily correct the situation with
systemctl restart vdsmd
on it, but possibly I could have raised some sort of bug.
And then if I try again to migrate it restarts from 49152... so I
cannot test it.
See here below a file with what written in vdsmd.log, with the two
attempts and the failure and the loop of type
Thread-61::DEBUG::2013-10-10
01:15:32,161::BindingXMLRPC::986::vds::(wrapper) return
getCapabilities with {'status': {'message': 'Recovering from crash or
Initializing', 'code': 99}}
https://docs.google.com/file/d/0BwoPbcrMv8mvamR3RmpzLU11OFE/edit?usp=sharing
1:40 a.m.
New subject: [Users] [libvirt] Migration issues with ovirt 3.3
On Wed, Oct 09, 2013 at 16:18:25 +0100, Dan Kenigsberg wrote:
On Wed, Oct 09, 2013 at 04:52:20PM +0200, Gianluca Cecchi wrote:
> On Wed, Oct 9, 2013 at 3:43 PM, Dan Kenigsberg wrote:
> > On Wed, Oct 09, 2013 at 02:42:22PM +0200, Gianluca Cecchi wrote:
> >> On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote:
> >>
> >> >
> >> >>
> >> >> But migration still fails
> >> >>
> >> >
> >> > It seems like an unrelated failure. I do not know what's blocking
> >> > migration traffic. Could you see if libvirtd.log and qemu logs at
source
> >> > and destinaiton have clues?
> >> >
> >>
> >> It seems that on VM.log under qemu of desdt host I have:
> >> ...
> >> -incoming tcp:[::]:49153: Failed to bind socket: Address already in use
> >
> > Is that port really taken (`ss -ntp` should tell by whom)?
>
> yeah !
> It seems gluster uses it on both sides
Since libvirt has been using this port range first, would you open a
bug on gluster to avoid it?
Dan. (prays that Vdsm is not expected to edit libvirt's migration port
range on installation)
If it makes you feel better, you can't even do that :-) Unfortunately,
the port range used for migration is hard-coded in libvirt. And since we
don't use port allocator yet (patches are in progress), we don't
automatically avoid ports that are already taken. All this is going to
change, though.
Jirka
4:41 p.m.
New subject: [Users] [libvirt] Migration issues with ovirt 3.3
On Thu, Oct 10, 2013 at 08:40:29AM +0200, Jiri Denemark wrote:
On Wed, Oct 09, 2013 at 16:18:25 +0100, Dan Kenigsberg wrote:
> On Wed, Oct 09, 2013 at 04:52:20PM +0200, Gianluca Cecchi wrote:
> > On Wed, Oct 9, 2013 at 3:43 PM, Dan Kenigsberg wrote:
> > > On Wed, Oct 09, 2013 at 02:42:22PM +0200, Gianluca Cecchi wrote:
> > >> On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote:
> > >>
> > >> >
> > >> >>
> > >> >> But migration still fails
> > >> >>
> > >> >
> > >> > It seems like an unrelated failure. I do not know what's
blocking
> > >> > migration traffic. Could you see if libvirtd.log and qemu logs at
source
> > >> > and destinaiton have clues?
> > >> >
> > >>
> > >> It seems that on VM.log under qemu of desdt host I have:
> > >> ...
> > >> -incoming tcp:[::]:49153: Failed to bind socket: Address already in
use
> > >
> > > Is that port really taken (`ss -ntp` should tell by whom)?
> >
> > yeah !
> > It seems gluster uses it on both sides
>
> Since libvirt has been using this port range first, would you open a
> bug on gluster to avoid it?
>
> Dan. (prays that Vdsm is not expected to edit libvirt's migration port
> range on installation)
If it makes you feel better, you can't even do that :-) Unfortunately,
the port range used for migration is hard-coded in libvirt. And since we
don't use port allocator yet (patches are in progress), we don't
automatically avoid ports that are already taken. All this is going to
change, though.
Are these patches posted? Is there a libvirt BZ that tracks this issue?
Dan.
12:07 p.m.
New subject: [Users] [libvirt] Migration issues with ovirt 3.3
On Fri, Oct 11, 2013 at 11:41 PM, Dan Kenigsberg wrote:
> > >
> > > yeah !
> > > It seems gluster uses it on both sides
> >
> > Since libvirt has been using this port range first, would you open a
> > bug on gluster to avoid it?
> >
> > Dan. (prays that Vdsm is not expected to edit libvirt's migration port
> > range on installation)
>
> If it makes you feel better, you can't even do that :-) Unfortunately,
> the port range used for migration is hard-coded in libvirt. And since we
> don't use port allocator yet (patches are in progress), we don't
> automatically avoid ports that are already taken. All this is going to
> change, though.
Are these patches posted? Is there a libvirt BZ that tracks this issue?
Dan.
In the mean time I got 3.4.1 source rpm for fedora 19 and rebuilt with
patches review as in comment#2 of
https://bugzilla.redhat.com/show_bug.cgi?id=1018178
Now range is 50152+ and I'm able to successfully migrate in oVirt my
CentOS 6.4 VM
Gianluca
3:30 a.m.
New subject: [Users] [libvirt] Migration issues with ovirt 3.3
On Fri, Oct 11, 2013 at 22:41:32 +0100, Dan Kenigsberg wrote:
On Thu, Oct 10, 2013 at 08:40:29AM +0200, Jiri Denemark wrote:
> On Wed, Oct 09, 2013 at 16:18:25 +0100, Dan Kenigsberg wrote:
> > On Wed, Oct 09, 2013 at 04:52:20PM +0200, Gianluca Cecchi wrote:
> > > On Wed, Oct 9, 2013 at 3:43 PM, Dan Kenigsberg wrote:
> > > > On Wed, Oct 09, 2013 at 02:42:22PM +0200, Gianluca Cecchi wrote:
> > > >> On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote:
> > > >>
> > > >> >
> > > >> >>
> > > >> >> But migration still fails
> > > >> >>
> > > >> >
> > > >> > It seems like an unrelated failure. I do not know what's
blocking
> > > >> > migration traffic. Could you see if libvirtd.log and qemu
logs at source
> > > >> > and destinaiton have clues?
> > > >> >
> > > >>
> > > >> It seems that on VM.log under qemu of desdt host I have:
> > > >> ...
> > > >> -incoming tcp:[::]:49153: Failed to bind socket: Address already
in use
> > > >
> > > > Is that port really taken (`ss -ntp` should tell by whom)?
> > >
> > > yeah !
> > > It seems gluster uses it on both sides
> >
> > Since libvirt has been using this port range first, would you open a
> > bug on gluster to avoid it?
> >
> > Dan. (prays that Vdsm is not expected to edit libvirt's migration port
> > range on installation)
>
> If it makes you feel better, you can't even do that :-) Unfortunately,
> the port range used for migration is hard-coded in libvirt. And since we
> don't use port allocator yet (patches are in progress), we don't
> automatically avoid ports that are already taken. All this is going to
> change, though.
Are these patches posted? Is there a libvirt BZ that tracks this issue?
https://www.redhat.com/archives/libvir-list/2013-October/msg00513.html
is v3 of a patch that makes libvirt skip ports that are already in use.
Of course, if something binds to that port just before QEMU is started,
it will still fail. But that's a separate issue that may only be solved
by giving QEMU a socket which is already bound rather than telling QEMU
to bind to it.
A patch for making the port range configurable was not post yet...
BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1018530
Jirka
10:12 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On Wed, Oct 2, 2013 at 12:07 AM, Jason Brooks wrote:
I'm having this issue on my ovirt 3.3 setup (two node, one is
AIO,
GlusterFS storage, both on F19) as well.
Jason
Me too with oVirt 3.3 setup and GlusterFS DataCenter.
One dedicated engine + 2 vdsm hosts. All fedora 19 + ovirt stable.
My trying to migrate VM is CentOS 6.4 fully updated
After trying to migrate in webadmin gui I get:
VM c6s is down. Exit message: 'iface'.
For a few moments the VM appears as down in the GUI but actually an
ssh session I had before is still alive.
Also the qemu process on source host is still there.
After a little, the VM results again as "Up" in the source node from the gui
Actually it never stopped:
[root@c6s ~]# uptime
16:36:56 up 19 min, 1 user, load average: 0.00, 0.00, 0.00
At target many errors such as
Thread-8609::ERROR::2013-10-03
16:17:13,086::task::850::TaskManager.Task::(_setError)
Task=`a102541e-fbe5-46a3-958c-e5f4026cac8c`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 857, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 2123, in getAllTasksStatuses
allTasksStatus = sp.getAllTasksStatuses()
File "/usr/share/vdsm/storage/securable.py", line 66, in wrapper
raise SecureError()
SecureError
But relevant perhaps is around the time of migration (16:33-16:34)
Thread-9968::DEBUG::2013-10-03
16:33:38,811::task::1168::TaskManager.Task::(prepare)
Task=`6c1d3161-edcd-4344-a32a-4a18f75f5ba3`::finished: {'taskStatus':
{'code': 0, 'message': 'Task is initializing',
'taskState': 'running',
'taskResult': '', 'taskID':
'0eaac2f3-3d25-4c8c-9738-708aba290404'}}
Thread-9968::DEBUG::2013-10-03
16:33:38,811::task::579::TaskManager.Task::(_updateState)
Task=`6c1d3161-edcd-4344-a32a-4a18f75f5ba3`::moving from state
preparing -> state finished
Thread-9968::DEBUG::2013-10-03
16:33:38,811::resourceManager::939::ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-9968::DEBUG::2013-10-03
16:33:38,812::resourceManager::976::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-9968::DEBUG::2013-10-03
16:33:38,812::task::974::TaskManager.Task::(_decref)
Task=`6c1d3161-edcd-4344-a32a-4a18f75f5ba3`::ref 0 aborting False
0eaac2f3-3d25-4c8c-9738-708aba290404::ERROR::2013-10-03
16:33:38,847::task::850::TaskManager.Task::(_setError)
Task=`0eaac2f3-3d25-4c8c-9738-708aba290404`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 857, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/storage/task.py", line 318, in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/share/vdsm/storage/sp.py", line 272, in startSpm
self.masterDomain.acquireHostId(self.id)
File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
self._clusterLock.acquireHostId(hostId, async)
File "/usr/share/vdsm/storage/clusterlock.py", line 189, in acquireHostId
raise se.AcquireHostIdFailure(self._sdUUID, e)
AcquireHostIdFailure: Cannot acquire host id:
('d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', SanlockException(5, 'Sanlock
lockspace add failure', 'Input/output error'))
0eaac2f3-3d25-4c8c-9738-708aba290404::DEBUG::2013-10-03
16:33:38,847::task::869::TaskManager.Task::(_run)
Task=`0eaac2f3-3d25-4c8c-9738-708aba290404`::Task._run:
0eaac2f3-3d25-4c8c-9738-708aba290404 () {} failed - stopping task
0eaac2f3-3d25-4c8c-9738-708aba290404::DEBUG::2013-10-03
16:33:38,847::task::1194::TaskManager.Task::(stop)
Task=`0eaac2f3-3d25-4c8c-9738-708aba290404`::stopping in state running
(force False)
Instead at source:
Thread-10402::ERROR::2013-10-03
16:35:03,713::vm::244::vm.Vm::(_recover)
vmId=`4147e0d3-19a7-447b-9d88-2ff19365bec0`::migration destination
error: Error creating the requested VM
Thread-10402::ERROR::2013-10-03 16:35:03,740::vm::324::vm.Vm::(run)
vmId=`4147e0d3-19a7-447b-9d88-2ff19365bec0`::Failed to migrate
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 311, in run
self._startUnderlyingMigration()
File "/usr/share/vdsm/vm.py", line 347, in _startUnderlyingMigration
response['status']['message'])
RuntimeError: migration destination error: Error creating the requested VM
Thread-1161::DEBUG::2013-10-03
16:35:04,243::fileSD::238::Storage.Misc.excCmd::(getReadDelay)
'/usr/bin/dd iflag=direct
if=/rhev/data-center/mnt/glusterSD/f18ovn01.mydomain:gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/metadata
bs=4096 count=1' (cwd None)
Thread-1161::DEBUG::2013-10-03
16:35:04,262::fileSD::238::Storage.Misc.excCmd::(getReadDelay)
SUCCESS: <err> = '0+1 records in\n0+1 records out\n512 bytes (512 B)
copied, 0.0015976 s, 320 kB/s\n'; <rc> = 0
Thread-1161::INFO::2013-10-03
16:35:04,269::clusterlock::174::SANLock::(acquireHostId) Acquiring
host id for domain d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291 (id: 2)
Thread-1161::DEBUG::2013-10-03
16:35:04,270::clusterlock::192::SANLock::(acquireHostId) Host id for
domain d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291 successfully acquired (id:
2)
Full logs here:
https://docs.google.com/file/d/0BwoPbcrMv8mvNVFBaGhDeUdMOFE/edit?usp=sharing
in the zip file:
engine.log
vdsm_source.log
vdsm_target.log
ovirt gluster 3.3 datacenter status.png
(note as many times it seems from a gui point of view that DC set to
non responsive...?)
One strange thing:
In Clusters --> Gluster01 --> Volumes I see my gluster volume (gvdata) as up
while in DC --> Gluster (My DC name) Storage --> gvdata (my storage
domain name) here is marked as down, but the related C6 VM is always
up and running???
source:
[root@f18ovn03 vdsm]# df -h | grep rhev
f18ovn01.mydomain:gvdata 30G 4.7G 26G 16%
/rhev/data-center/mnt/glusterSD/f18ovn01.mydomain:gvdata
f18engine.mydomain:/var/lib/exports/iso 35G 9.4G 24G 29%
/rhev/data-center/mnt/f18engine.mydomain:_var_lib_exports_iso
target:
[root@f18ovn01 vdsm]# df -h | grep rhev
f18ovn01.mydomain:gvdata 30G 4.7G 26G 16%
/rhev/data-center/mnt/glusterSD/f18ovn01.mydomain:gvdata
f18engine.mydomain:/var/lib/exports/iso 35G 9.4G 24G 29%
/rhev/data-center/mnt/f18engine.mydomain:_var_lib_exports_iso
I didn't try to activate again because I didn't want to make worse things:
it seems strange to me that both hosts are marked as up but the only
existing Storage Domain (so marked as "Data (Master)) in the Gui is
down.....
Gianluca
4:29 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On 10/03/2013 06:12 PM, Gianluca Cecchi wrote:
On Wed, Oct 2, 2013 at 12:07 AM, Jason Brooks wrote:
> I'm having this issue on my ovirt 3.3 setup (two node, one is AIO,
> GlusterFS storage, both on F19) as well.
>
> Jason
Me too with oVirt 3.3 setup and GlusterFS DataCenter.
One dedicated engine + 2 vdsm hosts. All fedora 19 + ovirt stable.
My trying to migrate VM is CentOS 6.4 fully updated
centos qemu/libvirt versions don't support glusterfs storage domain yet.
6.5 would be needed. we're trying to find a solution until then.
After trying to migrate in webadmin gui I get:
VM c6s is down. Exit message: 'iface'.
For a few moments the VM appears as down in the GUI but actually an
ssh session I had before is still alive.
Also the qemu process on source host is still there.
After a little, the VM results again as "Up" in the source node from the gui
Actually it never stopped:
[root@c6s ~]# uptime
16:36:56 up 19 min, 1 user, load average: 0.00, 0.00, 0.00
At target many errors such as
Thread-8609::ERROR::2013-10-03
16:17:13,086::task::850::TaskManager.Task::(_setError)
Task=`a102541e-fbe5-46a3-958c-e5f4026cac8c`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 857, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 2123, in getAllTasksStatuses
allTasksStatus = sp.getAllTasksStatuses()
File "/usr/share/vdsm/storage/securable.py", line 66, in wrapper
raise SecureError()
SecureError
But relevant perhaps is around the time of migration (16:33-16:34)
Thread-9968::DEBUG::2013-10-03
16:33:38,811::task::1168::TaskManager.Task::(prepare)
Task=`6c1d3161-edcd-4344-a32a-4a18f75f5ba3`::finished: {'taskStatus':
{'code': 0, 'message': 'Task is initializing',
'taskState': 'running',
'taskResult': '', 'taskID':
'0eaac2f3-3d25-4c8c-9738-708aba290404'}}
Thread-9968::DEBUG::2013-10-03
16:33:38,811::task::579::TaskManager.Task::(_updateState)
Task=`6c1d3161-edcd-4344-a32a-4a18f75f5ba3`::moving from state
preparing -> state finished
Thread-9968::DEBUG::2013-10-03
16:33:38,811::resourceManager::939::ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-9968::DEBUG::2013-10-03
16:33:38,812::resourceManager::976::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-9968::DEBUG::2013-10-03
16:33:38,812::task::974::TaskManager.Task::(_decref)
Task=`6c1d3161-edcd-4344-a32a-4a18f75f5ba3`::ref 0 aborting False
0eaac2f3-3d25-4c8c-9738-708aba290404::ERROR::2013-10-03
16:33:38,847::task::850::TaskManager.Task::(_setError)
Task=`0eaac2f3-3d25-4c8c-9738-708aba290404`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 857, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/storage/task.py", line 318, in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/share/vdsm/storage/sp.py", line 272, in startSpm
self.masterDomain.acquireHostId(self.id)
File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
self._clusterLock.acquireHostId(hostId, async)
File "/usr/share/vdsm/storage/clusterlock.py", line 189, in acquireHostId
raise se.AcquireHostIdFailure(self._sdUUID, e)
AcquireHostIdFailure: Cannot acquire host id:
('d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', SanlockException(5, 'Sanlock
lockspace add failure', 'Input/output error'))
0eaac2f3-3d25-4c8c-9738-708aba290404::DEBUG::2013-10-03
16:33:38,847::task::869::TaskManager.Task::(_run)
Task=`0eaac2f3-3d25-4c8c-9738-708aba290404`::Task._run:
0eaac2f3-3d25-4c8c-9738-708aba290404 () {} failed - stopping task
0eaac2f3-3d25-4c8c-9738-708aba290404::DEBUG::2013-10-03
16:33:38,847::task::1194::TaskManager.Task::(stop)
Task=`0eaac2f3-3d25-4c8c-9738-708aba290404`::stopping in state running
(force False)
Instead at source:
Thread-10402::ERROR::2013-10-03
16:35:03,713::vm::244::vm.Vm::(_recover)
vmId=`4147e0d3-19a7-447b-9d88-2ff19365bec0`::migration destination
error: Error creating the requested VM
Thread-10402::ERROR::2013-10-03 16:35:03,740::vm::324::vm.Vm::(run)
vmId=`4147e0d3-19a7-447b-9d88-2ff19365bec0`::Failed to migrate
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 311, in run
self._startUnderlyingMigration()
File "/usr/share/vdsm/vm.py", line 347, in _startUnderlyingMigration
response['status']['message'])
RuntimeError: migration destination error: Error creating the requested VM
Thread-1161::DEBUG::2013-10-03
16:35:04,243::fileSD::238::Storage.Misc.excCmd::(getReadDelay)
'/usr/bin/dd iflag=direct
if=/rhev/data-center/mnt/glusterSD/f18ovn01.mydomain:gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/metadata
bs=4096 count=1' (cwd None)
Thread-1161::DEBUG::2013-10-03
16:35:04,262::fileSD::238::Storage.Misc.excCmd::(getReadDelay)
SUCCESS: <err> = '0+1 records in\n0+1 records out\n512 bytes (512 B)
copied, 0.0015976 s, 320 kB/s\n'; <rc> = 0
Thread-1161::INFO::2013-10-03
16:35:04,269::clusterlock::174::SANLock::(acquireHostId) Acquiring
host id for domain d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291 (id: 2)
Thread-1161::DEBUG::2013-10-03
16:35:04,270::clusterlock::192::SANLock::(acquireHostId) Host id for
domain d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291 successfully acquired (id:
2)
Full logs here:
https://docs.google.com/file/d/0BwoPbcrMv8mvNVFBaGhDeUdMOFE/edit?usp=sharing
in the zip file:
engine.log
vdsm_source.log
vdsm_target.log
ovirt gluster 3.3 datacenter status.png
(note as many times it seems from a gui point of view that DC set to
non responsive...?)
One strange thing:
In Clusters --> Gluster01 --> Volumes I see my gluster volume (gvdata) as up
while in DC --> Gluster (My DC name) Storage --> gvdata (my storage
domain name) here is marked as down, but the related C6 VM is always
up and running???
source:
[root@f18ovn03 vdsm]# df -h | grep rhev
f18ovn01.mydomain:gvdata 30G 4.7G 26G 16%
/rhev/data-center/mnt/glusterSD/f18ovn01.mydomain:gvdata
f18engine.mydomain:/var/lib/exports/iso 35G 9.4G 24G 29%
/rhev/data-center/mnt/f18engine.mydomain:_var_lib_exports_iso
target:
[root@f18ovn01 vdsm]# df -h | grep rhev
f18ovn01.mydomain:gvdata 30G 4.7G 26G 16%
/rhev/data-center/mnt/glusterSD/f18ovn01.mydomain:gvdata
f18engine.mydomain:/var/lib/exports/iso 35G 9.4G 24G 29%
/rhev/data-center/mnt/f18engine.mydomain:_var_lib_exports_iso
I didn't try to activate again because I didn't want to make worse things:
it seems strange to me that both hosts are marked as up but the only
existing Storage Domain (so marked as "Data (Master)) in the Gui is
down.....
Gianluca
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
4:39 a.m.
New subject: [Users] Migration issues with ovirt 3.3
Hi,
if I understand Gianluca correct he has setup the ManagementNode and
2 ComputeNodes with Fedora 19.
CentOS 6.4 is the VM-OS (inside the VM).
So this should work?
HTH
Sven
On 04/10/13 11:29, Itamar Heim wrote:
>>
>> Me too with oVirt 3.3 setup and GlusterFS DataCenter.
>> One dedicated engine + 2 vdsm hosts. All fedora 19 + ovirt stable.
>> My trying to migrate VM is CentOS 6.4 fully updated
>
> centos qemu/libvirt versions don't support glusterfs storage domain yet.
> 6.5 would be needed. we're trying to find a solution until then.
4:52 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On Fri, Oct 4, 2013 at 11:39 AM, Sven Kieske wrote:
Hi,
if I understand Gianluca correct he has setup the ManagementNode and
2 ComputeNodes with Fedora 19.
CentOS 6.4 is the VM-OS (inside the VM).
So this should work?
Yes, this is my environment with plain F19 systems installed and
updated as of yesterday and ovirt repo stable enabled.
CentOS 6.4 x86_64 was configured as a server in oVirt, but wanting to
generate some significant I/O during install to test Gluster
efficiency I choosed Gnome Desktop in CentOS anaconda phase.
This led to 1089 packages installed.
After reboot I ran "yum update" on the VM, enabled epel and installed
the new ovirt-guest-agent so that I was correctly able to see ip and
kernel of the VM from the webadmin GUI.
But then I incurred in these problems with preliminary tests, donna if
correlated or not:
- migration issue
- shutdown (so power off ) and power on error
see:
https://bugzilla.redhat.com/show_bug.cgi?id=1007980
that coud be related to both of them
- gluster split brain
Gianluca
6:21 a.m.
New subject: [Users] Migration issues with ovirt 3.3
On 10/04/2013 12:39 PM, Sven Kieske wrote:
Hi,
if I understand Gianluca correct he has setup the ManagementNode and
2 ComputeNodes with Fedora 19.
CentOS 6.4 is the VM-OS (inside the VM).
So this should work?
yes, it should.
HTH
Sven
On 04/10/13 11:29, Itamar Heim wrote:
>>
>> Me too with oVirt 3.3 setup and GlusterFS DataCenter.
>> One dedicated engine + 2 vdsm hosts. All fedora 19 + ovirt stable.
>> My trying to migrate VM is CentOS 6.4 fully updated
>
> centos qemu/libvirt versions don't support glusterfs storage domain yet.
> 6.5 would be needed. we're trying to find a solution until then.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
6:17 p.m.
New subject: [Users] Migration issues with ovirt 3.3
On Wed, Oct 2, 2013 at 12:07 AM, Jason Brooks wrote:
I'm having this issue on my ovirt 3.3 setup (two node, one is
AIO,
GlusterFS storage, both on F19) as well.
Jason
I uploaded all my day logs to bugzilla
https://bugzilla.redhat.com/show_bug.cgi?id=1007980
I can reproduce the problem and the migrate phase is what generates
apparently files to heal in gluster volume
VM is running on f18ovn03
before migrate
[root@f18ovn01 vdsm]# gluster volume heal gvdata info
Gathering Heal info on volume gvdata has been successful
Brick 10.4.4.58:/gluster/DATA_GLUSTER/brick1
Number of entries: 0
Brick 10.4.4.59:/gluster/DATA_GLUSTER/brick1
Number of entries: 0
Start migrate that fails with the "iface" error
And now you see:
[root@f18ovn01 vdsm]# gluster volume heal gvdata info
Gathering Heal info on volume gvdata has been successful
Brick 10.4.4.58:/gluster/DATA_GLUSTER/brick1
Number of entries: 1
/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/ids
Brick 10.4.4.59:/gluster/DATA_GLUSTER/brick1
Number of entries: 1
/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/ids
While on f18ovn03:
[root@f18ovn03 vdsm]# gluster volume heal gvdata info
Gathering Heal info on volume gvdata has been successful
Brick 10.4.4.58:/gluster/DATA_GLUSTER/brick1
Number of entries: 0
Brick 10.4.4.59:/gluster/DATA_GLUSTER/brick1
Number of entries: 0
Gianluca
3846
days inactive
3865
days old
38 comments
11 participants
participants (11)
-
Dan Kenigsberg -
Daniel Berteaud -
emitor＠gmail.com -
Gianluca Cecchi -
Itamar Heim -
Jason Brooks -
Jiri Denemark -
Michal Skrivanek -
Michal Skrivanek -
Omer Frenkel -
Sven Kieske