--_9f58fcb7-bfb3-4566-8fab-e81bc92863f7_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable =20
Date: Fri=2C 26 Jul 2013 05:52:44 +0300 From: iheim@redhat.com To: cybertimber2000@hotmail.com CC: danken@redhat.com=3B users@ovirt.org Subject: Re: [Users] oVirt 3.2 - Migration failed due to error: migra= teerr
On 07/26/2013 05:40 AM=2C Nicholas Kesick wrote:
Replies inline.
Date: Thu=2C 25 Jul 2013 22:27:17 +0300 From: danken@redhat.com To: cybertimber2000@hotmail.com CC: users@ovirt.org Subject: Re: [Users] oVirt 3.2 - Migration failed due to error: migrateerr
On Thu=2C Jul 25=2C 2013 at 11:54:40AM -0400=2C Nicholas Kesick w=
rote:
> When I try to migrate a VM=2C any VM=2C between my two hosts=2C= I receive an error that says Migration failed due to error: migrateerr. Lookin= g in the log I don't see any thing that jumps out other than the final me= ssage > > VDSGenericException: VDSErrorException: Failed to MigrateStatus= VDS=2C error =3D Fatal error during migration > > Ovirt-engine is version 3.2.2-1.1.fc18.noarch=2C firewalld is disabled=2C and selinux is permissive.
Please do not say this in public=2C you're hurting Dan Walsh's fe= elings =3B-)
> > ovirt-node version is 2.6.1 on both hosts. > > Any suggestions would be welcome! >
I'd love to see /etc/vdsm/vdsm.log from source and destination. T= he intersting parts start with vmMigrate at the source and with vmMigrationCreate at the destination. Hmm=2C I probably should have pulled that sooner. So=2C I cleared th= e active VDSM (while nothing was running) and libvirtd.log=2C booted one vm= =2C and
I recall seeing his blog posts=2C and I agree. Not sure when I set i= t to permissive... maybe to get the 3.2 install w/ Firewalld setup to complete? I remember that was fixed in 3.2.1. I'll set it back to en= forcing. tried to migrate it. Attached are the logs. It looks like it boils d= own to (from the source): Traceback (most recent call last): File "/usr/share/vdsm/vm.py"=2C line 271=2C in run File "/usr/share/vdsm/libvirtvm.py"=2C line 505=2C in _startUnderlyingMigration File "/usr/share/vdsm/libvirtvm.py"=2C line 541=2C in f File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.p= y"=2C line 111=2C in wrapper File "/usr/lib64/python2.7/site-packages/libvirt.py"=2C line 1178= =2C in migrateToURI2 libvirtError: internal error Attempt to migrate guest to the same ho= st localhost Does this mean my UUIDs are the same? http://vaunaspada.babel.it/blog/?p=3D613 As far as the destination=2C I'm really not understanding what's goi= ng on on the destination between "Destination VM creation succeeded" and ":destroy Called" that would lead to it failing=2C except for what's= after the traceback: Traceback (most recent call last): File "/usr/share/vdsm/vm.py"=2C line 696=2C in _startUnderlyingVm File "/usr/share/vdsm/libvirtvm.py"=2C line 1907=2C in _waitForIncomingMigrationFinish File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.p= y"=2C line 111=2C in wrapper File "/usr/lib64/python2.7/site-packages/libvirt.py"=2C line 2822= =2C in lookupByUUIDString libvirtError: Domain not found: no domain with matching uuid '50171e1b-cf21-41d8-80f3-88ab1b980091' But that is the ID of the VM by the looks of it. Sorry Itamar=2C nothing was written to libvirtd.log after I cleared = it. =20 It could be that libvirtd is still writing to the files that you remove= d from the filesystem. To make sure libvirtd writes to your new file=2C restart the service. There may be clues there on why libvirt thinks tha= t
On Fri=2C Jul 26=2C 2013 at 02:03:28PM -0400=2C Nicholas Kesick wrote: the source and destination are one and the same. =20 =20 When clearing the logs=2C it should be enough to do '> /path/to/libvirtd.log' (in bash). Just checked and it seems some things were logged in there during my test= ing on Friday. I'll attach those.
Thread-800::ERROR::2013-07-26 01:57:16=2C198::vm::198::vm.Vm::(_recov=
er)=20
vmId=3D`50171e1b-cf21-41d8-80f3-88ab1b980091`::internal error Attempt= to=20 migrate guest to the same host localhost Thread-800::ERROR::2013-07-26 01:57:16=2C377::vm::286::vm.Vm::(run)=20 vmId=3D`50171e1b-cf21-41d8-80f3-88ab1b980091`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/vm.py"=2C line 271=2C in run File "/usr/share/vdsm/libvirtvm.py"=2C line 505=2C in=20 _startUnderlyingMigration File "/usr/share/vdsm/libvirtvm.py"=2C line 541=2C in f File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.py= "=2C=20 line 111=2C in wrapper File "/usr/lib64/python2.7/site-packages/libvirt.py"=2C line 1178= =2C in=20 migrateToURI2 libvirtError: internal error Attempt to migrate guest to the same hos= t=20 localhost
what are your hostnames?
"host001" on 192.168.0.103 and "host002" on 192.168.0.104 Even tried changing it=2C no luck.
=20 Are they resolving properly on those hosts? Is there a DNS or /etc/hosts entry related to this? There are /etc/hosts entries on both hosts to each other=2C and a "ping h= ost001" "ping host002" resolves correctly.I do however note that the termin= al session says root@localhost. I wonder if running hostnamectl set-hostnam= e {name} will fix anything. ...and after running hostnamectl set-hostname {= name}=2C migrations are working! I think maybe I found a bug:With the node = in maintenance mode=2C from the oVirt Node (SSH or local) if you go to Netw= ork=2C change the hostname ({newname})=2C and then go down to the configure= d System NIC and press enter=2C it says it is Setting Hostname. Now=2C if y= ou press F2=2C the terminal will show root@{newname}. If you reboot however= =2C under network it will say {newname}=2C but pressing F2 for the terminal= will show root@localhost. If it's localhost=2C it won't migrate.So=2C it l= ooks like the hostname isn't getting written persistantly. Even a hostnamec= tl set-hostname {name} gets lost on reboot. Or am I doing something wrong?
Could it be because the oVirt Node - Network tab - does not have any D= NS servers specified? =20 I do not think so. We do not see "name resolution" errors=2C or name resolutions at all. =20 What does libvirt (src + dst) say about =20 virsh -r capabilities|grep uuid =20 ? If uuids happen to be the same=2C you get the bug that you are reporting. host 001:virsh -r capabilities|grep uuid <uuid>a4dc7de7-e2d3-45f5-b75a-7101f71d2b17</uuid> host002: virsh -r capabilities|grep uuid <uuid>ce66bb7f-fbbb-432b-9f62-5bcf5cb732e4</uuid> =20 Although=2C as Dan says=2C having the same UUID for both hosts will repor= t
Date: Mon=2C 29 Jul 2013 09:56:30 +0200 From: mkletzan@redhat.com To: danken@redhat.com CC: cybertimber2000@hotmail.com=3B users@ovirt.org Subject: Re: [Users] oVirt 3.2 - Migration failed due to error: migrateer= r =20 On 07/27/2013 09:50 PM=2C Dan Kenigsberg wrote: the same error even when hostnames are different. =20 Do you have an UUID set in your libvirtd.conf? What do you have in the following files (if they exist) on both hosts? =20 /sys/devices/virtual/dmi/id/product_uuidhost001:cat /sys/devices/virtual/= dmi/id/product_uuid 44454C4C-3700-1047-8048-C3C04F4C4631 host002:cat /sys/devices/virtual/dmi/i= d/product_uuid 44454C4C-5900-1054-8034-B3C04F4E4631 Those match what I see on the webadmin= under hosts > hostname > hardware information
/sys/class/dmi/id/product_uuidhost001:cat /sys/class/dmi/id/product_uuid 44454C4C-3700-1047-8048-C3C04F4C4631 host002: cat /sys/class/dmi/id/product_uuid 44454C4C-5900-1054-8034-B3C04F4E4631
=20 Martin =
>=3B >=3B>=3B>=3B>=3B File "/usr/share/vdsm/libvirtvm.py"=2C =
>=3B >=3B>=3B>=3B>=3B File "/usr/share/vdsm/libvirtvm.py"=2C =
--_9f58fcb7-bfb3-4566-8fab-e81bc92863f7_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <style><!-- .hmmessage P { margin:0px=3B padding:0px } body.hmmessage { font-size: 12pt=3B font-family:Calibri } --></style></head> <body class=3D'hmmessage'><div dir=3D'ltr'><br> =3B<BR><div>>=3B Date= : Mon=2C 29 Jul 2013 09:56:30 +0200<br>>=3B From: mkletzan@redhat.com<br>= >=3B To: danken@redhat.com<br>>=3B CC: cybertimber2000@hotmail.com=3B u= sers@ovirt.org<br>>=3B Subject: Re: [Users] oVirt 3.2 - Migration failed = due to error: migrateerr<br>>=3B <br>>=3B On 07/27/2013 09:50 PM=2C Dan= Kenigsberg wrote:<br>>=3B >=3B On Fri=2C Jul 26=2C 2013 at 02:03:28PM = -0400=2C Nicholas Kesick wrote:<br>>=3B >=3B>=3B>=3B Date: Fri=2C 2= 6 Jul 2013 05:52:44 +0300<br>>=3B >=3B>=3B>=3B From: iheim@redhat.c= om<br>>=3B >=3B>=3B>=3B To: cybertimber2000@hotmail.com<br>>=3B &= gt=3B>=3B>=3B CC: danken@redhat.com=3B users@ovirt.org<br>>=3B >=3B= >=3B>=3B Subject: Re: [Users] oVirt 3.2 - Migration failed due to error= : migrateerr<br>>=3B >=3B>=3B>=3B<br>>=3B >=3B>=3B>=3B On 0= 7/26/2013 05:40 AM=2C Nicholas Kesick wrote:<br>>=3B >=3B>=3B>=3B&g= t=3B<br>>=3B >=3B>=3B>=3B>=3B Replies inline.<br>>=3B >=3B>= =3B>=3B>=3B >=3B Date: Thu=2C 25 Jul 2013 22:27:17 +0300<br>>=3B &= gt=3B>=3B>=3B>=3B >=3B From: danken@redhat.com<br>>=3B >=3B>= =3B>=3B>=3B >=3B To: cybertimber2000@hotmail.com<br>>=3B >=3B>= =3B>=3B>=3B >=3B CC: users@ovirt.org<br>>=3B >=3B>=3B>=3B>= =3B >=3B Subject: Re: [Users] oVirt 3.2 - Migration failed due to error:= <br>>=3B >=3B>=3B>=3B>=3B migrateerr<br>>=3B >=3B>=3B>=3B= >=3B >=3B<br>>=3B >=3B>=3B>=3B>=3B >=3B On Thu=2C Jul 25= =2C 2013 at 11:54:40AM -0400=2C Nicholas Kesick wrote:<br>>=3B >=3B>= =3B>=3B>=3B >=3B >=3B When I try to migrate a VM=2C any VM=2C betw= een my two hosts=2C I receive<br>>=3B >=3B>=3B>=3B>=3B an error t= hat says Migration failed due to error: migrateerr. Looking in<br>>=3B &g= t=3B>=3B>=3B>=3B the log I don't see any thing that jumps out other t= han the final message<br>>=3B >=3B>=3B>=3B>=3B >=3B >=3B<br>= >=3B >=3B>=3B>=3B>=3B >=3B >=3B VDSGenericException: VDSErro= rException: Failed to MigrateStatusVDS=2C<br>>=3B >=3B>=3B>=3B>= =3B error =3D Fatal error during migration<br>>=3B >=3B>=3B>=3B>= =3B >=3B >=3B<br>>=3B >=3B>=3B>=3B>=3B >=3B >=3B Ovirt-= engine is version 3.2.2-1.1.fc18.noarch=2C firewalld is<br>>=3B >=3B>= =3B>=3B>=3B disabled=2C and selinux is permissive.<br>>=3B >=3B>= =3B>=3B>=3B >=3B<br>>=3B >=3B>=3B>=3B>=3B >=3B Please d= o not say this in public=2C you're hurting Dan Walsh's feelings =3B-)<br>&g= t=3B >=3B>=3B>=3B>=3B >=3B<br>>=3B >=3B>=3B>=3B>=3B I = recall seeing his blog posts=2C and I agree. Not sure when I set it to<br>&= gt=3B >=3B>=3B>=3B>=3B permissive... maybe to get the 3.2 install w= / Firewalld setup to<br>>=3B >=3B>=3B>=3B>=3B complete? I remembe= r that was fixed in 3.2.1. I'll set it back to enforcing.<br>>=3B >=3B&= gt=3B>=3B>=3B >=3B >=3B<br>>=3B >=3B>=3B>=3B>=3B >=3B= >=3B ovirt-node version is 2.6.1 on both hosts.<br>>=3B >=3B>=3B&g= t=3B>=3B >=3B >=3B<br>>=3B >=3B>=3B>=3B>=3B >=3B >=3B= Any suggestions would be welcome!<br>>=3B >=3B>=3B>=3B>=3B >= =3B >=3B<br>>=3B >=3B>=3B>=3B>=3B >=3B<br>>=3B >=3B>= =3B>=3B>=3B >=3B I'd love to see /etc/vdsm/vdsm.log from source and = destination. The<br>>=3B >=3B>=3B>=3B>=3B >=3B intersting part= s start with vmMigrate at the source and with<br>>=3B >=3B>=3B>=3B&= gt=3B >=3B vmMigrationCreate at the destination.<br>>=3B >=3B>=3B&= gt=3B>=3B Hmm=2C I probably should have pulled that sooner. So=2C I clear= ed the active<br>>=3B >=3B>=3B>=3B>=3B VDSM (while nothing was ru= nning) and libvirtd.log=2C booted one vm=2C and<br>>=3B >=3B>=3B>= =3B>=3B tried to migrate it. Attached are the logs. It looks like it boil= s down<br>>=3B >=3B>=3B>=3B>=3B to (from the source):<br>>=3B &= gt=3B>=3B>=3B>=3B Traceback (most recent call last):<br>>=3B >=3B= >=3B>=3B>=3B File "/usr/share/vdsm/vm.py"=2C line 271=2C in run<br= line 505=2C in<br>>=3B >=3B>=3B>=3B>=3B _startUnderlyingMigration= <br>>=3B >=3B>=3B>=3B>=3B File "/usr/share/vdsm/libvirtvm.py"= =2C line 541=2C in f<br>>=3B >=3B>=3B>=3B>=3B File "/usr/lib64= /python2.7/site-packages/vdsm/libvirtconnection.py"=2C<br>>=3B >=3B>= =3B>=3B>=3B line 111=2C in wrapper<br>>=3B >=3B>=3B>=3B>=3B = File "/usr/lib64/python2.7/site-packages/libvirt.py"=2C line 1178=2C in<b= r>>=3B >=3B>=3B>=3B>=3B migrateToURI2<br>>=3B >=3B>=3B>= =3B>=3B libvirtError: internal error Attempt to migrate guest to the same= host<br>>=3B >=3B>=3B>=3B>=3B localhost<br>>=3B >=3B>=3B&g= t=3B>=3B Does this mean my UUIDs are the same?<br>>=3B >=3B>=3B>= =3B>=3B http://vaunaspada.babel.it/blog/?p=3D613<br>>=3B >=3B>=3B&g= t=3B>=3B As far as the destination=2C I'm really not understanding what's= going on<br>>=3B >=3B>=3B>=3B>=3B on the destination between "De= stination VM creation succeeded" and<br>>=3B >=3B>=3B>=3B>=3B ":d= estroy Called" that would lead to it failing=2C except for what's after<br>= >=3B >=3B>=3B>=3B>=3B the traceback:<br>>=3B >=3B>=3B>=3B= >=3B Traceback (most recent call last):<br>>=3B >=3B>=3B>=3B>= =3B File "/usr/share/vdsm/vm.py"=2C line 696=2C in _startUnderlyingVm<br= line 1907=2C in<br>>=3B >=3B>=3B>=3B>=3B _waitForIncomingMigratio= nFinish<br>>=3B >=3B>=3B>=3B>=3B File "/usr/lib64/python2.7/si= te-packages/vdsm/libvirtconnection.py"=2C<br>>=3B >=3B>=3B>=3B>= =3B line 111=2C in wrapper<br>>=3B >=3B>=3B>=3B>=3B File "/usr= /lib64/python2.7/site-packages/libvirt.py"=2C line 2822=2C in<br>>=3B >= =3B>=3B>=3B>=3B lookupByUUIDString<br>>=3B >=3B>=3B>=3B>=3B= libvirtError: Domain not found: no domain with matching uuid<br>>=3B >= =3B>=3B>=3B>=3B '50171e1b-cf21-41d8-80f3-88ab1b980091'<br>>=3B >= =3B>=3B>=3B>=3B But that is the ID of the VM by the looks of it.<br>&= gt=3B >=3B>=3B>=3B>=3B Sorry Itamar=2C nothing was written to libvi= rtd.log after I cleared it.<br>>=3B >=3B <br>>=3B >=3B It could be = that libvirtd is still writing to the files that you removed<br>>=3B >= =3B from the filesystem. To make sure libvirtd writes to your new file=2C<b= r>>=3B >=3B restart the service. There may be clues there on why libvir= t thinks that<br>>=3B >=3B the source and destination are one and the s= ame.<br>>=3B >=3B <br>>=3B <br>>=3B When clearing the logs=2C it sh= ould be enough to do '>=3B<br>>=3B /path/to/libvirtd.log' (in bash).<br=
>=3B </div><div>Just checked and it seems some things were logged in the= re during my testing on Friday. I'll attach those.<br>>=3B >=3B>=3B&g= t=3B<br>>=3B >=3B>=3B>=3B Thread-800::ERROR::2013-07-26 01:57:16=2C= 198::vm::198::vm.Vm::(_recover) <br>>=3B >=3B>=3B>=3B vmId=3D`50171= e1b-cf21-41d8-80f3-88ab1b980091`::internal error Attempt to <br>>=3B >= =3B>=3B>=3B migrate guest to the same host localhost<br>>=3B >=3B&g= t=3B>=3B Thread-800::ERROR::2013-07-26 01:57:16=2C377::vm::286::vm.Vm::(r= un) <br>>=3B >=3B>=3B>=3B vmId=3D`50171e1b-cf21-41d8-80f3-88ab1b980= 091`::Failed to migrate<br>>=3B >=3B>=3B>=3B Traceback (most recent= call last):<br>>=3B >=3B>=3B>=3B File "/usr/share/vdsm/vm.py"= =2C line 271=2C in run<br>>=3B >=3B>=3B>=3B File "/usr/share/vds= m/libvirtvm.py"=2C line 505=2C in <br>>=3B >=3B>=3B>=3B _startUnder= lyingMigration<br>>=3B >=3B>=3B>=3B File "/usr/share/vdsm/libvir= tvm.py"=2C line 541=2C in f<br>>=3B >=3B>=3B>=3B File "/usr/lib6= 4/python2.7/site-packages/vdsm/libvirtconnection.py"=2C <br>>=3B >=3B&g= t=3B>=3B line 111=2C in wrapper<br>>=3B >=3B>=3B>=3B File "/us= r/lib64/python2.7/site-packages/libvirt.py"=2C line 1178=2C in <br>>=3B &= gt=3B>=3B>=3B migrateToURI2<br>>=3B >=3B>=3B>=3B libvirtError: = internal error Attempt to migrate guest to the same host <br>>=3B >=3B&= gt=3B>=3B localhost<br>>=3B >=3B>=3B>=3B<br>>=3B >=3B>=3B&g= t=3B what are your hostnames?<br>>=3B >=3B>=3B<br>>=3B >=3B>=3B= "host001" on 192.168.0.103 and "host002" on 192.168.0.104<br>>=3B >=3B= >=3B Even tried changing it=2C no luck.<br>>=3B >=3B>=3B<br>>=3B = <br>>=3B Are they resolving properly on those hosts? Is there a DNS or<b= r>>=3B /etc/hosts entry related to this?<br>>=3B </div><div>There are /= etc/hosts entries on both hosts to each other=2C and a "ping host001" "ping= host002" resolves correctly.</div><div>I do however note that the terminal= session says <a href=3D"mailto:root@localhost">root@localhost</a>. I wonde= r if running hostnamectl set-hostname {name} will fix anything.</div><div>&= nbsp=3B</div><div>...and after running hostnamectl set-hostname {name}=2C m= igrations are working! I think maybe I found a bug:</div><div>With the node= in maintenance mode=2C =3Bfrom the oVirt Node (SSH or local) if you go= to Network=2C change the hostname ({newname})=2C and then go down to the c= onfigured System NIC and press enter=2C it says it is Setting Hostname. Now= =2C if you press F2=2C the terminal will show <a href=3D"mailto:root@{newna= me">root@{newname</a>}. If you reboot however=2C under network it will say = {newname}=2C but pressing F2 for the terminal will show <a href=3D"mailto:r= oot@localhost. If">root@localhost. If</a> it's localhost=2C it won't migrat= e.</div><div>So=2C it looks like the hostname isn't getting written persist= antly. Even a hostnamectl set-hostname {name} gets lost on reboot. Or am I = doing something wrong?</div><div><br>>=3B >=3B>=3B Could it be becaus= e the oVirt Node - Network tab - does not have any DNS servers specified?<b= r>>=3B >=3B <br>>=3B >=3B I do not think so. We do not see "name re= solution" errors=2C or name<br>>=3B >=3B resolutions at all.<br>>=3B = >=3B <br>>=3B >=3B What does libvirt (src + dst) say about<br>>=3B = >=3B <br>>=3B >=3B virsh -r capabilities|grep uuid<br>>=3B >= =3B <br>>=3B >=3B ? If uuids happen to be the same=2C you get the bug t= hat you are<br>>=3B >=3B reporting.<br>>=3B >=3B </div><div>host 00= 1:</div><div>virsh -r capabilities|grep uuid<br> =3B =3B =3B &l= t=3Buuid>=3Ba4dc7de7-e2d3-45f5-b75a-7101f71d2b17<=3B/uuid>=3B</div><d= iv> =3B</div><div>host002:<br>virsh -r capabilities|grep uuid<br> = =3B =3B =3B <=3Buuid>=3Bce66bb7f-fbbb-432b-9f62-5bcf5cb732e4<= =3B/uuid>=3B</div><div><br>>=3B <br>>=3B Although=2C as Dan says=2C h= aving the same UUID for both hosts will report<br>>=3B the same error eve= n when hostnames are different.<br>>=3B <br>>=3B Do you have an UUID se= t in your libvirtd.conf? What do you have in the<br>>=3B following files= (if they exist) on both hosts?<br>>=3B <br>>=3B /sys/devices/virtual/d= mi/id/product_uuid</div><div>host001:</div><div>cat /sys/devices/virtual/dm= i/id/product_uuid<br>44454C4C-3700-1047-8048-C3C04F4C4631</div><div> = =3B</div><div>host002:</div><div>cat /sys/devices/virtual/dmi/id/product_uu= id<br>44454C4C-5900-1054-8034-B3C04F4E4631</div><div> =3B</div><div>Tho= se match what I see on the webadmin under hosts >=3B hostname >=3B hard= ware information<br></div><div><br>>=3B /sys/class/dmi/id/product_uuid</d= iv><div>host001:</div><div>cat /sys/class/dmi/id/product_uuid<br>44454C4C-3= 700-1047-8048-C3C04F4C4631</div><div> =3B</div><div>host002:<br>cat /sy= s/class/dmi/id/product_uuid<br>44454C4C-5900-1054-8034-B3C04F4E4631<br><br>= >=3B <br>>=3B Martin<br></div> </div></body> </html>=
--_9f58fcb7-bfb3-4566-8fab-e81bc92863f7_--