From sokratis123k at outlook.com Mon Aug 3 12:39:39 2015 Content-Type: multipart/mixed; boundary="===============0659915951853895114==" MIME-Version: 1.0 From: s k To: users at ovirt.org Subject: [ovirt-users] VM Migration Fails Date: Mon, 03 Aug 2015 19:34:32 +0300 Message-ID: --===============0659915951853895114== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_b663e9f5-5d02-47b9-8bb0-d2f1409a3045_ Content-Type: text/plain; charset=3D"iso-8859-7" Content-Transfer-Encoding: quoted-printable Hi=3D2C I'm having trouble migrating VMs in a 2-node cluster. VMs can start normall= =3D y on both nodes if they are shutdown first but live migration fails and the= =3D following is thrown in the engine.log: 2015-08-03 19:28:28=3D2C566 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdat= eR=3D unTimeInfo] (DefaultQuartzScheduler_Worker-42) [46f46347] Rerun vm 2ceb9c65= =3D -1920-49fe-9db1-6c9470e50a65. Called from vds ovirt-srv-022015-08-03 19:28:= =3D 28=3D2C615 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVD= SC=3D ommand] (org.ovirt.thread.pool-8-thread-38) [46f46347] Failed in MigrateSta= =3D tusVDS method2015-08-03 19:28:28=3D2C617 ERROR [org.ovirt.engine.core.vdsbr= ok=3D er.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-8-thread-38) [= =3D 46f46347] Command MigrateStatusVDSCommand(HostName =3D3D ovirt-srv-02=3D2C = Host=3D Id =3D3D be3da0c4-f898-4aa4-89c7-239282a03959=3D2C vmId=3D3D2ceb9c65-1920-4= 9fe-9d=3D b1-6c9470e50a65) execution failed. Exception: VDSErrorException: VDSGeneric= =3D Exception: VDSErrorException: Failed to MigrateStatusVDS=3D2C error =3D3D F= atal=3D error during migration=3D2C code =3D3D 122015-08-03 19:28:28=3D2C623 ERROR= [org.=3D ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovir= =3D t.thread.pool-8-thread-38) [46f46347] Correlation ID: 43b38529=3D2C Job ID:= 2=3D 1468e21-c78c-41ea-bc99-c56ff4526820=3D2C Call Stack: null=3D2C Custom Event= ID:=3D -1=3D2C Message: Migration failed due to Error: Fatal error during migrati= on=3D . Trying to migrate to another Host (VM: testvm01=3D2C Source: ovirt-srv-02= =3D =3D2C Destination: ovirt-srv-03). Both nodes run on CentOS 6.6 with the following versions: OS Version: RHEL - 6 - 6.el6.centos.12.2Kernel Version: 2.6.32 - 504.30.3.e= =3D l6.x86_64KVM Version: 0.12.1.2 - 2.448.el6_6.4LIBVIRT Version: libvirt-0.10= =3D .2-46.el6_6.6VDSM Version: vdsm-4.16.20-1.git3a90f62.el6 Any thoughts? Thanks=3D2C Sokratis =3D --_b663e9f5-5d02-47b9-8bb0-d2f1409a3045_ Content-Type: text/html; charset=3D"iso-8859-7" Content-Transfer-Encoding: quoted-printable
Hi=3D2C

<= /div><=3D div>
I'm having trouble migrating VMs in a 2-node cluster. VM= =3D s can start normally on both nodes if they are shutdown first but live migr= =3D ation fails and the following is thrown in the engine.log:



2015-08-03 19:28:28=3D2C566 ERROR [o= rg=3D .ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_= =3D Worker-42) [46f46347] Rerun vm 2ceb9c65-1920-49fe-9db1-6c9470e50a65. Called= =3D from vds ovirt-srv-02
2015-08-03 19:28:28=3D2C615 ERROR [org.ovi= rt=3D .engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread= =3D .pool-8-thread-38) [46f46347] Failed in MigrateStatusVDS method
2= =3D 015-08-03 19:28:28=3D2C617 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker= .M=3D igrateStatusVDSCommand] (org.ovirt.thread.pool-8-thread-38) [46f46347] Comm= =3D and MigrateStatusVDSCommand(HostName =3D3D ovirt-srv-02=3D2C HostId =3D3D b= e3da0c=3D 4-f898-4aa4-89c7-239282a03959=3D2C vmId=3D3D2ceb9c65-1920-49fe-9db1-6c9470e= 50a6=3D 5) execution failed. Exception: VDSErrorException: VDSGenericException: VDS= =3D ErrorException: Failed to MigrateStatusVDS=3D2C error =3D3D Fatal error dur= ing =3D migration=3D2C code =3D3D 12
2015-08-03 19:28:28=3D2C623 ERROR [o= rg.ovi=3D rt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.t= =3D hread.pool-8-thread-38) [46f46347] Correlation ID: 43b38529=3D2C Job ID: 21= 46=3D 8e21-c78c-41ea-bc99-c56ff4526820=3D2C Call Stack: null=3D2C Custom Event ID= : -1=3D =3D2C Message: Migration failed due to Error: Fatal error during migration.= T=3D rying to migrate to another Host (VM: testvm01=3D2C Source: ovirt-srv-02=3D= 2C D=3D estination: ovirt-srv-03).


Both nod= =3D es run on CentOS 6.6 with the following versions:

= =3D
OS Version: RHEL - 6 - 6.el6.centos.12.2
Kernel Version: 2.6.32 - 504.3= 0.3.=3D el6.x86_64
KVM = Vers=3D ion: 0.12.1.2 - 2.448.el6_6.4
LIBVIRT Version: libvirt-0.10.2-46.el6_6.6
VDSM Version: vdsm-4.16.20-= 1.gi=3D t3a90f62.el6


Any t= =3D houghts?


Thanks=3D2C

=

Sokratis
=3D --_b663e9f5-5d02-47b9-8bb0-d2f1409a3045_-- --===============0659915951853895114== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fYjY2M2U5ZjUtNWQwMi00N2I5LThiYjAtZDJmMTQwOWEzMDQ1XwpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW47IGNoYXJzZXQ9Imlzby04ODU5LTciCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6 IHF1b3RlZC1wcmludGFibGUKCkhpPTJDCgpJJ20gaGF2aW5nIHRyb3VibGUgbWlncmF0aW5nIFZN cyBpbiBhIDItbm9kZSBjbHVzdGVyLiBWTXMgY2FuIHN0YXJ0IG5vcm1hbGw9Cnkgb24gYm90aCBu b2RlcyBpZiB0aGV5IGFyZSBzaHV0ZG93biBmaXJzdCBidXQgbGl2ZSBtaWdyYXRpb24gZmFpbHMg YW5kIHRoZT0KIGZvbGxvd2luZyBpcyB0aHJvd24gaW4gdGhlIGVuZ2luZS5sb2c6CgoKMjAxNS0w OC0wMyAxOToyODoyOD0yQzU2NiBFUlJPUiBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLnZkc2Jyb2tl ci5WZHNVcGRhdGVSPQp1blRpbWVJbmZvXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXIt NDIpIFs0NmY0NjM0N10gUmVydW4gdm0gMmNlYjljNjU9Ci0xOTIwLTQ5ZmUtOWRiMS02Yzk0NzBl NTBhNjUuIENhbGxlZCBmcm9tIHZkcyBvdmlydC1zcnYtMDIyMDE1LTA4LTAzIDE5OjI4Oj0KMjg9 MkM2MTUgRVJST1IgW29yZy5vdmlydC5lbmdpbmUuY29yZS52ZHNicm9rZXIudmRzYnJva2VyLk1p Z3JhdGVTdGF0dXNWRFNDPQpvbW1hbmRdIChvcmcub3ZpcnQudGhyZWFkLnBvb2wtOC10aHJlYWQt MzgpIFs0NmY0NjM0N10gRmFpbGVkIGluIE1pZ3JhdGVTdGE9CnR1c1ZEUyBtZXRob2QyMDE1LTA4 LTAzIDE5OjI4OjI4PTJDNjE3IEVSUk9SIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUudmRzYnJvaz0K ZXIudmRzYnJva2VyLk1pZ3JhdGVTdGF0dXNWRFNDb21tYW5kXSAob3JnLm92aXJ0LnRocmVhZC5w b29sLTgtdGhyZWFkLTM4KSBbPQo0NmY0NjM0N10gQ29tbWFuZCBNaWdyYXRlU3RhdHVzVkRTQ29t bWFuZChIb3N0TmFtZSA9M0Qgb3ZpcnQtc3J2LTAyPTJDIEhvc3Q9CklkID0zRCBiZTNkYTBjNC1m ODk4LTRhYTQtODljNy0yMzkyODJhMDM5NTk9MkMgdm1JZD0zRDJjZWI5YzY1LTE5MjAtNDlmZS05 ZD0KYjEtNmM5NDcwZTUwYTY1KSBleGVjdXRpb24gZmFpbGVkLiBFeGNlcHRpb246IFZEU0Vycm9y RXhjZXB0aW9uOiBWRFNHZW5lcmljPQpFeGNlcHRpb246IFZEU0Vycm9yRXhjZXB0aW9uOiBGYWls ZWQgdG8gTWlncmF0ZVN0YXR1c1ZEUz0yQyBlcnJvciA9M0QgRmF0YWw9CiBlcnJvciBkdXJpbmcg bWlncmF0aW9uPTJDIGNvZGUgPTNEIDEyMjAxNS0wOC0wMyAxOToyODoyOD0yQzYyMyBFUlJPUiBb b3JnLj0Kb3ZpcnQuZW5naW5lLmNvcmUuZGFsLmRiYnJva2VyLmF1ZGl0bG9naGFuZGxpbmcuQXVk aXRMb2dEaXJlY3Rvcl0gKG9yZy5vdmlyPQp0LnRocmVhZC5wb29sLTgtdGhyZWFkLTM4KSBbNDZm NDYzNDddIENvcnJlbGF0aW9uIElEOiA0M2IzODUyOT0yQyBKb2IgSUQ6IDI9CjE0NjhlMjEtYzc4 Yy00MWVhLWJjOTktYzU2ZmY0NTI2ODIwPTJDIENhbGwgU3RhY2s6IG51bGw9MkMgQ3VzdG9tIEV2 ZW50IElEOj0KIC0xPTJDIE1lc3NhZ2U6IE1pZ3JhdGlvbiBmYWlsZWQgZHVlIHRvIEVycm9yOiBG YXRhbCBlcnJvciBkdXJpbmcgbWlncmF0aW9uPQouIFRyeWluZyB0byBtaWdyYXRlIHRvIGFub3Ro ZXIgSG9zdCAoVk06IHRlc3R2bTAxPTJDIFNvdXJjZTogb3ZpcnQtc3J2LTAyPQo9MkMgRGVzdGlu YXRpb246IG92aXJ0LXNydi0wMykuCgpCb3RoIG5vZGVzIHJ1biBvbiBDZW50T1MgNi42IHdpdGgg dGhlIGZvbGxvd2luZyB2ZXJzaW9uczoKCk9TIFZlcnNpb246IFJIRUwgLSA2IC0gNi5lbDYuY2Vu dG9zLjEyLjJLZXJuZWwgVmVyc2lvbjogMi42LjMyIC0gNTA0LjMwLjMuZT0KbDYueDg2XzY0S1ZN IFZlcnNpb246IDAuMTIuMS4yIC0gMi40NDguZWw2XzYuNExJQlZJUlQgVmVyc2lvbjogbGlidmly dC0wLjEwPQouMi00Ni5lbDZfNi42VkRTTSBWZXJzaW9uOiB2ZHNtLTQuMTYuMjAtMS5naXQzYTkw ZjYyLmVsNgoKQW55IHRob3VnaHRzPwoKVGhhbmtzPTJDCgpTb2tyYXRpcyAJCSAJICAgCQkgID0K Ci0tX2I2NjNlOWY1LTVkMDItNDdiOS04YmIwLWQyZjE0MDlhMzA0NV8KQ29udGVudC1UeXBlOiB0 ZXh0L2h0bWw7IGNoYXJzZXQ9Imlzby04ODU5LTciCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6 IHF1b3RlZC1wcmludGFibGUKCjxodG1sPgo8aGVhZD4KPHN0eWxlPjwhLS0KLmhtbWVzc2FnZSBQ CnsKbWFyZ2luOjBweD0zQgpwYWRkaW5nOjBweAp9CmJvZHkuaG1tZXNzYWdlCnsKZm9udC1zaXpl OiAxMnB0PTNCCmZvbnQtZmFtaWx5OkNhbGlicmkKfQotLT48L3N0eWxlPjwvaGVhZD4KPGJvZHkg Y2xhc3M9M0QnaG1tZXNzYWdlJz48ZGl2IGRpcj0zRCdsdHInPjxkaXY+SGk9MkM8L2Rpdj48ZGl2 Pjxicj48L2Rpdj48PQpkaXY+PGJyPjwvZGl2PjxkaXY+SSdtIGhhdmluZyB0cm91YmxlIG1pZ3Jh dGluZyBWTXMgaW4gYSAyLW5vZGUgY2x1c3Rlci4gVk09CnMgY2FuIHN0YXJ0IG5vcm1hbGx5IG9u IGJvdGggbm9kZXMgaWYgdGhleSBhcmUgc2h1dGRvd24gZmlyc3QgYnV0IGxpdmUgbWlncj0KYXRp b24gZmFpbHMgYW5kIHRoZSBmb2xsb3dpbmcgaXMgdGhyb3duIGluIHRoZSBlbmdpbmUubG9nOjwv ZGl2PjxkaXY+PGJyPjwvPQpkaXY+PGRpdj48YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj4y MDE1LTA4LTAzIDE5OjI4OjI4PTJDNTY2IEVSUk9SIFtvcmc9Ci5vdmlydC5lbmdpbmUuY29yZS52 ZHNicm9rZXIuVmRzVXBkYXRlUnVuVGltZUluZm9dIChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyXz0K V29ya2VyLTQyKSBbNDZmNDYzNDddIFJlcnVuIHZtIDJjZWI5YzY1LTE5MjAtNDlmZS05ZGIxLTZj OTQ3MGU1MGE2NS4gQ2FsbGVkPQogZnJvbSB2ZHMgb3ZpcnQtc3J2LTAyPC9kaXY+PGRpdj4yMDE1 LTA4LTAzIDE5OjI4OjI4PTJDNjE1IEVSUk9SIFtvcmcub3ZpcnQ9Ci5lbmdpbmUuY29yZS52ZHNi cm9rZXIudmRzYnJva2VyLk1pZ3JhdGVTdGF0dXNWRFNDb21tYW5kXSAob3JnLm92aXJ0LnRocmVh ZD0KLnBvb2wtOC10aHJlYWQtMzgpIFs0NmY0NjM0N10gRmFpbGVkIGluIE1pZ3JhdGVTdGF0dXNW RFMgbWV0aG9kPC9kaXY+PGRpdj4yPQowMTUtMDgtMDMgMTk6Mjg6Mjg9MkM2MTcgRVJST1IgW29y Zy5vdmlydC5lbmdpbmUuY29yZS52ZHNicm9rZXIudmRzYnJva2VyLk09CmlncmF0ZVN0YXR1c1ZE U0NvbW1hbmRdIChvcmcub3ZpcnQudGhyZWFkLnBvb2wtOC10aHJlYWQtMzgpIFs0NmY0NjM0N10g Q29tbT0KYW5kIE1pZ3JhdGVTdGF0dXNWRFNDb21tYW5kKEhvc3ROYW1lID0zRCBvdmlydC1zcnYt MDI9MkMgSG9zdElkID0zRCBiZTNkYTBjPQo0LWY4OTgtNGFhNC04OWM3LTIzOTI4MmEwMzk1OT0y QyB2bUlkPTNEMmNlYjljNjUtMTkyMC00OWZlLTlkYjEtNmM5NDcwZTUwYTY9CjUpIGV4ZWN1dGlv biBmYWlsZWQuIEV4Y2VwdGlvbjogVkRTRXJyb3JFeGNlcHRpb246IFZEU0dlbmVyaWNFeGNlcHRp b246IFZEUz0KRXJyb3JFeGNlcHRpb246IEZhaWxlZCB0byBNaWdyYXRlU3RhdHVzVkRTPTJDIGVy cm9yID0zRCBGYXRhbCBlcnJvciBkdXJpbmcgPQptaWdyYXRpb249MkMgY29kZSA9M0QgMTI8L2Rp dj48ZGl2PjIwMTUtMDgtMDMgMTk6Mjg6Mjg9MkM2MjMgRVJST1IgW29yZy5vdmk9CnJ0LmVuZ2lu ZS5jb3JlLmRhbC5kYmJyb2tlci5hdWRpdGxvZ2hhbmRsaW5nLkF1ZGl0TG9nRGlyZWN0b3JdIChv cmcub3ZpcnQudD0KaHJlYWQucG9vbC04LXRocmVhZC0zOCkgWzQ2ZjQ2MzQ3XSBDb3JyZWxhdGlv biBJRDogNDNiMzg1Mjk9MkMgSm9iIElEOiAyMTQ2PQo4ZTIxLWM3OGMtNDFlYS1iYzk5LWM1NmZm NDUyNjgyMD0yQyBDYWxsIFN0YWNrOiBudWxsPTJDIEN1c3RvbSBFdmVudCBJRDogLTE9Cj0yQyBN ZXNzYWdlOiBNaWdyYXRpb24gZmFpbGVkIGR1ZSB0byBFcnJvcjogRmF0YWwgZXJyb3IgZHVyaW5n IG1pZ3JhdGlvbi4gVD0KcnlpbmcgdG8gbWlncmF0ZSB0byBhbm90aGVyIEhvc3QgKFZNOiB0ZXN0 dm0wMT0yQyBTb3VyY2U6IG92aXJ0LXNydi0wMj0yQyBEPQplc3RpbmF0aW9uOiBvdmlydC1zcnYt MDMpLjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+Qm90aCBub2Q9CmVz IHJ1biBvbiBDZW50T1MgNi42IHdpdGggdGhlIGZvbGxvd2luZyB2ZXJzaW9uczo8L2Rpdj48ZGl2 Pjxicj48L2Rpdj48ZGl2Pj0KPGJyPjwvZGl2PjxkaXY+PGRpdj5PUyBWZXJzaW9uOiA8Yj5SSEVM IC0gNiAtIDYuZWw2LmNlbnRvcy4xMi4yPC9iPjwvZGl2PjxkPQppdj48c3BhbiBzdHlsZT0zRCJm b250LXNpemU6IDEycHQ9M0IiPktlcm5lbCBWZXJzaW9uOiA8Yj4yLjYuMzIgLSA1MDQuMzAuMy49 CmVsNi54ODZfNjQ8L2I+PC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4gc3R5bGU9M0QiZm9udC1zaXpl OiAxMnB0PTNCIj5LVk0gVmVycz0KaW9uOiA8Yj4wLjEyLjEuMiAtIDIuNDQ4LmVsNl82LjQ8L2I+ PC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4gc3R5bGU9M0QiZm9udC1zPQppemU6IDEycHQ9M0IiPkxJ QlZJUlQgVmVyc2lvbjogPGI+bGlidmlydC0wLjEwLjItNDYuZWw2XzYuNjwvYj48L3NwYW4+PC9k aXY9Cj48ZGl2PjxzcGFuIHN0eWxlPTNEImZvbnQtc2l6ZTogMTJwdD0zQiI+VkRTTSBWZXJzaW9u OiA8Yj52ZHNtLTQuMTYuMjAtMS5naT0KdDNhOTBmNjIuZWw2PC9iPjwvc3Bhbj48L2Rpdj48L2Rp dj48ZGl2Pjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkFueSB0PQpob3VnaHRzPzwvZGl2 PjxkaXY+PGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+VGhhbmtzPTJDPC9kaXY+PGRpdj48 YnI+PC89CmRpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PlNva3JhdGlzPC9kaXY+IAkJIAkgICAJCSAg PC9kaXY+PC9ib2R5Pgo8L2h0bWw+PQoKLS1fYjY2M2U5ZjUtNWQwMi00N2I5LThiYjAtZDJmMTQw OWEzMDQ1Xy0tCg== --===============0659915951853895114==--