From michal.skrivanek at redhat.com Thu Apr 12 18:20:35 2018 Content-Type: multipart/mixed; boundary="===============0905976214948883206==" MIME-Version: 1.0 From: Michal Skrivanek To: users at ovirt.org Subject: Re: [ovirt-users] Frequent vm migration failure Date: Thu, 12 Apr 2018 20:20:29 +0200 Message-ID: <4FA82C16-954E-4AFC-AA4F-71E9D1C951D9@redhat.com> In-Reply-To: 1523550377.6503.5.camel@prismatelecomtesting.com --===============0905976214948883206== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_437561B1-E46A-4D8E-B586-1CF5DBD526CC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dutf-8 > On 12 Apr 2018, at 18:26, Stefano Stagnaro =3D wrote: >=3D20 > Hi, >=3D20 > I recently upgraded an oVirt deployment from 3.6 to 4.0 and then 4.1.9 = =3D (my actual release). Since then, when migrating many hosts =3D simultaneously I always experience few migrations failure like 1 on 10 =3D vms. The failure can occur on any host; moreover, after a couple of =3D failure the destination host fall in Error status and I have to manually = =3D re-activate or wait 30 min. >=3D20 > Tipical error found on vdsm log is (from the source host): > 2018-04-12 17:01:32,097+0200 ERROR (migsrc/3192dfe7) [virt.vm] =3D (vmId=3D3D'3192dfe7-eeac-4626-8c86-e49facc9006f') migration destination =3D error: Fatal error during migration (migration:287) >=3D20 > Please find the logs of source host (v15.ovirt), destination host =3D (v14.ovirt) and engine here: =3D https://www.dropbox.com/sh/xhf8ry4ih40poxd/AABxiFCIxDe14HSx2DqLE61ya?dl=3D3= D=3D 0 >=3D20 > Some of the vm affected from the migration failure are: > svn 3192dfe7-eeac-4626-8c86-e49facc9006f > wood a8e83ff0-dfed-4074-b6b6-e947b8ebb952 > qnx66 5697c4a4-9e40-4dd6-aba2-c8ab9904a584 can you also include qemu log from /var/log/libvirt/qemu/? btw you seem to be using the legacy migration policy throttling the =3D speed significantly. Please read into the migration enhancements in 4.0 =3D https://www.ovirt.org/develop/release-management/features/virt/migration-e= =3D nhancements/ =3D Thanks, michal >=3D20 > Thank you very much for your help. >=3D20 > --=3D20 > Stefano Stagnaro >=3D20 > Prisma Telecom Testing S.r.l. > Via Petrocchi, 4 > 20127 Milano =3DE2=3D80=3D93 Italy >=3D20 > Tel. 02 26113507 int 339 > e-mail: stefanos(a)prismatelecomtesting.com > skype: stefano.stagnaro > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users --Apple-Mail=3D_437561B1-E46A-4D8E-B586-1CF5DBD526CC Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dutf-8

On 12 Apr 2018, at 18:26, Stefano Stagnaro <stefanos(a)prismatelecomtesting.com> wrote:

H= i,
I recently upgraded an oVirt deployment from= =3D 3.6 to 4.0 and then 4.1.9 (my actual release). Since then, when =3D migrating many hosts simultaneously I always experience few migrations =3D failure like 1 on 10 vms. The failure can occur on any host; moreover, =3D after a couple of failure the destination host fall in Error status and =3D I have to manually re-activate or wait 30 min.

Tipical error found on vdsm log is (from the source host):
2018-04-12 17:01:32,097+0200 ERROR (migsrc/3192dfe7) =3D [virt.vm] (vmId=3D3D'3192dfe7-eeac-4626-8c86-e49facc9006f') migration =3D destination error: Fatal error during migration (migration:287)

Please find the logs of source host =3D (v15.ovirt), destination host (v14.ovirt) and engine here: https://www.dropbox.com/sh/xhf8ry4ih40poxd/AABxiFCIxDe14HSx2Dq= L=3D E61ya?dl=3D3D0

Some of the vm affected= =3D from the migration failure are:
svn =3D 3192dfe7-eeac-4626-8c86-e49facc9006f
wood =3D a8e83ff0-dfed-4074-b6b6-e947b8ebb952
qnx66 =3D 5697c4a4-9e40-4dd6-aba2-c8ab9904a584

can you = =3D also include qemu log from =3D /var/log/libvirt/qemu/<vmname>?

btw you seem to be using the legacy migration =3D policy throttling the speed significantly. Please read into the =3D migration enhancements in 4.0
https://www.ovirt.org/develop/release-management/features/virt= /=3D migration-enhancements/

Thanks,
michal


Thank you very much for your help.

--
Stefano Stagnaro

Prisma Telecom Testing S.r.l.
Via =3D Petrocchi, 4
20127 Milano =3DE2=3D80=3D93 Italy

Tel. 02 26113507 int 339
e-mail: = =3D stefanos(a)prismatelecomtesting.com
skype: =3D stefano.stagnaro
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

= =3D --Apple-Mail=3D_437561B1-E46A-4D8E-B586-1CF5DBD526CC-- --===============0905976214948883206== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fNDM3NTYxQjEtRTQ2QS00RDhFLUI1ODYtMUNGNURCRDUyNkNDCkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD11dGYtOAoKCgo+IE9uIDEyIEFwciAyMDE4LCBhdCAxODoyNiwgU3Rl ZmFubyBTdGFnbmFybyA9CjxzdGVmYW5vc0BwcmlzbWF0ZWxlY29tdGVzdGluZy5jb20+IHdyb3Rl Ogo+PTIwCj4gSGksCj49MjAKPiBJIHJlY2VudGx5IHVwZ3JhZGVkIGFuIG9WaXJ0IGRlcGxveW1l bnQgZnJvbSAzLjYgdG8gNC4wIGFuZCB0aGVuIDQuMS45ID0KKG15IGFjdHVhbCByZWxlYXNlKS4g U2luY2UgdGhlbiwgd2hlbiBtaWdyYXRpbmcgbWFueSBob3N0cyA9CnNpbXVsdGFuZW91c2x5IEkg YWx3YXlzIGV4cGVyaWVuY2UgZmV3IG1pZ3JhdGlvbnMgZmFpbHVyZSBsaWtlIDEgb24gMTAgPQp2 bXMuIFRoZSBmYWlsdXJlIGNhbiBvY2N1ciBvbiBhbnkgaG9zdDsgbW9yZW92ZXIsIGFmdGVyIGEg Y291cGxlIG9mID0KZmFpbHVyZSB0aGUgZGVzdGluYXRpb24gaG9zdCBmYWxsIGluIEVycm9yIHN0 YXR1cyBhbmQgSSBoYXZlIHRvIG1hbnVhbGx5ID0KcmUtYWN0aXZhdGUgb3Igd2FpdCAzMCBtaW4u Cj49MjAKPiBUaXBpY2FsIGVycm9yIGZvdW5kIG9uIHZkc20gbG9nIGlzIChmcm9tIHRoZSBzb3Vy Y2UgaG9zdCk6Cj4gMjAxOC0wNC0xMiAxNzowMTozMiwwOTcrMDIwMCBFUlJPUiAobWlnc3JjLzMx OTJkZmU3KSBbdmlydC52bV0gPQoodm1JZD0zRCczMTkyZGZlNy1lZWFjLTQ2MjYtOGM4Ni1lNDlm YWNjOTAwNmYnKSBtaWdyYXRpb24gZGVzdGluYXRpb24gPQplcnJvcjogRmF0YWwgZXJyb3IgZHVy aW5nIG1pZ3JhdGlvbiAobWlncmF0aW9uOjI4NykKPj0yMAo+IFBsZWFzZSBmaW5kIHRoZSBsb2dz IG9mIHNvdXJjZSBob3N0ICh2MTUub3ZpcnQpLCBkZXN0aW5hdGlvbiBob3N0ID0KKHYxNC5vdmly dCkgYW5kIGVuZ2luZSBoZXJlOiA9Cmh0dHBzOi8vd3d3LmRyb3Bib3guY29tL3NoL3hoZjhyeTRp aDQwcG94ZC9BQUJ4aUZDSXhEZTE0SFN4MkRxTEU2MXlhP2RsPTNEPQowCj49MjAKPiBTb21lIG9m IHRoZSB2bSBhZmZlY3RlZCBmcm9tIHRoZSBtaWdyYXRpb24gZmFpbHVyZSBhcmU6Cj4gc3ZuCTMx OTJkZmU3LWVlYWMtNDYyNi04Yzg2LWU0OWZhY2M5MDA2Zgo+IHdvb2QJYThlODNmZjAtZGZlZC00 MDc0LWI2YjYtZTk0N2I4ZWJiOTUyCj4gcW54NjYJNTY5N2M0YTQtOWU0MC00ZGQ2LWFiYTItYzhh Yjk5MDRhNTg0CgpjYW4geW91IGFsc28gaW5jbHVkZSBxZW11IGxvZyBmcm9tIC92YXIvbG9nL2xp YnZpcnQvcWVtdS88dm1uYW1lPj8KCmJ0dyB5b3Ugc2VlbSB0byBiZSB1c2luZyB0aGUgbGVnYWN5 IG1pZ3JhdGlvbiBwb2xpY3kgdGhyb3R0bGluZyB0aGUgPQpzcGVlZCBzaWduaWZpY2FudGx5LiBQ bGVhc2UgcmVhZCBpbnRvIHRoZSBtaWdyYXRpb24gZW5oYW5jZW1lbnRzIGluIDQuMAo9Cmh0dHBz Oi8vd3d3Lm92aXJ0Lm9yZy9kZXZlbG9wL3JlbGVhc2UtbWFuYWdlbWVudC9mZWF0dXJlcy92aXJ0 L21pZ3JhdGlvbi1lPQpuaGFuY2VtZW50cy8gPQo8aHR0cHM6Ly93d3cub3ZpcnQub3JnL2RldmVs b3AvcmVsZWFzZS1tYW5hZ2VtZW50L2ZlYXR1cmVzL3ZpcnQvbWlncmF0aW9uLT0KZW5oYW5jZW1l bnRzLz4KClRoYW5rcywKbWljaGFsCgo+PTIwCj4gVGhhbmsgeW91IHZlcnkgbXVjaCBmb3IgeW91 ciBoZWxwLgo+PTIwCj4gLS09MjAKPiBTdGVmYW5vIFN0YWduYXJvCj49MjAKPiBQcmlzbWEgVGVs ZWNvbSBUZXN0aW5nIFMuci5sLgo+IFZpYSBQZXRyb2NjaGksIDQKPiAyMDEyNyBNaWxhbm8gPUUy PTgwPTkzIEl0YWx5Cj49MjAKPiBUZWwuIDAyIDI2MTEzNTA3IGludCAzMzkKPiBlLW1haWw6IHN0 ZWZhbm9zQHByaXNtYXRlbGVjb210ZXN0aW5nLmNvbQo+IHNreXBlOiBzdGVmYW5vLnN0YWduYXJv Cj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KPiBVc2Vy cyBtYWlsaW5nIGxpc3QKPiBVc2Vyc0BvdmlydC5vcmcKPiBodHRwOi8vbGlzdHMub3ZpcnQub3Jn L21haWxtYW4vbGlzdGluZm8vdXNlcnMKCgotLUFwcGxlLU1haWw9XzQzNzU2MUIxLUU0NkEtNEQ4 RS1CNTg2LTFDRjVEQkQ1MjZDQwpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJp bnRhYmxlCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOwoJY2hhcnNldD11dGYtOAoKPGh0bWw+PGhl YWQ+PG1ldGEgaHR0cC1lcXVpdj0zRCJDb250ZW50LVR5cGUiIGNvbnRlbnQ9M0QidGV4dC9odG1s OyA9CmNoYXJzZXQ9M0R1dGYtOCI+PC9oZWFkPjxib2R5IHN0eWxlPTNEIndvcmQtd3JhcDogYnJl YWstd29yZDsgPQotd2Via2l0LW5ic3AtbW9kZTogc3BhY2U7IGxpbmUtYnJlYWs6IGFmdGVyLXdo aXRlLXNwYWNlOyIgY2xhc3M9M0QiIj48YnIgPQpjbGFzcz0zRCIiPjxkaXY+PGJyIGNsYXNzPTNE IiI+PGJsb2NrcXVvdGUgdHlwZT0zRCJjaXRlIiBjbGFzcz0zRCIiPjxkaXYgPQpjbGFzcz0zRCIi Pk9uIDEyIEFwciAyMDE4LCBhdCAxODoyNiwgU3RlZmFubyBTdGFnbmFybyAmbHQ7PGEgPQpocmVm PTNEIm1haWx0bzpzdGVmYW5vc0BwcmlzbWF0ZWxlY29tdGVzdGluZy5jb20iID0KY2xhc3M9M0Qi Ij5zdGVmYW5vc0BwcmlzbWF0ZWxlY29tdGVzdGluZy5jb208L2E+Jmd0OyB3cm90ZTo8L2Rpdj48 YnIgPQpjbGFzcz0zRCJBcHBsZS1pbnRlcmNoYW5nZS1uZXdsaW5lIj48ZGl2IGNsYXNzPTNEIiI+ PGRpdiBjbGFzcz0zRCIiPkhpLDxicj0KIGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+SSByZWNl bnRseSB1cGdyYWRlZCBhbiBvVmlydCBkZXBsb3ltZW50IGZyb20gPQozLjYgdG8gNC4wIGFuZCB0 aGVuIDQuMS45IChteSBhY3R1YWwgcmVsZWFzZSkuIFNpbmNlIHRoZW4sIHdoZW4gPQptaWdyYXRp bmcgbWFueSBob3N0cyBzaW11bHRhbmVvdXNseSBJIGFsd2F5cyBleHBlcmllbmNlIGZldyBtaWdy YXRpb25zID0KZmFpbHVyZSBsaWtlIDEgb24gMTAgdm1zLiBUaGUgZmFpbHVyZSBjYW4gb2NjdXIg b24gYW55IGhvc3Q7IG1vcmVvdmVyLCA9CmFmdGVyIGEgY291cGxlIG9mIGZhaWx1cmUgdGhlIGRl c3RpbmF0aW9uIGhvc3QgZmFsbCBpbiBFcnJvciBzdGF0dXMgYW5kID0KSSBoYXZlIHRvIG1hbnVh bGx5IHJlLWFjdGl2YXRlIG9yIHdhaXQgMzAgbWluLjxiciBjbGFzcz0zRCIiPjxiciA9CmNsYXNz PTNEIiI+VGlwaWNhbCBlcnJvciBmb3VuZCBvbiB2ZHNtIGxvZyBpcyAoZnJvbSB0aGUgc291cmNl IGhvc3QpOjxiciA9CmNsYXNzPTNEIiI+MjAxOC0wNC0xMiAxNzowMTozMiwwOTcrMDIwMCBFUlJP UiAobWlnc3JjLzMxOTJkZmU3KSA9Clt2aXJ0LnZtXSAodm1JZD0zRCczMTkyZGZlNy1lZWFjLTQ2 MjYtOGM4Ni1lNDlmYWNjOTAwNmYnKSBtaWdyYXRpb24gPQpkZXN0aW5hdGlvbiBlcnJvcjogRmF0 YWwgZXJyb3IgZHVyaW5nIG1pZ3JhdGlvbiAobWlncmF0aW9uOjI4Nyk8YnIgPQpjbGFzcz0zRCIi PjxiciBjbGFzcz0zRCIiPlBsZWFzZSBmaW5kIHRoZSBsb2dzIG9mIHNvdXJjZSBob3N0ID0KKHYx NS5vdmlydCksIGRlc3RpbmF0aW9uIGhvc3QgKHYxNC5vdmlydCkgYW5kIGVuZ2luZSBoZXJlOiA8 YSA9CmhyZWY9M0QiaHR0cHM6Ly93d3cuZHJvcGJveC5jb20vc2gveGhmOHJ5NGloNDBwb3hkL0FB QnhpRkNJeERlMTRIU3gyRHFMRTYxPQp5YT9kbD0zRDAiID0KY2xhc3M9M0QiIj5odHRwczovL3d3 dy5kcm9wYm94LmNvbS9zaC94aGY4cnk0aWg0MHBveGQvQUFCeGlGQ0l4RGUxNEhTeDJEcUw9CkU2 MXlhP2RsPTNEMDwvYT48YnIgY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj5Tb21lIG9mIHRoZSB2 bSBhZmZlY3RlZCA9CmZyb20gdGhlIG1pZ3JhdGlvbiBmYWlsdXJlIGFyZTo8YnIgY2xhc3M9M0Qi Ij5zdm48c3BhbiA9CmNsYXNzPTNEIkFwcGxlLXRhYi1zcGFuIiBzdHlsZT0zRCJ3aGl0ZS1zcGFj ZTpwcmUiPgk9Cjwvc3Bhbj4zMTkyZGZlNy1lZWFjLTQ2MjYtOGM4Ni1lNDlmYWNjOTAwNmY8YnIg Y2xhc3M9M0QiIj53b29kPHNwYW4gPQpjbGFzcz0zRCJBcHBsZS10YWItc3BhbiIgc3R5bGU9M0Qi d2hpdGUtc3BhY2U6cHJlIj4JPQo8L3NwYW4+YThlODNmZjAtZGZlZC00MDc0LWI2YjYtZTk0N2I4 ZWJiOTUyPGJyIGNsYXNzPTNEIiI+cW54NjY8c3BhbiA9CmNsYXNzPTNEIkFwcGxlLXRhYi1zcGFu IiBzdHlsZT0zRCJ3aGl0ZS1zcGFjZTpwcmUiPgk9Cjwvc3Bhbj41Njk3YzRhNC05ZTQwLTRkZDYt YWJhMi1jOGFiOTkwNGE1ODQ8YnIgPQpjbGFzcz0zRCIiPjwvZGl2PjwvZGl2PjwvYmxvY2txdW90 ZT48ZGl2PjxiciBjbGFzcz0zRCIiPjwvZGl2PmNhbiB5b3UgPQphbHNvIGluY2x1ZGUgcWVtdSBs b2cgZnJvbSA9Ci92YXIvbG9nL2xpYnZpcnQvcWVtdS8mbHQ7dm1uYW1lJmd0Oz88L2Rpdj48ZGl2 PjxiciA9CmNsYXNzPTNEIiI+PC9kaXY+PGRpdj5idHcgeW91IHNlZW0gdG8gYmUgdXNpbmcgdGhl IGxlZ2FjeSBtaWdyYXRpb24gPQpwb2xpY3kgdGhyb3R0bGluZyB0aGUgc3BlZWQgc2lnbmlmaWNh bnRseS4gUGxlYXNlIHJlYWQgaW50byB0aGUgPQptaWdyYXRpb24gZW5oYW5jZW1lbnRzIGluIDQu MDwvZGl2PjxkaXY+PGEgPQpocmVmPTNEImh0dHBzOi8vd3d3Lm92aXJ0Lm9yZy9kZXZlbG9wL3Jl bGVhc2UtbWFuYWdlbWVudC9mZWF0dXJlcy92aXJ0L21pZz0KcmF0aW9uLWVuaGFuY2VtZW50cy8i ID0KY2xhc3M9M0QiIj5odHRwczovL3d3dy5vdmlydC5vcmcvZGV2ZWxvcC9yZWxlYXNlLW1hbmFn ZW1lbnQvZmVhdHVyZXMvdmlydC89Cm1pZ3JhdGlvbi1lbmhhbmNlbWVudHMvPC9hPjwvZGl2Pjxk aXY+PGJyID0KY2xhc3M9M0QiIj48L2Rpdj48ZGl2PlRoYW5rcyw8L2Rpdj48ZGl2Pm1pY2hhbDwv ZGl2PjxkaXY+PGJyID0KY2xhc3M9M0QiIj48YmxvY2txdW90ZSB0eXBlPTNEImNpdGUiIGNsYXNz PTNEIiI+PGRpdiBjbGFzcz0zRCIiPjxkaXYgPQpjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPlRo YW5rIHlvdSB2ZXJ5IG11Y2ggZm9yIHlvdXIgaGVscC48YnIgPQpjbGFzcz0zRCIiPjxiciBjbGFz cz0zRCIiPi0tIDxiciBjbGFzcz0zRCIiPlN0ZWZhbm8gU3RhZ25hcm88YnIgPQpjbGFzcz0zRCIi PjxiciBjbGFzcz0zRCIiPlByaXNtYSBUZWxlY29tIFRlc3RpbmcgUy5yLmwuPGJyIGNsYXNzPTNE IiI+VmlhID0KUGV0cm9jY2hpLCA0PGJyIGNsYXNzPTNEIiI+MjAxMjcgTWlsYW5vID1FMj04MD05 MyBJdGFseTxiciBjbGFzcz0zRCIiPjxiciA9CmNsYXNzPTNEIiI+VGVsLiAwMiAyNjExMzUwNyBp bnQgMzM5PGJyIGNsYXNzPTNEIiI+PGEgPQpocmVmPTNEIm1haWx0bzpzdGVmYW5vc0BwcmlzbWF0 ZWxlY29tdGVzdGluZy5jb20iIGNsYXNzPTNEIiI+ZS1tYWlsOiA9CnN0ZWZhbm9zQHByaXNtYXRl bGVjb210ZXN0aW5nLmNvbTwvYT48YnIgY2xhc3M9M0QiIj5za3lwZTogPQpzdGVmYW5vLnN0YWdu YXJvPGJyID0KY2xhc3M9M0QiIj5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fXzxiciA9CmNsYXNzPTNEIiI+VXNlcnMgbWFpbGluZyBsaXN0PGJyIGNsYXNzPTNE IiI+VXNlcnNAb3ZpcnQub3JnPGJyID0KY2xhc3M9M0QiIj5odHRwOi8vbGlzdHMub3ZpcnQub3Jn L21haWxtYW4vbGlzdGluZm8vdXNlcnM8YnIgPQpjbGFzcz0zRCIiPjwvZGl2PjwvZGl2PjwvYmxv Y2txdW90ZT48L2Rpdj48YnIgY2xhc3M9M0QiIj48L2JvZHk+PC9odG1sPj0KCi0tQXBwbGUtTWFp bD1fNDM3NTYxQjEtRTQ2QS00RDhFLUI1ODYtMUNGNURCRDUyNkNDLS0K --===============0905976214948883206==--