Re: [ovirt-users] Bad volume specification after hung migration

26 Oct 2017

      --Apple-Mail=_7EF46EB5-FF10-4695-889D-DC90FC772541
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8
...
On 26 Oct 2017, at 12:32, Roberto Nunin <robnunin@gmail.com> wrote:
=20
Hi Michael
=20
By frozen I mean the action to put host in maintenance while some VM =
were running on it.
This action wasn't completed after more than one hour.
ok, and was the problem in this last VM not finishing the migration? Was =
it migrating at all? If yes, what was the progress in UI, any failures? =
There are various timeouts which should have been triggered, so if they =
were not triggered it would indeed point to some internal issue. Would =
be great to attach source and destination vdsm.log
...
Thinking that shutting down the VM could help, I've done it. Looking =
at results, not.
...
=20
Yes, I've restarted the ovirt-engine service, I've still not restarted =
What was the result? Did it fail to shut down? Did you use Power Off to =
force immediate shutdown? If it was migrating, did you try to cancel the =
migration first?

the hosted-engine VM.

well, it=E2=80=99s not only a universal =E2=80=9Cfix=E2=80=9D of various =
things, sometimes it does more harm than benefit too. Logs would be =
helpful.
...
Hosts still not restarted. Do you think can help ?
hard to say. Either way please salvage logs first
...
=20
Obviously we will migrate, this activities are enabling us to have =
redundancy at the storage level, then we will migrate to 4.1.x
...
=20
Thanks
=20
2017-10-26 12:26 GMT+02:00 Michal Skrivanek =
<michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>>:
=20
...
On 26 Oct 2017, at 10:20, Roberto Nunin <robnunin@gmail.com =
<mailto:robnunin@gmail.com>> wrote:
=20
We are running 4.0.1.1-1.el7.centos
=20
Hi,
any reason not to upgrade to 4.1?
=20
=20
After a frozen migration attempt, we have two VM that after shutdown, =
are not anymore able to be started up again.
=20
what do you mean by frozen? Are you talking about "VM live migration" =
or =E2=80=9Clive storage migration=E2=80=9D?
How exactly did you resolve that situation, you only shut down those =
VMs? No other troubleshooting steps, e.g. restarting engine, hosts, =
great:)

Thanks,
michal

things like that?
...
=20
Thanks,
michal
...
=20
Message returned is :
=20
Bad volume specification {'index': '0', u'domainID': =
u'731d95a9-61a7-4c7a-813b-fb1c3dde47ea', 'reqsize': '0', u'format': =
u'cow', u'optional': u'false', u'address': {u'function': u'0x0', u'bus': =
u'0x00', u'domain': u'0x0000', u'type': u'pci', u'slot': u'0x05'}, =
u'volumeID': u'cffc70ff-ed72-46ef-a369-4be95de72260', 'apparentsize': =
'3221225472', u'imageID': u'3fe5a849-bcc2-42d3-93c5aca4c504515b', =
u'specParams': {}, u'readonly': u'false', u'iface': u'virtio', =
u'deviceId': u'3fe5a849bcc2-42d3-93c5-aca4c504515b', 'truesize': =
'3221225472', u'poolID': u'00000001-0001-0001-0001-0000000001ec', =
u'device': u'disk', u'shared': u'false', u'propagateErrors': =
u'off',u'type':u'disk'}
=20
Probably this is caused by a wrong pointer into the database that =
still refer to the migration image-id.
=20
If we search within all_disks view, we can find that parentid field =
isn't 00000000-0000-0000-0000-000000000000 like all other running vm, =
but it has a value:
=20
       vm_names       |               parentid
----------------------+--------------------------------------
 working01.company.xx | 00000000-0000-0000-0000-000000000000
 working02.company.xx | 00000000-0000-0000-0000-000000000000
 working03.company.xx | 00000000-0000-0000-0000-000000000000
 working04.company.xx | 00000000-0000-0000-0000-000000000000
 broken001.company.xx | 30533842-2c83-4d0e-95d2-48162dbe23bd =
<<<<<<<<<
 working05.company.xx | 00000000-0000-0000-0000-000000000000
=20
=20
How we can recover from this ?
=20
Thanks in advance
Regards,
=20
--=20
Robert=E2=80=8Bo=E2=80=8B
=20
=20
=20
_______________________________________________
Users mailing list
Users@ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
=20
=20
=20
=20
--=20
Roberto
=20
=20
=20
--Apple-Mail=_7EF46EB5-FF10-4695-889D-DC90FC772541
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div =
class=3D"">On 26 Oct 2017, at 12:32, Roberto Nunin <<a =
href=3D"mailto:robnunin@gmail.com" class=3D"">robnunin@gmail.com</a>> =
wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div =
dir=3D"ltr" class=3D""><div class=3D"gmail_default" style=3D"">Hi =
Michael</div><div class=3D"gmail_default" style=3D""><br =
class=3D""></div><div class=3D"gmail_default" style=3D"">By frozen I =
mean the action to put host in maintenance while some VM were running on =
it.</div><div class=3D"gmail_default" style=3D"">This action wasn't =
completed after more than one =
hour.</div></div></div></blockquote><div><br class=3D""></div>ok, and =
was the problem in this last VM not finishing the migration? Was it =
migrating at all? If yes, what was the progress in UI, any failures? =
There are various timeouts which should have been triggered, so if they =
were not triggered it would indeed point to some internal issue. Would =
be great to attach source and destination vdsm.log</div><div><br =
class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div =
dir=3D"ltr" class=3D""><div class=3D"gmail_default" style=3D"">Thinking =
that shutting down the VM could help, I've done it. Looking at results, =
not.</div></div></div></blockquote><div><br class=3D""></div>What was =
the result? Did it fail to shut down? Did you use Power Off to force =
immediate shutdown? If it was migrating, did you try to cancel the =
migration first?</div><div><br class=3D""><blockquote type=3D"cite" =
class=3D""><div class=3D""><div dir=3D"ltr" class=3D""><div =
class=3D"gmail_default" style=3D""><br class=3D""></div><div =
class=3D"gmail_default" style=3D"">Yes, I've restarted the ovirt-engine =
service, I've still not restarted the hosted-engine =
VM.</div></div></div></blockquote><div><br class=3D""></div>well, it=E2=80=
=99s not only a universal =E2=80=9Cfix=E2=80=9D of various things, =
sometimes it does more harm than benefit too. Logs would be =
helpful.</div><div><br class=3D""><blockquote type=3D"cite" =
class=3D""><div class=3D""><div dir=3D"ltr" class=3D""><div =
class=3D"gmail_default" style=3D"">Hosts still not restarted. Do you =
think can help ?</div></div></div></blockquote><div><br =
class=3D""></div>hard to say. Either way please salvage logs =
first</div><div><br class=3D""><blockquote type=3D"cite" class=3D""><div =
class=3D""><div dir=3D"ltr" class=3D""><div class=3D"gmail_default" =
style=3D""><br class=3D""></div><div class=3D"gmail_default" =
style=3D"">Obviously we will migrate, this activities are enabling us to =
have redundancy at the storage level, then we will migrate to =
4.1.x</div></div></div></blockquote><div><br =
class=3D""></div>great:)</div><div><br =
class=3D""></div><div>Thanks,</div><div>michal</div><div><br =
class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div =
dir=3D"ltr" class=3D""><div class=3D"gmail_default" style=3D""><br =
class=3D""></div><div class=3D"gmail_default" style=3D"">Thanks</div><div =
class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">2017-10-26=
 12:26 GMT+02:00 Michal Skrivanek <span dir=3D"ltr" class=3D""><<a =
href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" =
class=3D"">michal.skrivanek@redhat.com</a>></span>:<br =
class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div =
style=3D"word-wrap:break-word" class=3D""><br class=3D""><div =
class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On 26 =
Oct 2017, at 10:20, Roberto Nunin <<a =
href=3D"mailto:robnunin@gmail.com" target=3D"_blank" =
class=3D"">robnunin@gmail.com</a>> wrote:</div><br =
class=3D"m_-7560322293096975458Apple-interchange-newline"><div =
class=3D""><div dir=3D"ltr" class=3D""><div class=3D"">We are =
running<span style=3D"font-size:large" class=3D""> <span =
style=3D"font-family:'Arial Unicode =
MS',Arial,sans-serif;text-align:-webkit-center;font-size:small" =
class=3D"">4.0.1.1-1.el7.centos</span></span></div></div></div></blockquot=
e><div class=3D""><br class=3D""></div>Hi,</div><div class=3D"">any =
reason not to upgrade to 4.1?</div><div class=3D""><span class=3D""><br =
class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div =
dir=3D"ltr" class=3D""><div style=3D"font-size:large" class=3D""><span =
style=3D"font-family:'Arial Unicode =
MS',Arial,sans-serif;text-align:-webkit-center;font-size:small" =
class=3D""><br class=3D""></span></div><div style=3D"font-size:large" =
class=3D""><span style=3D"font-family:'Arial Unicode =
MS',Arial,sans-serif;text-align:-webkit-center;font-size:small" =
class=3D"">After a frozen migration attempt, we have two VM that after =
shutdown, are not anymore able to be started up =
again.</span></div></div></div></blockquote><div class=3D""><br =
class=3D""></div></span>what do you mean by frozen? Are you talking =
about "VM live migration" or =E2=80=9Clive storage =
migration=E2=80=9D?</div><div class=3D"">How exactly did you resolve =
that situation, you only shut down those VMs? No other troubleshooting =
steps, e.g. restarting engine, hosts, things like that?</div><div =
class=3D""><br class=3D""></div><div class=3D"">Thanks,</div><div =
class=3D"">michal</div><div class=3D""><blockquote type=3D"cite" =
class=3D""><div class=3D""><span class=3D""><div dir=3D"ltr" =
class=3D""><div style=3D"font-size:large" class=3D""><span =
style=3D"font-family:'Arial Unicode =
MS',Arial,sans-serif;text-align:-webkit-center;font-size:small" =
class=3D""><br class=3D""></span></div><div style=3D"font-size:large" =
class=3D""><span style=3D"font-family:'Arial Unicode =
MS',Arial,sans-serif;text-align:-webkit-center;font-size:small" =
class=3D"">Message returned is :</span></div><div =
style=3D"font-size:large" class=3D""><span style=3D"font-family:'Arial =
Unicode MS',Arial,sans-serif;text-align:-webkit-center;font-size:small" =
class=3D""><br class=3D""></span></div><div class=3D""><span =
style=3D"font-family:'Arial Unicode =
MS',Arial,sans-serif;text-align:-webkit-center" class=3D""><div =
style=3D"text-align:left" class=3D"">Bad volume specification {'index': =
'0', u'domainID': u'731d95a9-61a7-4c7a-813b-<wbr class=3D"">fb1c3dde47ea',=
 'reqsize': '0', u'format': u'cow', u'optional': u'false', u'address': =
{u'function': u'0x0', u'bus': u'0x00', u'domain': u'0x0000', u'type': =
u'pci', u'slot': u'0x05'}, u'volumeID': u'cffc70ff-ed72-46ef-a369-<wbr =
class=3D"">4be95de72260', 'apparentsize': '3221225472', u'imageID': =
u'3fe5a849-bcc2-42d3-<wbr class=3D"">93c5aca4c504515b', u'specParams': =
{}, u'readonly': u'false', u'iface': u'virtio', u'deviceId': =
u'3fe5a849bcc2-42d3-93c5-<wbr class=3D"">aca4c504515b', 'truesize': =
'3221225472', u'poolID': u'00000001-0001-0001-0001-<wbr =
class=3D"">0000000001ec', u'device': u'disk', u'shared': u'false', =
u'propagateErrors': u'off',u'type':u'disk'}</div><div =
style=3D"text-align:left;font-size:small" class=3D""><br =
class=3D""></div></span></div><div style=3D"font-size:large" =
class=3D""><span style=3D"font-family:'Arial Unicode =
MS',Arial,sans-serif;text-align:-webkit-center;font-size:small" =
class=3D"">Probably this is caused by a wrong pointer into the database =
that still refer to the migration image-id.</span></div><div =
style=3D"font-size:large" class=3D""><span style=3D"font-family:'Arial =
Unicode MS',Arial,sans-serif;text-align:-webkit-center;font-size:small" =
class=3D""><br class=3D""></span></div><div class=3D""><span =
style=3D"font-size:small;font-family:'Arial Unicode =
MS',Arial,sans-serif;text-align:-webkit-center" class=3D"">If we search =
within all_disks view, we can find that parentid field =
isn't </span><font face=3D"Arial Unicode MS, Arial, sans-serif" =
class=3D"">00000000-0000-0000-0000-<wbr class=3D"">000000000000 like all =
other running vm, but it has a value:</font></div><div class=3D""><font =
face=3D"Arial Unicode MS, Arial, sans-serif" class=3D""><br =
class=3D""></font></div><div class=3D""><font class=3D""><div =
class=3D""><font face=3D"monospace, monospace" class=3D"">    =
   vm_names       |      =
         parentid</font></div><div =
class=3D""><font face=3D"monospace, monospace" =
class=3D"">----------------------+-------<wbr =
class=3D"">------------------------------<wbr =
class=3D"">-</font></div><div class=3D""><font face=3D"monospace, =
monospace" class=3D""> working01.company.xx | =
00000000-0000-0000-0000-<wbr class=3D"">000000000000</font></div><div =
class=3D""><font face=3D"monospace, monospace" =
class=3D""> working02.company.xx | 00000000-0000-0000-0000-<wbr =
class=3D"">000000000000</font></div><div class=3D""><font =
face=3D"monospace, monospace" class=3D""> working03.company.xx | =
00000000-0000-0000-0000-<wbr class=3D"">000000000000</font></div><div =
class=3D""><font face=3D"monospace, monospace" =
class=3D""> working04.company.xx | 00000000-0000-0000-0000-<wbr =
class=3D"">000000000000</font></div><div class=3D""><font =
face=3D"monospace, monospace" class=3D""> broken001.company.xx | =
30533842-2c83-4d0e-95d2-<wbr class=3D"">48162dbe23bd =
<<<<<<<<<</font></div><div class=3D""><font =
face=3D"monospace, monospace" class=3D""> working05.company.xx | =
00000000-0000-0000-0000-<wbr class=3D"">000000000000</font></div><div =
style=3D"font-family:"Arial Unicode MS",Arial,sans-serif" =
class=3D""><br class=3D""></div></font></div><div class=3D""><font =
face=3D"Arial Unicode MS, Arial, sans-serif" class=3D""><br =
class=3D""></font></div><div class=3D""><font face=3D"Arial Unicode MS, =
Arial, sans-serif" class=3D"">How we can recover from this =
?</font></div><div class=3D""><font face=3D"Arial Unicode MS, Arial, =
sans-serif" class=3D""><br class=3D""></font></div><div class=3D""><font =
face=3D"Arial Unicode MS, Arial, sans-serif" class=3D"">Thanks in =
advance</font></div><div class=3D""><font face=3D"Arial Unicode MS, =
Arial, sans-serif" class=3D"">Regards,</font></div><div class=3D""><br =
class=3D""></div>-- <br class=3D""><div =
class=3D"m_-7560322293096975458gmail_signature"><div dir=3D"ltr" =
class=3D""><div dir=3D"ltr" class=3D""><font size=3D"4" =
class=3D"">Robert<div style=3D"font-size:large;display:inline" =
class=3D"">=E2=80=8Bo=E2=80=8B</div><br class=3D""><br =
class=3D""></font><span class=3D""><font size=3D"4" class=3D""><br =
class=3D""><br class=3D""></font></span></div></div></div>
</div></span>
______________________________<wbr class=3D"">_________________<br =
class=3D"">Users mailing list<br class=3D""><a =
href=3D"mailto:Users@ovirt.org" target=3D"_blank" =
class=3D"">Users@ovirt.org</a><br class=3D""><a =
href=3D"http://lists.ovirt.org/mailman/listinfo/users" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/<wbr =
class=3D"">mailman/listinfo/users</a><br =
class=3D""></div></blockquote></div><br =
class=3D""></div></blockquote></div><br class=3D""><br clear=3D"all" =
class=3D""><div class=3D""><br class=3D""></div>-- <br class=3D""><div =
class=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div =
dir=3D"ltr" class=3D""><div dir=3D"ltr" class=3D""><font size=3D"4" =
class=3D"">Roberto<br class=3D""><br class=3D""></font><span =
class=3D""><font size=3D"4" class=3D""><br class=3D""><br =
class=3D""></font></span></div></div></div>
</div></div>
</div></blockquote></div><br class=3D""></body></html>=

--Apple-Mail=_7EF46EB5-FF10-4695-889D-DC90FC772541--