VM storage issue after snapshot deletion
Hi everyone. I'm having an issue with a VM after initiating a snapshot deletion. Shortly after the deletion started the VM was paused and has not recovered since, this was over 12 hours ago. I've dug into the logs and here's some things that jumped out at me: engine.log Message: VM XXXXXX has been paused due to unknown storage error. vdsm.log Improbable extension request for volume cb180795-1da8-4a4c-90a9-b42d00485018 on domain 0243ebd9-fd3f-4674-b4e7-05af734280e7, pausing the VM to avoid corruptions (capacity: 53687091200, allocated: 5750586880, physical: 1073741824, next physical size: 2147483648) The vdsm.log line I've quoted is repeated approximately every 2 seconds from the first occurrence and is still going. I've attached engine.log and uploaded vdsm.log to my server [1] as it was 10M. The VM in question has the ID 184f89f4-8427-4a28-8787-8912f5c5f038. The VM is still paused and cannot be resumed, seemingly as a precaution to avoid corruption. Is there anything I can do to recover the data on the disk? All help is greatly appreciated. I'm running oVirt 3.6.6.2-1 on CentOS 7. Cheers, Ollie [1] https://ollie.io/vdsm.log
This is a multi-part message in MIME format. ------=_NextPartTM-000-fa1a2595-072c-45aa-b459-d7e1bf34ef6f Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Von: users-bounces@ovirt.org [users-bounces@ovirt.org]" im Auftrag v= on "Ollie Armstrong >[ollie@fubra.com]=0A= Gesendet: Freitag, 5. August 2016 11:39=0A= An: users@ovirt.org=0A= Betreff: [ovirt-users] VM storage issue after snapshot deletion=0A= =0A= Hi everyone.=0A= =0A= I'm having an issue with a VM after initiating a snapshot deletion.=0A= Shortly after the deletion started the VM was paused and has not=0A= recovered since, this was over 12 hours ago.=0A= =0A= Could you check https://bugzilla.redhat.com/show_bug.cgi?id=3D1329543 ?=0A= =0A= Best regards.=0A= =0A= Markus=0A= ------=_NextPartTM-000-fa1a2595-072c-45aa-b459-d7e1bf34ef6f Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt"
**************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-fa1a2595-072c-45aa-b459-d7e1bf34ef6f--
On 5 August 2016 at 11:22, Markus Stockhausen <stockhausen@collogia.de> wrote:
I'm having an issue with a VM after initiating a snapshot deletion. Shortly after the deletion started the VM was paused and has not recovered since, this was over 12 hours ago.
Could you check https://bugzilla.redhat.com/show_bug.cgi?id=1329543 ?
Thanks, looks like that may be the issue. Is there any way I can cancel the current operation (it is still ongoing) so that I can get the hosts upgraded? -- [image: Fubra Ltd.] <http://www.fubra.com/> Ollie Armstrong Sysadmin ollie@fubra.com fubra.com Fubra is a company limited by shares and registered in England and Wales with number 3967214 at Manor Coach House, Church Hill, Aldershot, Hampshire, GU12 4RQ. We are registered for VAT with number GB733667024, and as a data controller with number Z5193400. We are members of RIPE, Nominet, the Italian RA and registered with OfCom as a provider of electronic communications services. *Calls to this number will cost 5p per minute plus your network access charge
------=_NextPartTM-000-db38ee37-9a90-4a55-82a5-b42afd1fcb21 Content-Type: multipart/alternative; boundary="_000_12EF8D94C6F8734FB2FF37B9FBEDD1739B81B5DDEXCHANGEcollogi_" --_000_12EF8D94C6F8734FB2FF37B9FBEDD1739B81B5DDEXCHANGEcollogi_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Bad news, the only option that worked in our case was to kill the VM. Afterwards we s= ometimes had to compare OVirt DB and storage filesystem status. Because some of the disk snapshots = completed in between but OVirt did not recognize. In the worst case we had to fix the state manually= be moving image files (on our NFS), merging images on the command line or manipulating the DB. First shot would be: Stop VM & backup images + snapshots for the recovery c= ase. Afterwards try to start the VM and see what happens. Best regards. Markus Stockhausen ________________________________ Von: Ollie Armstrong [ollie@fubra.com] Gesendet: Freitag, 5. August 2016 13:10 An: Markus Stockhausen Cc: users@ovirt.org Betreff: Re: [ovirt-users] VM storage issue after snapshot deletion On 5 August 2016 at 11:22, Markus Stockhausen <stockhausen@collogia.de<mail= to:stockhausen@collogia.de>> wrote:
I'm having an issue with a VM after initiating a snapshot deletion. Shortly after the deletion started the VM was paused and has not recovered since, this was over 12 hours ago.
Could you check https://bugzilla.redhat.com/show_bug.cgi?id=3D1329543 ? Thanks, looks like that may be the issue. Is there any way I can cancel the current operation (it is still ongoing) s= o that I can get the hosts upgraded? -- [Fubra Ltd.]<http://www.fubra.com/> Ollie Armstrong Sysadmin ollie@fubra.com<mailto:ollie@fubra.com> fubra.com<http://fubra.com/> Fubra is a company limited by shares and registered in England and Wales wi= th number 3967214 at Manor Coach House, Church Hill, Aldershot, Hampshire, = GU12 4RQ. We are registered for VAT with number GB733667024, and as a data = controller with number Z5193400. We are members of RIPE, Nominet, the Itali= an RA and registered with OfCom as a provider of electronic communications = services. *Calls to this number will cost 5p per minute plus your network access char= ge --_000_12EF8D94C6F8734FB2FF37B9FBEDD1739B81B5DDEXCHANGEcollogi_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html dir=3D"ltr"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style type=3D"text/css" id=3D"owaParaStyle"></style> </head> <body fpstyle=3D"1" ocsi=3D"0"> <div style=3D"direction: ltr;font-family: Arial;color: #000000;font-size: 1= 0pt;"> <div> <div style=3D"font-family:Tahoma; font-size:13px"> <div style=3D"font-family:Tahoma; font-size:13px"> <div style=3D"font-family:Tahoma; font-size:13px"> <div style=3D"font-family:Tahoma; font-size:13px"> <div style=3D"font-family:Tahoma; font-size:13px"> <div> <div> <div>Bad news,</div> <div><br> </div> <div>the only option that worked in our case was to kill the VM. Afterwards= we sometimes had to compare</div> <div>OVirt DB and storage filesystem status. Because some of the disk snaps= hots completed in between but</div> <div>OVirt did not recognize. In the worst case we had to fix the state man= ually be moving image files</div> <div>(on our NFS), merging images on the command line or manipulating the D= B. </div> <div><br> </div> <div>First shot would be: Stop VM & backup images + snapshots for t= he recovery case. Afterwards try to</div> <div>start the VM and see what happens. </div> <div><br> </div> <div>Best regards.</div> <div><br> </div> <div>Markus Stockhausen</div> <div><br> </div> </div> </div> </div> </div> </div> </div> <div id=3D"ofmeet-extension-installed" style=3D"display:none"></div> </div> </div> <div style=3D"font-family: Times New Roman; color: #000000; font-size: 16px= "> <hr tabindex=3D"-1"> <div id=3D"divRpF901747" style=3D"direction: ltr;"><font face=3D"Tahoma" si= ze=3D"2" color=3D"#000000"><b>Von:</b> Ollie Armstrong [ollie@fubra.com]<br=
<b>Gesendet:</b> Freitag, 5. August 2016 13:10<br> <b>An:</b> Markus Stockhausen<br> <b>Cc:</b> users@ovirt.org<br> <b>Betreff:</b> Re: [ovirt-users] VM storage issue after snapshot deletion<= br> </font><br> </div> <div></div> <div> <div dir=3D"ltr"> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">On 5 August 2016 at 11:22, Markus Stockhausen <s= pan dir=3D"ltr"> <<a href=3D"mailto:stockhausen@collogia.de" target=3D"_blank">stockhause= n@collogia.de</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex; border-left:1= px #ccc solid; padding-left:1ex"> <span class=3D"">> I'm having an issue with a VM after initiating a snap= shot deletion.<br> > Shortly after the deletion started the VM was paused and has not<br> > recovered since, this was over 12 hours ago.<br> <br> </span>Could you check <a href=3D"https://bugzilla.redhat.com/show_bug.cgi?= id=3D1329543" rel=3D"noreferrer" target=3D"_blank"> https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=3D1329543</a> ?</blockquot= e> </div> <br> </div> <div class=3D"gmail_extra">Thanks, looks like that may be the issue.<br> <br> </div> <div class=3D"gmail_extra">Is there any way I can cancel the current operat= ion (it is still ongoing) so that I can get the hosts upgraded?<br> </div> <div class=3D"gmail_extra"><br clear=3D"all"> <br> -- <br> <div class=3D"gmail_signature"> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <table bgcolor=3D"#fff" border=3D"0" cellpadding=3D"0" cellspacing=3D"0" he= ight=3D"125" width=3D"520" style=3D"border:1px solid rgb(228,228,228); font= -family:Arial; line-height:18px; letter-spacing:1px"> <tbody> <tr> <td height=3D"75" width=3D"120" style=3D"border:none; line-height:0; color:= rgb(153,153,153)"> <a href=3D"http://www.fubra.com/" target=3D"_blank"><img alt=3D"Fubra Ltd."= src=3D"http://signatures.fubra.com/logos/slim/fubra.png" height=3D"75" wid= th=3D"120" border=3D"0"></a></td> <td bgcolor=3D"#fff" height=3D"75" width=3D"400" style=3D"padding-left:15px= ; font-family:Helvetica,'Lucida Grande',Arial,Times; font-size:12px; border= -left-color:rgb(228,228,228); border-left-style:solid; border-left-width:1p= x"> <div style=3D"min-height:5px; line-height:5px"> </div> <div style=3D"font-size:18px; overflow:visible">Ollie Armstrong <div style=3D"height:16px; float:right; padding-right:15px"></div> </div> <div style=3D"min-height:5px; line-height:5px"> </div> <div style=3D"color:rgb(119,119,119)">Sysadmin<br> </div> <div style=3D"font-size:10px; color:rgb(95,131,173)"><a href=3D"mailto:olli= e@fubra.com" target=3D"_blank">ollie@fubra.com</a> = <a href=3D"http://fubra.com/" target=3D"_blank">fubra.com</a></div> </td> </tr> <tr> <td colspan=3D"2" bgcolor=3D"#fff" height=3D"50" width=3D"520" style=3D"fon= t-size:7px; padding:5px 10px; line-height:10px; letter-spacing:0px; color:r= gb(102,102,102); font-weight:bold; border-top-color:rgb(228,228,228); borde= r-top-style:solid; border-top-width:1px"> Fubra is a company limited by shares and registered in England and Wales wi= th number 3967214 at Manor Coach House, Church Hill, Aldershot, Hampshire, = GU12 4RQ. We are registered for VAT with number GB733667024, and as a data = controller with number Z5193400. We are members of RIPE, Nominet, the Italian RA and registered with OfCom = as a provider of electronic communications services.<br> <small>*Calls to this number will cost 5p per minute plus your network acce= ss charge</small></td> </tr> </tbody> </table> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <div id=3D"ofmeet-extension-installed" style=3D"display: none;"></div> </div> </body> </html> --_000_12EF8D94C6F8734FB2FF37B9FBEDD1739B81B5DDEXCHANGEcollogi_-- ------=_NextPartTM-000-db38ee37-9a90-4a55-82a5-b42afd1fcb21 Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-db38ee37-9a90-4a55-82a5-b42afd1fcb21--
On 5 August 2016 at 14:21, Markus Stockhausen <stockhausen@collogia.de> wrote:
Bad news,
the only option that worked in our case was to kill the VM. Afterwards we sometimes had to compare OVirt DB and storage filesystem status. Because some of the disk snapshots completed in between but OVirt did not recognize. In the worst case we had to fix the state manually be moving image files (on our NFS), merging images on the command line or manipulating the DB.
First shot would be: Stop VM & backup images + snapshots for the recovery case. Afterwards try to start the VM and see what happens. ________________________________ Von: Ollie Armstrong [ollie@fubra.com] Gesendet: Freitag, 5. August 2016 13:10 An: Markus Stockhausen Cc: users@ovirt.org Betreff: Re: [ovirt-users] VM storage issue after snapshot deletion
Is there any way I can cancel the current operation (it is still ongoing) so that I can get the hosts upgraded?
Thanks for your help, Markus. I'll think it through over the weekend and see if anyone else has any bright ideas to get me out of this mess and give it a shot Monday. Have a good weekend :)
participants (2)
-
Markus Stockhausen -
Ollie Armstrong