4:39 a.m.
Hi everyone.
I'm having an issue with a VM after initiating a snapshot deletion.
Shortly after the deletion started the VM was paused and has not
recovered since, this was over 12 hours ago.
I've dug into the logs and here's some things that jumped out at me:
engine.log
Message: VM XXXXXX has been paused due to unknown storage error.
vdsm.log
Improbable extension request for volume
cb180795-1da8-4a4c-90a9-b42d00485018 on domain
0243ebd9-fd3f-4674-b4e7-05af734280e7, pausing the VM to avoid
corruptions (capacity: 53687091200, allocated: 5750586880, physical:
1073741824, next physical size: 2147483648)
The vdsm.log line I've quoted is repeated approximately every 2
seconds from the first occurrence and is still going. I've attached
engine.log and uploaded vdsm.log to my server [1] as it was 10M. The
VM in question has the ID 184f89f4-8427-4a28-8787-8912f5c5f038.
The VM is still paused and cannot be resumed, seemingly as a
precaution to avoid corruption.
Is there anything I can do to recover the data on the disk? All help
is greatly appreciated.
I'm running oVirt 3.6.6.2-1 on CentOS 7.
Cheers,
Ollie
[1] https://ollie.io/vdsm.log
5:22 a.m.
This is a multi-part message in MIME format.
------=_NextPartTM-000-fa1a2595-072c-45aa-b459-d7e1bf34ef6f
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Von: users-bounces(a)ovirt.org [users-bounces(a)ovirt.org]" im
Auftrag v=
on "Ollie Armstrong >[ollie(a)fubra.com]=0A=
Gesendet: Freitag, 5. August 2016 11:39=0A=
An: users(a)ovirt.org=0A=
Betreff: [ovirt-users] VM storage issue after snapshot deletion=0A=
=0A=
Hi everyone.=0A=
=0A=
I'm having an issue with a VM after initiating a snapshot deletion.=0A=
Shortly after the deletion started the VM was paused and has not=0A=
recovered since, this was over 12 hours ago.=0A=
=0A=
Could you check https://bugzilla.redhat.com/show_bug.cgi?id=3D1329543 ?=0A=
=0A=
Best regards.=0A=
=0A=
Markus=0A=
------=_NextPartTM-000-fa1a2595-072c-45aa-b459-d7e1bf34ef6f
Content-Type: text/plain;
name="InterScan_Disclaimer.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="InterScan_Disclaimer.txt"
****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.
Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
Vorstand:
Kadir Akin
Dr. Michael Höhnerbach
Vorsitzender des Aufsichtsrates:
Hans Kristian Langva
Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
executive board:
Kadir Akin
Dr. Michael Höhnerbach
President of the supervisory board:
Hans Kristian Langva
Registry office: district court Cologne
Register number: HRB 52 497
****************************************************************************
------=_NextPartTM-000-fa1a2595-072c-45aa-b459-d7e1bf34ef6f--
6:10 a.m.
On 5 August 2016 at 11:22, Markus Stockhausen <stockhausen(a)collogia.de>
wrote:
> I'm having an issue with a VM after initiating a snapshot
deletion.
> Shortly after the deletion started the VM was paused and has not
> recovered since, this was over 12 hours ago.
Could you check https://bugzilla.redhat.com/show_bug.cgi?id=1329543 ?
Thanks, looks like that may be the issue.
Is there any way I can cancel the current operation (it is still ongoing)
so that I can get the hosts upgraded?
--
[image: Fubra Ltd.] <http://www.fubra.com/>
Ollie Armstrong
Sysadmin
ollie(a)fubra.com fubra.com
Fubra is a company limited by shares and registered in England and Wales
with number 3967214 at Manor Coach House, Church Hill, Aldershot,
Hampshire, GU12 4RQ. We are registered for VAT with number GB733667024, and
as a data controller with number Z5193400. We are members of RIPE, Nominet,
the Italian RA and registered with OfCom as a provider of electronic
communications services.
*Calls to this number will cost 5p per minute plus your network access
charge
8:21 a.m.
------=_NextPartTM-000-db38ee37-9a90-4a55-82a5-b42afd1fcb21
Content-Type: multipart/alternative;
boundary="_000_12EF8D94C6F8734FB2FF37B9FBEDD1739B81B5DDEXCHANGEcollogi_"
--_000_12EF8D94C6F8734FB2FF37B9FBEDD1739B81B5DDEXCHANGEcollogi_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Bad news,
the only option that worked in our case was to kill the VM. Afterwards we s=
ometimes had to compare
OVirt DB and storage filesystem status. Because some of the disk snapshots =
completed in between but
OVirt did not recognize. In the worst case we had to fix the state manually=
be moving image files
(on our NFS), merging images on the command line or manipulating the DB.
First shot would be: Stop VM & backup images + snapshots for the recovery c=
ase. Afterwards try to
start the VM and see what happens.
Best regards.
Markus Stockhausen
________________________________
Von: Ollie Armstrong [ollie(a)fubra.com]
Gesendet: Freitag, 5. August 2016 13:10
An: Markus Stockhausen
Cc: users(a)ovirt.org
Betreff: Re: [ovirt-users] VM storage issue after snapshot deletion
On 5 August 2016 at 11:22, Markus Stockhausen <stockhausen(a)collogia.de<mail=
to:stockhausen@collogia.de>> wrote:
Ollie Armstrong
Sysadmin
ollie@fubra.com<mailto:ollie@fubra.com> fubra.com<http://fubra.com/
Fubra is a company limited by shares and registered in England and Wales wi=
th number 3967214 at Manor Coach House, Church Hill, Aldershot, Hampshire, =
GU12 4RQ. We are registered for VAT with number GB733667024, and as a data =
controller with number Z5193400. We are members of RIPE, Nominet, the Itali=
an RA and registered with OfCom as a provider of electronic communications =
services.
*Calls to this number will cost 5p per minute plus your network access char=
ge
--_000_12EF8D94C6F8734FB2FF37B9FBEDD1739B81B5DDEXCHANGEcollogi_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html dir=3D"ltr" <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Diso-8859-=
1" <style type=3D"text/css"
id=3D"owaParaStyle"></style
</head <body fpstyle=3D"1"
ocsi=3D"0" <div style=3D"direction:
ltr;font-family: Arial;color: #000000;font-size: 1=
0pt;" <div <div
style=3D"font-family:Tahoma; font-size:13px" <div
style=3D"font-family:Tahoma; font-size:13px" <div
style=3D"font-family:Tahoma; font-size:13px" <div
style=3D"font-family:Tahoma; font-size:13px" <div
style=3D"font-family:Tahoma; font-size:13px"
<div <div
<div>Bad news,</div <div><br </div
<div>the only option that worked in our case was to kill the VM. Afterwards=
we sometimes had to compare</div
<div>OVirt DB and storage filesystem status. Because some of the disk snaps=
hots completed in between but</div
<div>OVirt did not recognize. In the worst case we had to fix the state man=
ually be moving image files</div
<div>(on our NFS), merging images on the command line or manipulating the D=
B. </div <div><br </div
<div>First shot would be: Stop VM & backup images + snapshots
for t=
he recovery case. Afterwards try to</div
<div>start the VM and see what happens. </div <div><br
</div <div>Best regards.</div <div><br
</div <div>Markus
Stockhausen</div <div><br </div
</div </div </div
</div </div </div <div
id=3D"ofmeet-extension-installed"
style=3D"display:none"></div
</div </div <div style=3D"font-family: Times New Roman; color:
#000000; font-size: 16px=
" <hr tabindex=3D"-1" <div
id=3D"divRpF901747" style=3D"direction: ltr;"><font
face=3D"Tahoma" si=
ze=3D"2" color=3D"#000000"><b>Von:</b> Ollie Armstrong
[ollie(a)fubra.com]<br=
<b>Gesendet:</b> Freitag, 5. August 2016
13:10<br <b>An:</b> Markus
Stockhausen<br <b>Cc:</b>
users(a)ovirt.org<br <b>Betreff:</b> Re:
[ovirt-users] VM storage issue after snapshot deletion<=
br </font><br
</div <div></div <div <div dir=3D"ltr" <div class=3D"gmail_extra"><br <div class=3D"gmail_quote">On 5 August 2016 at
11:22, Markus Stockhausen <s=
pan dir=3D"ltr" <<a
href=3D"mailto:stockhausen@collogia.de"
target=3D"_blank">stockhause=
n(a)collogia.de</a>&gt;</span> wrote:<br
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;
border-left:1=
px #ccc solid; padding-left:1ex" <span
class=3D"">> I'm having an issue with a VM after initiating a
snap=
shot deletion.<br > Shortly after the deletion
started the VM was paused and has not<br >
recovered since, this was over 12 hours ago.<br
<br </span>Could you check <a
href=3D"https://bugzilla.redhat.com/show_bug.cgi?=
id=3D1329543" rel=3D"noreferrer" target=3D"_blank"
https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=3D1329543</a>
?</blockquot=
e </div
<br </div <div class=3D"gmail_extra">Thanks, looks like
that may be the issue.<br <br
</div <div
class=3D"gmail_extra">Is there any way I can cancel the current operat=
ion (it is still ongoing) so that I can get the hosts upgraded?<br </div <div
class=3D"gmail_extra"><br clear=3D"all" <br -- <br <div class=3D"gmail_signature" <div dir=3D"ltr"
<div <div dir=3D"ltr" <div <div dir=3D"ltr" <table bgcolor=3D"#fff" border=3D"0"
cellpadding=3D"0" cellspacing=3D"0" he=
ight=3D"125" width=3D"520" style=3D"border:1px solid
rgb(228,228,228); font=
-family:Arial; line-height:18px; letter-spacing:1px"
<tbody <tr <td
height=3D"75" width=3D"120" style=3D"border:none; line-height:0;
color:=
rgb(153,153,153)" <a
href=3D"http://www.fubra.com/" target=3D"_blank"><img
alt=3D"Fubra Ltd."=
src=3D"http://signatures.fubra.com/logos/slim/fubra.png"
height=3D"75" wid=
th=3D"120" border=3D"0"></a></td <td bgcolor=3D"#fff" height=3D"75"
width=3D"400" style=3D"padding-left:15px=
; font-family:Helvetica,'Lucida Grande',Arial,Times; font-size:12px; border=
-left-color:rgb(228,228,228); border-left-style:solid; border-left-width:1p=
x" <div style=3D"min-height:5px;
line-height:5px"> </div <div
style=3D"font-size:18px; overflow:visible">Ollie Armstrong
<div style=3D"height:16px; float:right; padding-right:15px"></div </div <div
style=3D"min-height:5px; line-height:5px"> </div <div
style=3D"color:rgb(119,119,119)">Sysadmin<br
</div <div
style=3D"font-size:10px; color:rgb(95,131,173)"><a
href=3D"mailto:olli=
e(a)fubra.com" target=3D"_blank">ollie(a)fubra.com</a>
=
<a href=3D"http://fubra.com/"
target=3D"_blank">fubra.com</a></div
</td </tr
<tr <td colspan=3D"2"
bgcolor=3D"#fff" height=3D"50" width=3D"520"
style=3D"fon=
t-size:7px; padding:5px 10px; line-height:10px; letter-spacing:0px; color:r=
gb(102,102,102); font-weight:bold; border-top-color:rgb(228,228,228); borde=
r-top-style:solid; border-top-width:1px" Fubra is
a company limited by shares and registered in England and Wales wi=
th number 3967214 at Manor Coach House, Church Hill, Aldershot, Hampshire, =
GU12 4RQ. We are registered for VAT with number GB733667024, and as a data =
controller with number Z5193400.
We are members of RIPE, Nominet, the Italian RA and registered with OfCom =
as a provider of electronic communications services.<br
<small>*Calls to this number will cost 5p per minute plus your network acce=
ss charge</small></td </tr
</tbody </table </div
</div </div </div
</div </div </div
</div </div </div <div
id=3D"ofmeet-extension-installed" style=3D"display:
none;"></div </div </body
</html
--_000_12EF8D94C6F8734FB2FF37B9FBEDD1739B81B5DDEXCHANGEcollogi_--
------=_NextPartTM-000-db38ee37-9a90-4a55-82a5-b42afd1fcb21
Content-Type: text/plain;
name="InterScan_Disclaimer.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="InterScan_Disclaimer.txt"
****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.
Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
Vorstand:
Kadir Akin
Dr. Michael Höhnerbach
Vorsitzender des Aufsichtsrates:
Hans Kristian Langva
Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
executive board:
Kadir Akin
Dr. Michael Höhnerbach
President of the supervisory board:
Hans Kristian Langva
Registry office: district court Cologne
Register number: HRB 52 497
****************************************************************************
------=_NextPartTM-000-db38ee37-9a90-4a55-82a5-b42afd1fcb21--
I'm having an issue with a VM after initiating a snapshot
deletion.
Shortly after the deletion started the VM was paused and has not
recovered since, this was over 12 hours ago.
Could you check https://bugzilla.redhat.com/show_bug.cgi?id=3D1329543 ?
Thanks, looks like that may be the issue.
Is there any way I can cancel the current operation (it is still ongoing) s=
o that I can get the hosts upgraded?
--
[Fubra Ltd.]<http://www.fubra.com/![src=3D"http://signatures.fubra.com/logos/slim/fubra.png"](http://src=3D"http://signatures.fubra.com/logos/slim/fubra.png%22){kind=link}
8:34 a.m.
On 5 August 2016 at 14:21, Markus Stockhausen <stockhausen(a)collogia.de> wrote:
Bad news,
the only option that worked in our case was to kill the VM. Afterwards we sometimes had
to compare
OVirt DB and storage filesystem status. Because some of the disk snapshots completed in
between but
OVirt did not recognize. In the worst case we had to fix the state manually be moving
image files
(on our NFS), merging images on the command line or manipulating the DB.
First shot would be: Stop VM & backup images + snapshots for the recovery case.
Afterwards try to
start the VM and see what happens.
________________________________
Von: Ollie Armstrong [ollie(a)fubra.com]
Gesendet: Freitag, 5. August 2016 13:10
An: Markus Stockhausen
Cc: users(a)ovirt.org
Betreff: Re: [ovirt-users] VM storage issue after snapshot deletion
Is there any way I can cancel the current operation (it is still ongoing) so that I can
get the hosts upgraded?
Thanks for your help, Markus. I'll think it through over the weekend
and see if anyone else has any bright ideas to get me out of this mess
and give it a shot Monday.
Have a good weekend :)
3078
days inactive
3078
days old
4 comments
2 participants
participants (2)
-
Markus Stockhausen -
Ollie Armstrong