3:17 a.m.
This is a multi-part message in MIME format.
--------------030500050202090506010300
Content-Type: text/plain; charset=WINDOWS-1252; format=flowed
Content-Transfer-Encoding: quoted-printable
Hi everyone,
Sunday fun :)
One of our critical Linux VMs ran out of disk space. I cleared up space=20
by removing old application logs within the guest, rebooted the guest=20
and was presented with the following information message during bootup=20
when the filesystem ext3/4 scan runs:
/Clearing orhpaned inode ...../
Shortly after this message the the guest is paused by oVirt and the "VM=20
has paused due to unknown storage error" message is displayed in the=20
oVirt web GUI.
On the physical node itself, I found the following error=20
in/var/log/vdsm/vdsm.log:
/libvirtEventLoop::INFO::2012-12-02=20
09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)=20
vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device=20
ide0-0-1 error eother/
Any ideas on how to get more informationon what exactly the error is?
I've tried booting the guest into rescue mode, run a manual fsck on the=20
logical volume but each time the guest is automaticall paused due to the=20
"unknown storage error".
We are using a FC SAN for storage, sowhat I've decided to try is export=20
the disk using "dd" on the physical node and then try repairing it by=20
mounting it as a loop filesystem on another physical server, and then=20
use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--=20
Get important Linux and industry-related news at: facebook.com/dcdata=20
<http://facebook.com/dcdata
Kind regards,
David Wilson
CNS,CLS, LINUX+, CLA, DCTS, LPIC3
*LinuxTech CC t/a DcData*
CK number: 2001/058368/23
*Website:* =09http://www.dcdata.co.za
*Support:* =09+27(0)860-1-LINUX
*Mobile:* =09+27(0)824147413
*Tel:* =09+27(0)333446100
*Fax:* =09+27(0)866878971
--------------030500050202090506010300
Content-Type: text/html; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable
<html <head
<meta http-equiv=3D"content-type" content=3D"text/html;
charset=3DISO-8=
859-1" </head <body bgcolor=3D"#FFFFFF"
text=3D"#000000"
<font size=3D"-1"><font size=3D"-1">H</font>i
everyone,<br <br <font size=3D"-1"><font
size=3D"-1">Sunday fun :<font size=3D"-1">)<b=
r <br
</font>One of our cr<font size=3D"-1">itical <font
size=3D"-1">Li=
nux
VMs</font> ran out of </font>disk space. I cleared up
space by removing old application logs within the guest,
rebooted the g<font size=3D"-1">uest and was presented with the
following information message during bootup when the
filesystem <font size=3D"-1">ext3/<font
size=3D"-1">4</font></f=
ont scan runs:<br
<font size=3D"-1"><i>Clearing orhpaned inode
.....</i><br <font
size=3D"-1"><font size=3D"-1">Shortly after this messag=
e
the the guest is p<font size=3D"-1">aused by oVirt and
the "VM has paused due to unknown storage error"
message is displayed in the <font size=3D"-1">oVirt
web <font size=3D"-1">GUI.<br <br
<font size=3D"-1">On the physical node itself, <fon=
t
size=3D"-1">I fo<font
size=3D"-1">und t<font
size=3D"-1">he following error in<font
size=3D"-1"> /var/log/vdsm/vd<font
size=3D"-1">sm.log:<br <font
size=3D"-1"><i>libvirtEventLoop::=
<a class=3D"moz-txt-link-freetext"
href=3D"INFO::2012-12-02">INFO::2012-12-=
02</a
09:05:50,296::libvirtvm::1965::vm.V=
m::(_onAbnormalStop)
vmId=3D`23b9212c-1e25-4003-aa18-b1e=
819bf6bb1`::abnormal
vm stop device ide0-0-1 error
eother</i><br <br <font
size=3D"-1">Any ideas on how
to get more inf<font
size=3D"-1">or=
mation<font
size=3D"-1"> on what exactly
the <font size=3D"-1">error
is?<br
</font></font></font><br <font
size=3D"-1">I've <font
size=3D"-1">tried booting the
guest <font size=3D"-1">into
r<font
size=3D"-1">es<font
size=3D"-1">cue
mode<font
size=3D"-1">, run a
manual fsck <font
size=3D"-1">on the
logical v<font
size=3D"-1">olume
but each</font></fo=
nt></font></font></font></font></font></font time the guest is
automaticall
paused due to the "unknown
storage error".<br <br We are using a FC SAN
for s<font
size=3D"-1">torage</font></font>,
s<font size=3D"-1">o<font
size=3D"-1"=
what I've decided
to</font></font=
try is expo<font
size=3D"-1">rt the
disk using "dd" <font
size=3D"-1">o=
n
the physical node </font>and
t<font size=3D"-1">hen try
repairing it <font
size=3D"-1">by
<font size=3D"-1">mounting it
as a loop filesystem on <font
size=3D"-1">a<font
size=3D"-1">nother
physical</font></font></f=
ont></font></font></font
ser<font size=3D"-1">ver<font
size=3D"-1">, and then use
"dd"
to import it back in again.</font=
><br
<br
<font size=3D"-1">Is this the
right approach to take to get
the system back online?<br <br
</font></font></font></font></font>=
</font></font></font></font></font></font></font></font></font></font></fon=
t></font></font><br
<br </font
<div class=3D"moz-signature">-- <br
<br <br Get
important Linux and industry-related news at: <a
href=3D"http://facebook.com/dcdata">facebook.com/dcdata</... <br
<font color=3D"#000000"><font face=3D"arial"
size=3D"2">Kind regards,=
</font><br <br <font
color=3D"#000000"><font face=3D"arial"><font
size=3D"4"=
<font color=3D"#000000"><font
face=3D"arial"><font size=3D"1"=
<font color=3D"#0779b5"><font
face=3D"arial"><font size=3D"3"=
<font color=3D"#0779b5"><font
face=3D"arial"><font size=3D"1"=
<table border=3D"0" <tbody <tr
<td width=3D"75"><font color=3D"#000000"><font
face=3D"arial"=
<td><font
color=3D"#000000"><font face=3D"arial" </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial" <td><font
color=3D"#000000"><font face=3D"arial" </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial" <td><font
color=3D"#000000"><font face=3D"arial" </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial" <td><font
color=3D"#000000"><font face=3D"arial" </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial" <td><font
color=3D"#000000"><font face=3D"arial" </tr
</tbody </table </font></div =20
<br><br <p style=3D"font-family:
Verdana; font-size:8pt; color:#666666;" align=3D"j=
ustify"
Time to evaluate your email security provider? <a href=3D"http://www.mimeca=
st.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast=
=92s first ever limited promotion.
</p </body
</html
--------------030500050202090506010300--
David
Wilson</font></font></font><brCNS,CLS,
LINUX+, CLA, DCTS,
LPIC3</font></font></font><br<b>LinuxTech
CC t/a
DcData</b></font></font></font><brCK
number:
2001/058368/23</font></font></font><br<font
size=3D"2"><b>Website:</b></font></font></font></td<font
size=3D"2"><a
class=3D"moz-txt-link-freetext" href=3D=
"http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></font><=
/td<font
size=3D"2"><b>Support:</b></font></font></font></td<font
size=3D"2">+27(0)860-1-LINUX</font></font></font></td=
<font
size=3D"2"><b>Mobile:</b></font></font></font></td<font
size=3D"2">+27(0)824147413</font></font></font></td<font
size=3D"2"><b>Tel:</b></font></font></font></td<font
size=3D"2">+27(0)333446100</font></font></font></td<font
size=3D"2"><b>Fax:</b></font></font></font></td<font
size=3D"2">+27(0)866878971</font></font></font></td
2:36 a.m.
It seems that your guest file system is corrupted.
The dd approach won't help cause this is a virtual disk,
you need to fix the virtual disk inside the lv, that's why any operations on the lv
won't help.
You can try shutting down the vm (not paused, but literally down) and add the guest vm
disk as
a second disk to another vm.
In the second vm try to run fsck on the disk and see if it helps.
In any case before trying to manipulate the disk image please save a backup!!!
let me know how it goes...
----- Original Message -----
From: "David Wilson" <dave(a)dcdata.co.za>
To: users(a)ovirt.org
Sent: Sunday, December 2, 2012 11:17:43 AM
Subject: [Users] VM has paused due to unknown storage error
H i everyone,
Sunday fun : )
One of our cr itical Linux VMs ran out of disk space. I cleared up
space by removing old application logs within the guest, rebooted
the g uest and was presented with the following information message
during bootup when the filesystem ext3/ 4 scan runs:
Clearing orhpaned inode .....
Shortly after this message the the guest is p aused by oVirt and the
"VM has paused due to unknown storage error" message is displayed in
the oVirt web GUI.
On the physical node itself, I fo und t he following error in
/var/log/vdsm/vd sm.log:
libvirtEventLoop:: INFO::2012-12-02
09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)
vmId=`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device
ide0-0-1 error eother
Any ideas on how to get more inf ormation on what exactly the error
is?
I've tried booting the guest into r es cue mode , run a manual fsck
on the logical v olume but each time the guest is automaticall
paused due to the "unknown storage error".
We are using a FC SAN for s torage , s o what I've decided to try is
expo rt the disk using "dd" on the physical node and t hen try
repairing it by mounting it as a loop filesystem on a nother
physical ser ver , and then use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--
Get important Linux and industry-related news at: facebook.com/dcdata
Kind regards,
David Wilson
CNS,CLS, LINUX+, CLA, DCTS, LPIC3
LinuxTech CC t/a DcData
CK number: 2001/058368/23
Website: http://www.dcdata.co.za
Support: +27(0)860-1-LINUX
Mobile: +27(0)824147413
Tel: +27(0)333446100
Fax: +27(0)866878971
Time to evaluate your email security provider? Watch the video and
take advantage of Mimecast’s first ever limited promotion.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
2:55 a.m.
This is a multi-part message in MIME format.
--------------070609000607040903050604
Content-Type: text/plain; charset=WINDOWS-1252; format=flowed
Content-Transfer-Encoding: quoted-printable
Hi everyone,
Some feedbackon this error, and a question I hope that someone could=20
possibly assist with.
I managed to get the guest running again byusing dd and qemu-img to=20
export the filesystem image and base image, convert to raw format and=20
run a fsck on the looped partition.
When mounted as a loop filesystem, fsck easily repaired it.
Any ideas why oVirt kept pausing the guest and did not allow fsck to=20
repair the filesystem?
Get important Linux and industry-related news at: facebook.com/dcdata=20
<http://facebook.com/dcdata
Kind regards,
David Wilson
CNS,CLS, LINUX+, CLA, DCTS, LPIC3
*LinuxTech CC t/a DcData*
CK number: 2001/058368/23
*Website:* =09http://www.dcdata.co.za
*Support:* =09+27(0)860-1-LINUX
*Mobile:* =09+27(0)824147413
*Tel:* =09+27(0)333446100
*Fax:* =09+27(0)866878971
On 12/02/2012 11:17 AM, David Wilson wrote:
<head <meta content=3D"text/html;
charset=3DISO-8859-1"
http-equiv=3D"Content-Type"
</head <body
bgcolor=3D"#FFFFFF" text=3D"#000000"
<div class=3D"moz-cite-prefix"><font
size=3D"-1">H<font size=3D"-1">i</=
font everyone,<br
<br <font size=3D"-1"><font
size=3D"-1">Some feedback<font size=3D"-1">=
on
this error<font size=3D"-1">, and a <font
size=3D"-1">questio=
n
I hope that someone could possibly assist with.</font></f=
ont><br <font
size=3D"-1">I managed to <font size=3D"-1">get the
guest running again by<font size=3D"-1"> using dd
<font
size=3D"-1">and q<font
size=3D"-1">emu-img to export
the filesystem image and base image<font
size=3D"-1">, convert to raw format and run a
fsck on the <font size=3D"-1">loop<font
size=3D"-1">ed part<font
size=3D"-1">ition.<b=
r <font
size=3D"-1">When mounted as a loop
filesystem, fsck easily repaired <font
size=3D"-1">it<font
size=3D"-1">.</font=
</font><font
size=3D"-1">Any ideas why
oVirt kept pausing the guest and did
no<font size=3D"-1">t allow fsck to
<font
size=3D"-1">repair the <font
size=3D"-1">filesystem?<br <br
</font></font></font></font></font></=
font></font></font></font></font></font></font></font></font></font></font>=
<br </font
<div class=3D"moz-signature"
<br <br
Get important Linux and industry-related news at: <a
href=3D"http://facebook.com/dcdata">facebook.com/dcdata</... <br
<font color=3D"#000000"><font face=3D"arial"
size=3D"2">Kind regard=
s,</font><br <br <font
color=3D"#000000"><font face=3D"arial"><font
size=3D"=
4">David
Wilson</font></font <tbody <tr
<td width=3D"75"><font color=3D"#000000"><font
face=3D"aria=
l"><font
size=3D"2"><b>Website:</b></font></font></font></td=
<td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><a
class=3D"moz-txt-link-freetext" href=
=3D"http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></fon=
t></td </tr <tr
<td><font color=3D"#000000"><font
face=3D"arial"><font
size=3D"2"><b>Support:</b></font></font></font></td=
<td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2">+27(0)860-1-LINUX</font></font></font></=
td </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><b>Mobile:</b></font></font></font></td <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2">+27(0)824147413</font></font></font></td=
</tr
<tr <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><b>Tel:</b></font></font></font></td <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2">+27(0)333446100</font></font></font></td=
</tr
<tr <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><b>Fax:</b></font></font></font></td <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2">+27(0)866878971</font></font></font></td=
</tr
</tbody </table </font></div On
12/02/2012 11:17 AM, David Wilson wrote:<br
</div <blockquote
cite=3D"mid:50BB1CB7.8020007@dcdata.co.za" type=3D"cite" <meta http-equiv=3D"content-type"
content=3D"text/html;
charset=3DISO-8859-1" <font
size=3D"-1"><font size=3D"-1">H</font>i
everyone,<br <br <font size=3D"-1"><font
size=3D"-1">Sunday fun :<font size=3D"-1">)=
<br <br
</font>One of our cr<font size=3D"-1">itical <font
size=3D"-1">=
Linux
VMs</font> ran out of </font>disk space. I cleared up
space by removing old application logs within the guest,
rebooted the g<font size=3D"-1">uest and was presented with
the following information message during bootup when the
filesystem <font size=3D"-1">ext3/<font
size=3D"-1">4</font><=
/font scan runs:<br
<font size=3D"-1"><i>Clearing orhpaned inode
.....</i><br <font
size=3D"-1"><font size=3D"-1">Shortly after this
message the the guest is p<font size=3D"-1">aused by
oVirt and the "VM has paused due to unknown
storage error" message is displayed in the <font
size=3D"-1">oVirt web <font
size=3D"-1">GUI.<br
<br <font
size=3D"-1">On the physical node itself, <f=
ont
size=3D"-1">I fo<font
size=3D"-1">und t<font
size=3D"-1">he following error in<font
size=3D"-1"> /var/log/vdsm/vd<font
size=3D"-1">sm.log:<br <font
size=3D"-1"><i>libvirtEventLoop=
::<a
moz-do-not-send=3D"true"
class=3D"moz-txt-link-freetext"
href=3D"INFO::2012-12-02">INFO:=
:2012-12-02</a
09:05:50,296::libvirtvm::1965::vm=
.Vm::(_onAbnormalStop)
vmId=3D`23b9212c-1e25-4003-aa18-b=
1e819bf6bb1`::abnormal
vm stop device ide0-0-1 error
eother</i><br <br <font
size=3D"-1">Any ideas on how
to get more inf<font
size=3D"-1">=
ormation<font
size=3D"-1"> on what exactly
the <font size=3D"-1">error
is?<br
</font></font time the guest is
automaticall
paused due to the "unknown
storage error".<br <br We are using a FC SAN
for s<font
size=3D"-1">torage</font></font=
ser<font size=3D"-1">ver<font
size=3D"-1">, and then use
"dd" to import it back in
again.</font><br <br <font
size=3D"-1">Is this the
right approach to take to
get the system back online?<br <br
</font></font></font></font></fon=
t></font></font></font></font></font></font></font></font></font></font></f=
ont></font </font
<div class=3D"moz-signature">-- <br
<br <br
Get important Linux and industry-related news at: <a
moz-do-not-send=3D"true"
href=3D"http://facebook.com/dcdata">face=
book.com/dcdata</a><br <br <font color=3D"#000000"><font
face=3D"arial" size=3D"2">Kind regard=
s,</font><br <br <font
color=3D"#000000"><font face=3D"arial"><font
size=3D"=
4">David
Wilson</font></font <tbody <tr
<td width=3D"75"><font color=3D"#000000"><font
face=3D"aria=
l"><font
size=3D"2"><b>Website:</b></font></font></font></td=
<td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><a moz-do-not-send=3D"true"
class=3D"moz-txt-link-freetext"
href=3D"http://www.dcdata.co.za">http://www.dcdat=
a.co.za</a></font></font></font></td </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><b>Support:</b></font></font></font></td=
<td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2">+27(0)860-1-LINUX</font></font></font></=
td </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><b>Mobile:</b></font></font></font></td <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2">+27(0)824147413</font></font></font></td=
</tr
<tr <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><b>Tel:</b></font></font></font></td <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2">+27(0)333446100</font></font></font></td=
</tr
<tr <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><b>Fax:</b></font></font></font></td <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2">+27(0)866878971</font></font></font></td=
</tr
</tbody </table </font></div
</blockquote <br =20
<br><br <p style=3D"font-family:
Verdana; font-size:8pt; color:#666666;" align=3D"j=
ustify"
Time to evaluate your email security provider? <a href=3D"http://www.mimeca=
st.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast=
=92s first ever limited promotion.
</p </body
</html
--------------070609000607040903050604--
Hi everyone,
Sunday fun :)
One of our critical Linux VMs ran out of disk space. I cleared up=20
space by removing old application logs within the guest, rebooted the=20
guest and was presented with the following information message during=20
bootup when the filesystem ext3/4 scan runs:
/Clearing orhpaned inode ...../
Shortly after this message the the guest is paused by oVirt and the=20
"VM has paused due to unknown storage error" message is displayed in=20
the oVirt web GUI.
On the physical node itself, I found the following error=20
in/var/log/vdsm/vdsm.log:
/libvirtEventLoop::INFO::2012-12-02=20
09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)=20
vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device=20
ide0-0-1 error eother/
Any ideas on how to get more informationon what exactly the error is?
I've tried booting the guest into rescue mode, run a manual fsck on=20
the logical volume but each time the guest is automaticall paused due=20
to the "unknown storage error".
We are using a FC SAN for storage, sowhat I've decided to try is=20
export the disk using "dd" on the physical node and then try repairing=20
it by mounting it as a loop filesystem on another physical server, and=20
then use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--=20
Get important Linux and industry-related news at: facebook.com/dcdata=20
<http://facebook.com/dcdata
Kind regards,
David Wilson
CNS,CLS, LINUX+, CLA, DCTS, LPIC3
*LinuxTech CC t/a DcData*
CK number: 2001/058368/23
*Website:* =09http://www.dcdata.co.za
*Support:* =09+27(0)860-1-LINUX
*Mobile:* =09+27(0)824147413
*Tel:* =09+27(0)333446100
*Fax:* =09+27(0)866878971
--------------070609000607040903050604
Content-Type: text/html; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable
<html</font><br>
<br</font><br>
<font
color=3D"#000000"><font face=3D"arial"><font
size=3D"=
1">CNS,CLS,
LINUX+, CLA, DCTS, LPIC3</font></font</font><br>
<font
color=3D"#0779b5"><font face=3D"arial"><font
size=3D"=
3"><b>LinuxTech
CC t/a DcData</b></font></font</font><br>
<font
color=3D"#0779b5"><font face=3D"arial"><font
size=3D"=
1">CK
number: 2001/058368/23</font></font</font><br>
<table
border=3D"0"</font><br>
<font size=3D"-1">I've <font
size=3D"-1">tried booting
the guest <font
size=3D"-1">i=
nto
r<font
size=3D"-1">es<font
size=3D"-1">cue
mode<fo=
nt
size=3D"-1">, run a
manual fsck <font
size=3D"-1">on the
logical v<font
size=3D"-1">olume
but each</font></=
font></font></font></font></font></font></font,
s<font
size=3D"-1">o<font
size=3D"-1"> what I've
decided
to</font></font> try is
expo<fo=
nt
size=3D"-1">rt the disk using
"dd" <font
size=3D"-1">on the
physical node </font>and t<font
size=3D"-1">hen try repairing
it <font size=3D"-1">by
<font
size=3D"-1">mounting it as
a loop filesystem on <font
size=3D"-1">a<font
size=3D"-1">nother
physical</font></font><=
/font></font></font></font</font><br>
<br</font><br>
<font
color=3D"#000000"><font face=3D"arial"><font
size=3D"=
1">CNS,CLS,
LINUX+, CLA, DCTS, LPIC3</font></font</font><br>
<font
color=3D"#0779b5"><font face=3D"arial"><font
size=3D"=
3"><b>LinuxTech
CC t/a DcData</b></font></font</font><br>
<font
color=3D"#0779b5"><font face=3D"arial"><font
size=3D"=
1">CK
number: 2001/058368/23</font></font</font><br>
<table
border=3D"0"
3:30 a.m.
This is a multi-part message in MIME format.
--------------000007070806030102080207
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
Thanks for your reply Yeela. I appreciate your feedback.
Please see my feedback to the list earlier..... weird that I was able to=20
repair the disk using another physical system.Butwhen attached to the=20
guestoVirt kept pausing the guest which prevented fsck from completing.
Get important Linux and industry-related news at: facebook.com/dcdata=20
<http://facebook.com/dcdata
Kind regards,
David Wilson
CNS,CLS, LINUX+, CLA, DCTS, LPIC3
*LinuxTech CC t/a DcData*
CK number: 2001/058368/23
*Website:* =09http://www.dcdata.co.za
*Support:* =09+27(0)860-1-LINUX
*Mobile:* =09+27(0)824147413
*Tel:* =09+27(0)333446100
*Fax:* =09+27(0)866878971
On 12/03/2012 10:36 AM, Yeela Kaplan wrote:
<head <meta content=3D"text/html; charset=3DUTF-8"
http-equiv=3D"Content-Type=
" </head
<body bgcolor=3D"#FFFFFF" text=3D"#000000"
<div class=3D"moz-cite-prefix"><font
size=3D"-1">Thanks for your reply
Yeela. I appreciate your feedback.<br
<br <font size=3D"-1">Please see my
<font size=3D"-1">feedback to the
list earlier..... weird that I was a<font size=3D"-1">ble to
repair the disk using another physical system<font
size=3D"-1"><font size=3D"-1">.<font
size=3D"-1"> Bu<font
size=3D"-1">t<font size=3D"-1"> when
attached to the
guest</font></font></font></font><font
size=3D"-1" oVirt kept pausing
the guest which prevented fsck from
completing<font size=3D"-1">.<br <font size=3D"-1"><font
size=3D"-1"></font></font><br
<br
</font></font></font></font></font></font><br </font
<div class=3D"moz-signature"
<br <br
Get important Linux and industry-related news at: <a
href=3D"http://facebook.com/dcdata">facebook.com/dcdata</... <br
<font color=3D"#000000"><font face=3D"arial"
size=3D"2">Kind regard=
s,</font><br <br =C2=A0<font
color=3D"#000000"><font face=3D"arial"><font
size=3D"=
4">David
Wilson</font></font></font><br =C2=A0<font
color=3D"#000000"><font face=3D"arial"><font
size=3D"=
1">CNS,CLS,
LINUX+, CLA, DCTS, LPIC3</font></font></font><br =C2=A0<font
color=3D"#0779b5"><font face=3D"arial"><font
size=3D"=
3"><b>LinuxTech
CC t/a DcData</b></font></font></font><br =C2=A0<font
color=3D"#0779b5"><font face=3D"arial"><font
size=3D"=
1">CK
number: 2001/058368/23</font></font></font><br <table border=3D"0" <tbody
<tr <td
width=3D"75"><font color=3D"#000000"><font
face=3D"aria=
l"><font
size=3D"2"><b>Website:</b></font></font></font></td=
<td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><a
class=3D"moz-txt-link-freetext" href=
=3D"http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></fon=
t></td </tr <tr
<td><font color=3D"#000000"><font
face=3D"arial"><font
size=3D"2"><b>Support:</b></font></font></font></td=
<td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2">+27(0)860-1-LINUX</font></font></font></=
td </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><b>Mobile:</b></font></font></font></td <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2">+27(0)824147413</font></font></font></td=
</tr
<tr <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><b>Tel:</b></font></font></font></td <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2">+27(0)333446100</font></font></font></td=
</tr
<tr <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2"><b>Fax:</b></font></font></font></td <td><font
color=3D"#000000"><font face=3D"arial"><font
size=3D"2">+27(0)866878971</font></font></font></td=
</tr
</tbody </table </font></div On
12/03/2012 10:36 AM, Yeela Kaplan wrote:<br
</div <blockquote
cite=3D"mid:636934950.27906854.1354523800172.JavaMail.root@redhat.com=
"
type=3D"cite" <pre
wrap=3D"">It seems that your guest file system is corrupted.
The dd approach won't help cause this is a virtual disk,
you need to fix the virtual disk inside the lv, that's why any operations o=
n the lv won't help.=20
You can try shutting down the vm (not paused, but literally down) and add t=
he guest vm disk as
a second disk to another vm.
In the second vm try to run fsck on the disk and see if it helps.
In any case before trying to manipulate the disk image please save a backup=
!!!
let me know how it goes...
----- Original Message -----
</pre <blockquote
type=3D"cite" <pre
wrap=3D"">From: "David Wilson" <a
class=3D"moz-txt-link-rfc239=
6E"
href=3D"mailto:dave@dcdata.co.za"><dave@dcdata.co.za></a To: <a class=3D"moz-txt-link-abbreviated"
href=3D"mailto:users@ovirt.org">u=
sers(a)ovirt.org</a Sent: Sunday, December 2, 2012
11:17:43 AM
Subject: [Users] VM has paused due to unknown storage error
H i everyone,
Sunday fun : )
One of our cr itical Linux VMs ran out of disk space. I cleared up
space by removing old application logs within the guest, rebooted
the g uest and was presented with the following information message
during bootup when the filesystem ext3/ 4 scan runs:
Clearing orhpaned inode .....
Shortly after this message the the guest is p aused by oVirt and the
"VM has paused due to unknown storage error" message is displayed in
the oVirt web GUI.
On the physical node itself, I fo und t he following error in
/var/log/vdsm/vd sm.log:
libvirtEventLoop:: <a class=3D"moz-txt-link-freetext"
href=3D"INFO::2012-12=
-02">INFO::2012-12-02</a
09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)
vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device
ide0-0-1 error eother
Any ideas on how to get more inf ormation on what exactly the error
is?
I've tried booting the guest into r es cue mode , run a manual fsck
on the logical v olume but each time the guest is automaticall
paused due to the "unknown storage error".
We are using a FC SAN for s torage , s o what I've decided to try is
expo rt the disk using "dd" on the physical node and t hen try
repairing it by mounting it as a loop filesystem on a nother
physical ser ver , and then use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--
Get important Linux and industry-related news at: facebook.com/dcdata
Kind regards,
David Wilson
CNS,CLS, LINUX+, CLA, DCTS, LPIC3
LinuxTech CC t/a DcData
CK number: 2001/058368/23
=09Website: =09<a class=3D"moz-txt-link-freetext"
href=3D"http://www.dcdata=
.co.za">http://www.dcdata.co.za</a
=09Support: =09+27(0)860-1-LINUX
=09Mobile: =09+27(0)824147413
=09Tel: =09+27(0)333446100
=09Fax: =09+27(0)866878971
Time to evaluate your email security provider? Watch the video and
take advantage of Mimecast=E2=80=99s first ever limited promotion.
_______________________________________________
Users mailing list
<a class=3D"moz-txt-link-abbreviated"
href=3D"mailto:Users@ovirt.org">Users=
@ovirt.org</a <a
class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.org/mailman/l=
istinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a
</pre </blockquote <pre wrap=3D""
</pre </blockquote <br =20
<br><br <p style=3D"font-family:
Verdana; font-size:8pt; color:#666666;" align=3D"j=
ustify"
Time to evaluate your email security provider? <a href=3D"http://www.mimeca=
st.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast=
=E2=80=99s first ever limited promotion.
</p </body
</html
--------------000007070806030102080207--
It seems that your guest file system is corrupted.
The dd approach won't help cause this is a virtual disk,
you need to fix the virtual disk inside the lv, that's why any operations=
on
the lv won't help.
You can try shutting down the vm (not paused, but literally down) and
add=
the guest vm disk as
a second disk to another vm.
In the second vm try to run fsck on the disk and see if it helps.
In any case before trying to manipulate the disk image please save a back=
up!!!
let me know how it goes...
----- Original Message -----
> From: "David Wilson" <dave(a)dcdata.co.za > To:
users(a)ovirt.org
> Sent: Sunday, December 2, 2012 11:17:43 AM
> Subject: [Users] VM has paused due to unknown storage error
> H i everyone,
> Sunday fun : )
> One of our cr itical Linux VMs ran out of disk space. I
cleared up
> space by removing old application logs within the guest, rebooted
> the g uest and was presented with the following information message
> during bootup when the filesystem ext3/ 4 scan runs:
> Clearing orhpaned inode .....
> Shortly after this message the the guest is p aused by oVirt and the
> "VM has paused due to unknown storage error" message is displayed in
> the oVirt web GUI.
> On the physical node itself, I fo und t he following error
in
> /var/log/vdsm/vd sm.log:
> libvirtEventLoop:: INFO::2012-12-02
> 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)
> vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device
> ide0-0-1 error eother
> Any ideas on how to get more inf ormation on what exactly
the error
> is?
> I've tried booting the guest into r es cue mode , run
a manual fsck
> on the logical v olume but each time the guest is automaticall
> paused due to the "unknown storage error".
> We are using a FC SAN for s torage , s o what I've
decided to try is
> expo rt the disk using "dd" on the physical node and t hen try
> repairing it by mounting it as a loop filesystem on a nother
> physical ser ver , and then use "dd" to import it back in again.
> Is this the right approach to take to get the system back
online?
> --
> Get important Linux and
industry-related news at: facebook.com/dcdata
> Kind regards,
> David Wilson
> CNS,CLS, LINUX+, CLA, DCTS, LPIC3
> LinuxTech CC t/a DcData
> CK number: 2001/058368/23
> =09Website: =09http://www.dcdata.co.za
> =09Support: =09+27(0)860-1-LINUX
> =09Mobile: =09+27(0)824147413
> =09Tel: =09+27(0)333446100
> =09Fax: =09+27(0)866878971
> Time
to evaluate your email security provider? Watch the video and
> take advantage of Mimecast=E2=80=99s first ever limited promotion.
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
--------------000007070806030102080207
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<html
3:37 a.m.
Glad to hear it worked out.
When oVirt started the vm it mounted the corrupt disk image that seemed fine,
but it couldn't find the OS because of the corrupted fs,
and the error caused it to pause the guest.
----- Original Message -----
From: "David Wilson" <dave(a)dcdata.co.za>
To: users(a)ovirt.org
Sent: Monday, December 3, 2012 10:55:37 AM
Subject: Re: [Users] VM has paused due to unknown storage error
H i everyone,
Some feedback on this error , and a question I hope that someone
could possibly assist with.
I managed to get the guest running again by using dd and q emu-img to
export the filesystem image and base image , convert to raw format
and run a fsck on the loop ed part ition.
When mounted as a loop filesystem, fsck easily repaired it .
Any ideas why oVirt kept pausing the guest and did no t allow fsck to
repair the filesystem?
Get important Linux and industry-related news at: facebook.com/dcdata
Kind regards,
David Wilson
CNS,CLS, LINUX+, CLA, DCTS, LPIC3
LinuxTech CC t/a DcData
CK number: 2001/058368/23
Website: http://www.dcdata.co.za
Support: +27(0)860-1-LINUX
Mobile: +27(0)824147413
Tel: +27(0)333446100
Fax: +27(0)866878971 On 12/02/2012 11:17 AM, David Wilson wrote:
H i everyone,
Sunday fun : )
One of our cr itical Linux VMs ran out of disk space. I cleared up
space by removing old application logs within the guest, rebooted
the g uest and was presented with the following information message
during bootup when the filesystem ext3/ 4 scan runs:
Clearing orhpaned inode .....
Shortly after this message the the guest is p aused by oVirt and the
"VM has paused due to unknown storage error" message is displayed in
the oVirt web GUI.
On the physical node itself, I fo und t he following error in
/var/log/vdsm/vd sm.log:
libvirtEventLoop:: INFO::2012-12-02
09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)
vmId=`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device
ide0-0-1 error eother
Any ideas on how to get more inf ormation on what exactly the error
is?
I've tried booting the guest into r es cue mode , run a manual fsck
on the logical v olume but each time the guest is automaticall
paused due to the "unknown storage error".
We are using a FC SAN for s torage , s o what I've decided to try is
expo rt the disk using "dd" on the physical node and t hen try
repairing it by mounting it as a loop filesystem on a nother
physical ser ver , and then use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--
Get important Linux and industry-related news at: facebook.com/dcdata
Kind regards,
David Wilson
CNS,CLS, LINUX+, CLA, DCTS, LPIC3
LinuxTech CC t/a DcData
CK number: 2001/058368/23
Website: http://www.dcdata.co.za
Support: +27(0)860-1-LINUX
Mobile: +27(0)824147413
Tel: +27(0)333446100
Fax: +27(0)866878971
Time to evaluate your email security provider? Watch the video and
take advantage of Mimecast’s first ever limited promotion.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
4:58 a.m.
On Mon, Dec 3, 2012 at 10:37 AM, Yeela Kaplan wrote:
Glad to hear it worked out.
When oVirt started the vm it mounted the corrupt disk image that seemed fine,
but it couldn't find the OS because of the corrupted fs,
and the error caused it to pause the guest.
But if this is the case, I think this should be considered as a bug, correct?
The VM should start and give an error itself, such as stopping at
grub, error on pivot_root, panic, ecc...
without impact on its ability to power on itself...
or not?
5:07 a.m.
On Mon, Dec 03, 2012 at 04:37:01AM -0500, Yeela Kaplan wrote:
Glad to hear it worked out.
When oVirt started the vm it mounted the corrupt disk image that seemed fine,
but it couldn't find the OS because of the corrupted fs,
and the error caused it to pause the guest.
I think we should be a bit more exact here: a VM without an installed OS
does not pause. The cause of the pause was, most probably, and attempt
to read from a corrupted qcow. When qemu fails to serve the guest with
data due to a bug in the underlying storage, qemu stops and waits for
further instructions from management. We use this feature for automatic
lv-extend (on enospace error). But here, with eother error, a human
intervension is required.
8:47 a.m.
This is a multi-part message in MIME format.
--------------040205020003020503010804
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
David Wilson:
Hi everyone,
Some feedbackon this error, and a question I hope that someone could
possibly assist with.
I managed to get the guest running again byusing dd and qemu-img to
export the filesystem image and base image, convert to raw format and
run a fsck on the looped partition.
I am happy you make that work. Can you
describe more details about how
you dd the guest image? And how did you use qemu-img to export the
filesystem image and base image.
When mounted as a loop filesystem, fsck easily repaired it.
Any ideas why oVirt kept pausing the guest and did not allow fsck to
repair the filesystem?
Get important Linux and industry-related news at: facebook.com/dcdata
<http://facebook.com/dcdata>
Kind regards,
David Wilson
CNS,CLS, LINUX+, CLA, DCTS, LPIC3
*LinuxTech CC t/a DcData*
CK number: 2001/058368/23
*Website:* http://www.dcdata.co.za
*Support:* +27(0)860-1-LINUX
*Mobile:* +27(0)824147413
*Tel:* +27(0)333446100
*Fax:* +27(0)866878971
On 12/02/2012 11:17 AM, David Wilson wrote:
> Hi everyone,
>
> Sunday fun :)
>
> One of our critical Linux VMs ran out of disk space. I cleared up
> space by removing old application logs within the guest, rebooted the
> guest and was presented with the following information message during
> bootup when the filesystem ext3/4 scan runs:
> /Clearing orhpaned inode ...../
> Shortly after this message the the guest is paused by oVirt and the
> "VM has paused due to unknown storage error" message is displayed in
> the oVirt web GUI.
>
> On the physical node itself, I found the following error
> in/var/log/vdsm/vdsm.log:
> /libvirtEventLoop::INFO::2012-12-02
> 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)
> vmId=`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device
> ide0-0-1 error eother/
>
> Any ideas on how to get more informationon what exactly the error is?
>
> I've tried booting the guest into rescue mode, run a manual fsck on
> the logical volume but each time the guest is automaticall paused due
> to the "unknown storage error".
>
> We are using a FC SAN for storage, sowhat I've decided to try is
> export the disk using "dd" on the physical node and then try
> repairing it by mounting it as a loop filesystem on another physical
> server, and then use "dd" to import it back in again.
>
> Is this the right approach to take to get the system back online?
>
>
>
> --
>
>
> Get important Linux and industry-related news at: facebook.com/dcdata
> <http://facebook.com/dcdata>
>
> Kind regards,
>
> David Wilson
> CNS,CLS, LINUX+, CLA, DCTS, LPIC3
> *LinuxTech CC t/a DcData*
> CK number: 2001/058368/23
> *Website:* http://www.dcdata.co.za
> *Support:* +27(0)860-1-LINUX
> *Mobile:* +27(0)824147413
> *Tel:* +27(0)333446100
> *Fax:* +27(0)866878971
>
Time to evaluate your email security provider? Watch the video
<http://www.mimecast.com/emailsecurity/> and take advantage of
Mimecast's first ever limited promotion.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
---
?? Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming(a)cn.ibm.com or
shuming(a)linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193,
PRC
--------------040205020003020503010804
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">David Wilson:<br>
</div>
<blockquote cite="mid:50BC6909.7050602@dcdata.co.za"
type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix"><font
size="-1">H<font size="-1">i</font>
everyone,<br>
<br>
<font size="-1"><font size="-1">Some
feedback<font size="-1">
on this error<font size="-1">, and a <font
size="-1">question
I hope that someone could possibly assist
with.</font></font><br>
<font size="-1">I managed to <font
size="-1">get the
guest running again by<font size="-1"> using dd
<font
size="-1">and q<font
size="-1">emu-img to export
the filesystem image and base image<font
size="-1">, convert to raw format and run a
fsck on the <font size="-1">loop<font
size="-1">ed part<font
size="-1">ition.<br>
</font></font></font></font></font></font></font></font></font></font></font></font></font></div>
</blockquote>
I am happy you make that work. Can you describe more details
about
how you dd the guest image? And how did you use qemu-img to export
the filesystem image and base image.<br>
<br>
<blockquote cite="mid:50BC6909.7050602@dcdata.co.za"
type="cite">
<div class="moz-cite-prefix"><font
size="-1"><font size="-1"><font
size="-1"><font size="-1"><font
size="-1"><font size="-1"><font
size="-1"><font size="-1"><font
size="-1"><font
size="-1"><font
size="-1"><font size="-1"><font
size="-1"> <font
size="-1">When
mounted as a loop filesystem, fsck
easily repaired <font
size="-1">it<font
size="-1">.</font></font><br>
<br>
</font><font size="-1">Any ideas
why
oVirt kept pausing the guest and did
no<font size="-1">t allow fsck to
<font
size="-1">repair the <font
size="-1">filesystem?<br>
<br>
</font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font><br>
</font>
<div class="moz-signature"> <br>
<br>
Get important Linux and industry-related news at: <a
moz-do-not-send="true"
href="http://facebook.com/dcdata">facebook.com/dcdata</a&...
<br>
<font color="#000000"><font face="arial"
size="2">Kind
regards,</font><br>
<br>
<font color="#000000"><font
face="arial"><font size="4">David
Wilson</font></font></font><br>
<font color="#000000"><font
face="arial"><font size="1">CNS,CLS,
LINUX+, CLA, DCTS,
LPIC3</font></font></font><br>
<font color="#0779b5"><font
face="arial"><font size="3"><b>LinuxTech
CC t/a
DcData</b></font></font></font><br>
<font color="#0779b5"><font
face="arial"><font size="1">CK
number: 2001/058368/23</font></font></font><br>
<table border="0">
<tbody>
<tr>
<td width="75"><font
color="#000000"><font
face="arial"><font
size="2"><b>Website:</b></font></font></font></td>
<td><font color="#000000"><font
face="arial"><font
size="2"><a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></font></td>
</tr>
<tr>
<td><font color="#000000"><font
face="arial"><font
size="2"><b>Support:</b></font></font></font></td>
<td><font color="#000000"><font
face="arial"><font
size="2">+27(0)860-1-LINUX</font></font></font></td>
</tr>
<tr>
<td><font color="#000000"><font
face="arial"><font
size="2"><b>Mobile:</b></font></font></font></td>
<td><font color="#000000"><font
face="arial"><font
size="2">+27(0)824147413</font></font></font></td>
</tr>
<tr>
<td><font color="#000000"><font
face="arial"><font
size="2"><b>Tel:</b></font></font></font></td>
<td><font color="#000000"><font
face="arial"><font
size="2">+27(0)333446100</font></font></font></td>
</tr>
<tr>
<td><font color="#000000"><font
face="arial"><font
size="2"><b>Fax:</b></font></font></font></td>
<td><font color="#000000"><font
face="arial"><font
size="2">+27(0)866878971</font></font></font></td>
</tr>
</tbody>
</table>
</font></div>
On 12/02/2012 11:17 AM, David Wilson wrote:<br>
</div>
<blockquote cite="mid:50BB1CB7.8020007@dcdata.co.za"
type="cite">
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<font size="-1"><font size="-1">H</font>i
everyone,<br>
<br>
<font size="-1"><font size="-1">Sunday fun
:<font size="-1">)<br>
<br>
</font>One of our cr<font size="-1">itical <font
size="-1">Linux VMs</font> ran out of
</font>disk
space. I cleared up space by removing old application logs
within the guest, rebooted the g<font size="-1">uest and
was presented with the following information message
during bootup when the filesystem <font
size="-1">ext3/<font
size="-1">4</font></font> scan
runs:<br>
<font size="-1"><i>Clearing orhpaned inode
.....</i><br>
<font size="-1"><font size="-1">Shortly
after this
message the the guest is p<font size="-1">aused by
oVirt and the "VM has paused due to unknown
storage error" message is displayed in the <font
size="-1">oVirt web <font
size="-1">GUI.<br>
<br>
<font size="-1">On the physical node itself,
<font size="-1">I fo<font
size="-1">und t<font
size="-1">he following error in<font
size="-1"> /var/log/vdsm/vd<font
size="-1">sm.log:<br>
<font
size="-1"><i>libvirtEventLoop::<a
moz-do-not-send="true"
class="moz-txt-link-freetext"
href="INFO::2012-12-02">INFO::2012-12-02</a>
09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)
vmId=`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal
vm stop device ide0-0-1
error eother</i><br>
<br>
<font size="-1">Any ideas on
how to get more inf<font
size="-1">ormation<font
size="-1"> on what
exactly the <font
size="-1">error
is?<br>
</font></font></font><br>
<font size="-1">I've
<font
size="-1">tried booting
the guest <font
size="-1">into r<font
size="-1">es<font
size="-1">cue
mode<font
size="-1">, run
a manual fsck <font
size="-1">on
the logical v<font
size="-1">olume
but
each</font></font></font></font></font></font></font></font>
time the guest is
automaticall paused due to
the "unknown storage
error".<br>
<br>
We are using a FC SAN for s<font
size="-1">torage</font></font>,
s<font size="-1">o<font
size="-1"> what I've
decided to</font></font>
try is expo<font size="-1">rt
the disk using "dd" <font
size="-1">on the physical
node </font>and t<font
size="-1">hen try
repairing it <font
size="-1">by <font
size="-1">mounting it
as a loop filesystem
on <font
size="-1">a<font
size="-1">nother
physical</font></font></font></font></font></font>
ser<font size="-1">ver<font
size="-1">, and then use
"dd" to import it back in
again.</font><br>
<br>
<font size="-1">Is this the
right approach to take to
get the system back
online?<br>
<br>
</font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font><br>
<br>
</font>
<div class="moz-signature">-- <br>
<br>
<br>
Get important Linux and industry-related news at: <a
moz-do-not-send="true"
href="http://facebook.com/dcdata">facebook.com/dcdata</a&...
<br>
<font color="#000000"><font face="arial"
size="2">Kind
regards,</font><br>
<br>
<font color="#000000"><font
face="arial"><font size="4">David
Wilson</font></font></font><br>
<font color="#000000"><font
face="arial"><font size="1">CNS,CLS,
LINUX+, CLA, DCTS,
LPIC3</font></font></font><br>
<font color="#0779b5"><font
face="arial"><font size="3"><b>LinuxTech
CC t/a
DcData</b></font></font></font><br>
<font color="#0779b5"><font
face="arial"><font size="1">CK
number: 2001/058368/23</font></font></font><br>
<table border="0">
<tbody>
<tr>
<td width="75"><font
color="#000000"><font
face="arial"><font
size="2"><b>Website:</b></font></font></font></td>
<td><font color="#000000"><font
face="arial"><font
size="2"><a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></font></td>
</tr>
<tr>
<td><font color="#000000"><font
face="arial"><font
size="2"><b>Support:</b></font></font></font></td>
<td><font color="#000000"><font
face="arial"><font
size="2">+27(0)860-1-LINUX</font></font></font></td>
</tr>
<tr>
<td><font color="#000000"><font
face="arial"><font
size="2"><b>Mobile:</b></font></font></font></td>
<td><font color="#000000"><font
face="arial"><font
size="2">+27(0)824147413</font></font></font></td>
</tr>
<tr>
<td><font color="#000000"><font
face="arial"><font
size="2"><b>Tel:</b></font></font></font></td>
<td><font color="#000000"><font
face="arial"><font
size="2">+27(0)333446100</font></font></font></td>
</tr>
<tr>
<td><font color="#000000"><font
face="arial"><font
size="2"><b>Fax:</b></font></font></font></td>
<td><font color="#000000"><font
face="arial"><font
size="2">+27(0)866878971</font></font></font></td>
</tr>
</tbody>
</table>
</font></div>
</blockquote>
<br>
<br>
<br>
<p style="font-family: Verdana; font-size:8pt; color:#666666;"
align="justify">
Time to evaluate your email security provider? <a
moz-do-not-send="true"
href="http://www.mimecast.com/emailsecurity/">Watch the
video</a>
and take advantage of Mimecast’s first ever limited promotion.
</p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626 Tieline: 9051626 E-mail: <a
class="moz-txt-link-abbreviated"
href="mailto:shuming@cn.ibm.com">shuming@cn.ibm.com</a> or <a
class="moz-txt-link-abbreviated"
href="mailto:shuming@linux.vnet.ibm.com">shuming@linux.vnet.ibm.com</a>
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193,
PRC</pre>
</body>
</html>
--------------040205020003020503010804--
9:51 a.m.
This is a multi-part message in MIME format.
--------------050904000702030906090307
Content-Type: text/plain; charset=WINDOWS-1252; format=flowed
Content-Transfer-Encoding: quoted-printable
Hi everyone,
Thank you for your responses.
Yeela:
/>When oVirt started the vm it mounted the corrupt disk image that=20
seemed fine, but it couldn't find the OS because of the corrupted fs,=20
and the error caused it to pause the guest./
To clear things up it was the /var mount that had the corrupted=20
filesystem, not the guest's root filesystem.
oVirt still continued to pause the guest even when I had booted the=20
guest off a CentOS DVD ISO, run rescue mode and manually activated the=20
/var logical volume. In this case I did not activate or mount the guests=20
root filesystem and only activated the guests /var filesystem so that I=20
could fsck it. fsck would run for around 5-10 minutes with the message=20
"Deleting orphaned inode......" and then oVirt would simply pause the=20
entire guest.
The only information I could find was on the physical host's vdsm.log,=20
which specified the following:
/libvirtEventLoop::INFO::2012-12-02=20
09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)=20
vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device=20
ide0-0-1 error eother
/Perhaps there was another log I should have examined to see if more=20
information was provided about why oVirt was pausing the guest?
//
Shu:
This is what I did to dd the images off, and to work around the problem:
1.) On the physical host: Created an NFS mount to another temporary=20
Linux system that had sufficient storage for the 500GB filesystem
2.) On the physical host: Used 'dd' to dump the /var filesystem's=20
logical volume to an image filevia NFS on the temporary Linux system.
3.) On the temporary Linux system that now contained the filesystem=20
image file, I ran "qemu-img info" and noticed that thefilesystem image=20
was qcow2 type and specified a backing file.
4.) On the physical host: Used 'dd' to dump the logical volume specified=20
as a backing file, to an image file via NFS on the temporary Linux system.
5.) On the temporary Linux system: Used 'qemu-img rebase' to change the=20
backing file to the local copyof the backing file image.
6.) On the temporary Linux system: Used 'qemu-img commit' to commit the=20
changes stored in the filesystem image file to the backing file image.
7.) On the temporary Linux system: Used 'qemu-img convert' to convert=20
the backing file image to raw format.
8.) Onthe temporary Linux system: Used 'losetup', 'kpart'and
'fsck' to=20
repair the backing file image. Fsck displayed the same 'Deleting=20
orphaned inode....' message but managed to continue and completed ok.
9.) On the temporary Linux system: Mounted the loop filesystem and=20
confirmed that the data was intact and was current.
10.) In the oVirt GUI: Deactivated the faulty Virtual Disk attached to=20
the guest.
11.) In the oVirt GUI: Created a new 'preallocated' Virtual Disk of=20
sufficient sizefor the guest.
12.) On the physical host: Used 'dd' to upload the raw backing file=20
image from (7) to the new logical volume.
13.) I then configured the guest to boot from the CentOS DVD ISO into=20
rescue modeto confirm that the logical volume for the guest's /var=20
filesystem was accessible and mountable.
14.) Reconfigured the guest to bootfrom it's primary Virtual Disk and=20
started up the guest.
//
Get important Linux and industry-related news at: facebook.com/dcdata=20
<http://facebook.com/dcdata
Kind regards,
David Wilson
CNS,CLS, LINUX+, CLA, DCTS, LPIC3
*LinuxTech CC t/a DcData*
CK number: 2001/058368/23
*Website:* =09http://www.dcdata.co.za
*Support:* =09+27(0)860-1-LINUX
*Mobile:* =09+27(0)824147413
*Tel:* =09+27(0)333446100
*Fax:* =09+27(0)866878971
On 12/03/2012 01:07 PM, Dan Kenigsberg wrote:
<head <meta content=3D"text/html;
charset=3Dwindows-1252"
http-equiv=3D"Content-Type"
</head <body
bgcolor=3D"#FFFFFF" text=3D"#000000"
<div class=3D"moz-cite-prefix"><font size=3D"-1">Hi
everyone,<br <br <font size=3D"-1">Thank you for your
responses.<br <br <font
size=3D"-1">Yeela:</font></font></font><br <small><i>>When oVirt started the vm
it mounted the corrupt
disk image that seemed fine, but it couldn't find the OS
because of the corrupted fs,
and the error caused it to pause the guest.</i><br <br
To clear things up it was the /var mount that had the corrupted
filesystem, not the guest's root filesystem.<br oVirt still continued to pause the guest even when I
had booted
the guest off a CentOS DVD ISO, run rescue mode and manually
activated the /var logical volume. In this case I did not
activate or mount the guests root filesystem and only activated
the guests /var filesystem so that I could fsck it. fsck would
run for around 5-10 minutes with the message "Deleting orphaned
inode......" and then oVirt would simply pause the entire guest.<br=
The only information I could find was on the physical
host's
vdsm.log, which specified the following:<br
</small><font size=3D"-1"><font size=3D"-1"><font
size=3D"-1"
09:05:50,296::libvirtvm::1965::vm=
.Vm::(_onAbnormalStop)
vmId=3D`23b9212c-1e25-4003-aa18-b=
1e819bf6bb1`::abnormal
vm stop device ide0-0-1 error
eother<br
<br
</i><font size=3D"-1" <br
</font></font></font></fo=
nt></font></font><i></i <br Shu:<br <font
size=3D"-1">This is what
I did to dd the images
off, and to <font
size=3D"-1">work around
the problem<font
size=3D"-1">:<br
</font></font
<font size=3D"-1">C</fo=
nt></font>reated
an NFS mount to
another temporary
Linux system that had
sufficient storage for
the 500GB filesystem<br <font
size=3D"-1">2.<font
size=3D"-1">)
O<font
size=3D"-1">n</font=
the
physical host<fon=
t
size=3D"-1">:
<font
size=3D"-1">U</fo=
nt></font>sed
'dd' to <font
size=3D"-1">dump
the /var
filesystem's
logical vol<font
size=3D"-1">ume
to an image
file<font
size=3D"-1">
<fon=
t
size=3D"-1">via
NFS on t<font
size=3D"-1">he
temporar<font
size=3D"-1">y</fo=
nt Linux
<font
size=3D"-1">syste=
m</font></font></font>.</font></font></font></font></font><br <font
size=3D"-1">3.) On
the temporary Linux
s<font
size=3D"-1">yste=
m
that now contained
the filesystem
image file, I ran
"qemu-img info</font>=
</font>"
and noticed that the<font
size=3D"-1"> fi<font
size=3D"-1">lesystem<=
/font image was
<font
size=3D"-1">qc<font
size=3D"-1">ow2
<fo=
nt
size=3D"-1">type<=
/font
and</font></font
specified a ba<font
size=3D"-1">cking
file.<br <font
size=3D"-1">4.)
On t<font
size=3D"-1">he
physical host<fon=
t
size=3D"-1">:
Used <font
size=3D"-1">'dd'
to dump <font
size=3D"-1">the
lo<font
size=3D"-1">gical
volume
specified as a
backing file,
to an image
file </font></fon=
t></font></font></font></font></font></font></font></font></font></font></f=
ont></font></font></font></font></font></font></font></font></font></font><=
/font></font></font></font></font></font></font Linux
<font
size=3D"-1">syste=
m.<br
<font
size=3D"-1">5.<fo=
nt
size=3D"-1">) On
the <font
size=3D"-1">tempo=
rary
Linux system:
Used '<font
size=3D"-1">qemu-=
img
reba<font
size=3D"-1">se</f=
ont>'
to change the
backing file <fon=
t
size=3D"-1">to
the local copy<fo=
nt
size=3D"-1"> of
the back<font
size=3D"-1">ing
file image.</font=
></font></font></font></font></font></font><br
<font
size=3D"-1">6.)
On the
temporary
Linux system:
Used 'qemu-img
commit' to
commit the
changes stored
in the
filesystem
image file to
the backing
file image.<br
<font
size=3D"-1">7.)
O<font
size=3D"-1">n
the temporary
Linux sy<font
size=3D"-1">ste<f=
ont
size=3D"-1">m<fon=
t
size=3D"-1">:
U<f=
ont
size=3D"-1">sed
'qemu-img
convert' to
convert the
backing file
image to raw
format.<br
<font
size=3D"-1">8.)
On<font
size=3D"-1">
<fon=
t
size=3D"-1">the
<font
size=3D"-1">tempo=
rary
Linux system:
<font
size=3D"-1">Used
'losetup', <font
size=3D"-1">'kpar=
t'<font
size=3D"-1" to
<font
size=3D"-1">repai=
r
the backing
file image.
Fsck displayed
t<font
size=3D"-1">he
same 'Del<font
size=3D"-1">eting
<font
size=3D"-1">orpha=
ned
in<font
size=3D"-1">ode<f=
ont
size=3D"-1">....'
<font
size=3D"-1">messa=
ge</font></font></font></font></font></font></font
<font
size=3D"-1">but
managed to <font
size=3D"-1">con<f=
ont
size=3D"-1">tinue
and completed
ok.<br
<font
size=3D"-1">9.)
O<font
size=3D"-1">n
the tempo<font
size=3D"-1">rary
Linux system:
<font
size=3D"-1">Mo<fo=
nt
size=3D"-1">unted
the loop
filesystem and
confirmed that
<font
size=3D"-1">the
<font
size=3D"-1">data
was intact and
was current.</fon=
t></font></font></font></font></font></font><br
</font></font></f=
ont
<font
size=3D"-1"
<font
size=3D"-1">1<fon=
t
size=3D"-1">2</fo=
nt>.)
O<font
size=3D"-1">n
the physical
host: Used <font
size=3D"-1">'dd'
to upload the
<font
size=3D"-1">raw
ba<font
size=3D"-1">cking
file image
from (7) to
the new lo<font
size=3D"-1">gical
volume.</font></f=
ont></font></font></font></font><br
<font
size=3D"-1">1<fon=
t
size=3D"-1">3</fo=
nt>.)
<font
size=3D"-1">I
then conf<font
size=3D"-1">igure=
d
the guest to
boot from the
CentOS <font
size=3D"-1">D<fon=
t
size=3D"-1">VD
ISO into res<font
size=3D"-1">cue
mode<font
size=3D"-1"> to
confirm that
the <font
size=3D"-1">lo<fo=
nt
size=3D"-1">gical
<font
size=3D"-1">volum=
e
<font
size=3D"-1"
was accessi<font
size=3D"-1">ble
and mountable.<br=
<font
size=3D"-1">1<fon=
t
size=3D"-1">4</fo=
nt>.)
<font
size=3D"-1">Recon=
figured
the guest to
boo<font
size=3D"-1">t<fon=
t
size=3D"-1" from
it's <font
size=3D"-1">prima=
ry
<font
size=3D"-1">Virtu=
al
Disk</font></font=
and
sta<font
size=3D"-1">rted
up the guest.</fo=
nt></font></font></font></font><br
</font></font></f=
ont></font></font></font></font></font></font></font></font></font></font><=
/font></font></font></font></font></font></font></font></font></font></font=
><br
</font><br
</font></font></f=
ont></font></font></font></font></font></font></font></font></font></font><=
/font></font></font></font></font></font></font></font></font></font></font=
<br
<br <br
<br <div class=3D"moz-signature" <br
<br Get important Linux and industry-related news at:
<a
href=3D"http://facebook.com/dcdata">facebook.com/dcdata</... <br
<font color=3D"#000000"><font face=3D"arial"
size=3D"2">Kind regard=
s,</font><br <br =A0<font color=3D"#000000"><font
face=3D"arial"><font size=3D"4">=
David
Wilson</font></font></font><br =A0<font color=3D"#000000"><font
face=3D"arial"><font size=3D"1">=
CNS,CLS,
LINUX+, CLA, DCTS, LPIC3</font></font></font><br =A0<font color=3D"#0779b5"><font
face=3D"arial"><font size=3D"3">=
<b>LinuxTech
CC t/a DcData</b></font></font></font><br =A0<font color=3D"#0779b5"><font
face=3D"arial"><font size=3D"1">=
CK
number: 2001/058368/23</font></font></font><br <table border=3D"0" <tbody
<tr <td
width=3D"75"><font color=3D"#000000"><font
face=3D"aria=
l" <td><font
color=3D"#000000"><font face=3D"arial" </tr <tr
<td><font color=3D"#000000"><font
face=3D"arial" <td><font
color=3D"#000000"><font face=3D"arial" </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial" <td><font
color=3D"#000000"><font face=3D"arial" </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial" <td><font
color=3D"#000000"><font face=3D"arial" </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial" <td><font
color=3D"#000000"><font face=3D"arial" </tr
</tbody </table </font></div On
12/03/2012 01:07 PM, Dan Kenigsberg wrote:<br
</div <blockquote
cite=3D"mid:20121203110737.GD20719@redhat.com" type=3D"cite=
" <pre wrap=3D"">On Mon, Dec 03, 2012 at
04:37:01AM -0500, Yeela Kaplan=
wrote:
</pre <blockquote
type=3D"cite" <pre
wrap=3D"">Glad to hear it worked out.
When oVirt started the vm it mounted the corrupt disk image that seemed fin=
e, =20
but it couldn't find the OS because of the corrupted fs,
and the error caused it to pause the guest.
</pre </blockquote <pre wrap=3D"" I think
we should be a bit more exact here: a VM without an installed OS
does not pause. The cause of the pause was, most probably, and attempt
to read from a corrupted qcow. When qemu fails to serve the guest with
data due to a bug in the underlying storage, qemu stops and waits for
further instructions from management. We use this feature for automatic
lv-extend (on enospace error). But here, with eother error, a human
intervension is required.
</pre </blockquote <br =20
<br><br <p style=3D"font-family:
Verdana; font-size:8pt; color:#666666;" align=3D"j=
ustify"
Time to evaluate your email security provider? <a href=3D"http://www.mimeca=
st.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast=
=92s first ever limited promotion.
</p </body
</html
--------------050904000702030906090307--
On Mon, Dec 03, 2012 at 04:37:01AM -0500, Yeela Kaplan wrote:
> Glad to hear it worked out.
> When oVirt started the vm it mounted the corrupt disk
image that seemed =
fine,
> but it couldn't find the OS because of the corrupted fs,
> and the error caused it to pause the guest.
I think we should be a bit more exact here: a VM without an installed OS
does not pause. The cause of the pause was, most probably, and attempt
to read from a corrupted qcow. When qemu fails to serve the guest with
data due to a bug in the underlying storage, qemu stops and waits for
further instructions from management. We use this feature for automatic
lv-extend (on enospace error). But here, with eother error, a human
intervension is required.
--------------050904000702030906090307
Content-Type: text/html; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable
<html<font
size=3D"-1"><font size=3D"-1"><font
size=3D"-1"><font size=3D=
"-1"<font
size=3D"-1"><font size=3D"-1"><font
size=3D"-1"<font
size=3D"-1"><font size=3D"-1"><font
size=3D"-1"=
<font
size=3D"-1"><font size=3D"-1"<font
size=3D"-1"><font
size=3D"-1"><i>libv=
irtEventLoop::<a
class=3D"moz-txt-link-freetext"
href=3D"INFO::2012-12-02">INFO::2012-12-02<=
/a<font
size=3D"-1">Perh<font
size=3D"-1">aps there was
another log I should have
examined to see if more i<fon=
t
size=3D"-1">nforma<font
size=3D"-1">tion was
provided about wh<font
size=3D"-1">y oVirt
was pausing the
guest?<br<font
size=3D"-1"><font
size=3D"-1"><br=
<font
size=3D"-1">1.) <font
size=3D"-1">O<font
size=3D"-1">n the
physical host<font
size=3D"-1">:</font<font
size=3D"-1"><font
size=3D"-1"><font size=3D"-1"><font
size=3D"-1"><=
font
size=3D"-1"><font size=3D"-1"><font
size=3D"-1"<font
size=3D"-1"><font size=3D"-1"><font
size=3D"-1"<font
size=3D"-1"><font size=3D"-1"><font
size=3D"-1"=
<font
size=3D"-1"><font size=3D"-1"<font
size=3D"-1"><font
size=3D"-1"<font
size=3D"-1"><font size=3D"-1"><fo=
nt
size=3D"-1"><font
size=3D"-1"=
<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1"<font
size=3D"-1">via
NFS on t<font
size=3D"-1">he
temporar<font
size=3D"-1">y</fo=
nt<font
size=3D"-1"> and
</font>'fsck'</fo=
nt></font></font></font></font></font></font<font
size=3D"-1"<font
size=3D"-1">10</f=
ont>.)
I<font
size=3D"-1">n
<fo=
nt
size=3D"-1">the
oVirt GUI:
Deactivated
the faulty
Virtual <font
size=3D"-1">D</fo=
nt>isk
attached to
the guest.</font>=
</font></font><br<font
size=3D"-1">11</f=
ont>.)
In the oVirt
GUI: Created a
new
'preallocated<fon=
t
size=3D"-1">'
<fo=
nt
size=3D"-1">Virtu=
al
Disk <font
size=3D"-1">of
sufficient
size<font
size=3D"-1"> for
the guest.<br<font
size=3D"-1">for
th<font
size=3D"-1">e
<fo=
nt
size=3D"-1"<font
size=3D"-1">guest=
<font
size=3D"-1">'s
/var </font></fon=
t>filesystem</font></font</font></font></font></font></font></font></font></font></font></font></fo=
nt></font></font></font></font></font></font></font></font><i></i></font></=
font></font></font></font></font></font></font></font></font></font></font>=
</font></font></font></font></font><br<font
size=3D"2"><b>Website:</b></font></font></font></td=
<font
size=3D"2"><a
class=3D"moz-txt-link-freetext" href=
=3D"http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></fon=
t></td<font
size=3D"2"><b>Support:</b></font></font></font></td=
<font
size=3D"2">+27(0)860-1-LINUX</font></font></font></=
td<font
size=3D"2"><b>Mobile:</b></font></font></font></td<font
size=3D"2">+27(0)824147413</font></font></font></td=
<font
size=3D"2"><b>Tel:</b></font></font></font></td<font
size=3D"2">+27(0)333446100</font></font></font></td=
<font
size=3D"2"><b>Fax:</b></font></font></font></td<font
size=3D"2">+27(0)866878971</font></font></font></td=
4:01 a.m.
Hi David,
You can also check the libvirt log.
Please send me the vdsm and libvirt logs so I can look at it too.
thanks.
----- Original Message -----
From: "David Wilson" <dave(a)dcdata.co.za>
To: "Dan Kenigsberg" <danken(a)redhat.com>
Cc: "Yeela Kaplan" <ykaplan(a)redhat.com>, users(a)ovirt.org
Sent: Monday, December 3, 2012 5:51:26 PM
Subject: Re: [Users] VM has paused due to unknown storage error
Hi everyone,
Thank you for your responses.
Yeela:
>When oVirt started the vm it mounted the corrupt disk image that
>seemed fine, but it couldn't find the OS because of the corrupted
>fs, and the error caused it to pause the guest.
To clear things up it was the /var mount that had the corrupted
filesystem, not the guest's root filesystem.
oVirt still continued to pause the guest even when I had booted the
guest off a CentOS DVD ISO, run rescue mode and manually activated
the /var logical volume. In this case I did not activate or mount
the guests root filesystem and only activated the guests /var
filesystem so that I could fsck it. fsck would run for around 5-10
minutes with the message "Deleting orphaned inode......" and then
oVirt would simply pause the entire guest.
The only information I could find was on the physical host's
vdsm.log, which specified the following:
libvirtEventLoop:: INFO::2012-12-02
09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)
vmId=`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device
ide0-0-1 error eother
Perh aps there was another log I should have examined to see if more
i nforma tion was provided about wh y oVirt was pausing the guest?
Shu:
This is what I did to dd the images off, and to work around the
problem :
1.) O n the physical host : C reated an NFS mount to another
temporary Linux system that had sufficient storage for the 500GB
filesystem
2. ) O n the physical host : U sed 'dd' to dump the /var filesystem's
logical vol ume to an image file via NFS on t he temporar y Linux
system .
3.) On the temporary Linux s ystem that now contained the filesystem
image file, I ran "qemu-img info " and noticed that the fi lesystem
image was qc ow2 type and specified a ba cking file.
4.) On t he physical host : Used 'dd' to dump the lo gical volume
specified as a backing file, to an image file via NFS on t he
temporar y Linux system.
5. ) On the temporary Linux system: Used ' qemu-img reba se ' to
change the backing file to the local copy of the back ing file
image.
6.) On the temporary Linux system: Used 'qemu-img commit' to commit
the changes stored in the filesystem image file to the backing file
image.
7.) O n the temporary Linux sy ste m : U sed 'qemu-img convert' to
convert the backing file image to raw format.
8.) On the temporary Linux system: Used 'losetup', 'kpart' and
'fsck'
to repair the backing file image. Fsck displayed t he same 'Del
eting orphaned in ode ....' message but managed to con tinue and
completed ok.
9.) O n the tempo rary Linux system: Mo unted the loop filesystem and
confirmed that the data was intact and was current.
10 .) I n the oVirt GUI: Deactivated the faulty Virtual D isk
attached to the guest.
11 .) In the oVirt GUI: Created a new 'preallocated ' Virtual Disk of
sufficient size for the guest.
1 2 .) O n the physical host: Used 'dd' to upload the raw ba cking
file image from (7) to the new lo gical volume.
1 3 .) I then conf igured the guest to boot from the CentOS D VD ISO
into res cue mode to confirm that the lo gical volume for th e guest
's /var filesystem was accessi ble and mountable.
1 4 .) Reconfigured the guest to boo t from it's primary Virtual Disk
and sta rted up the guest.
Get important Linux and industry-related news at: facebook.com/dcdata
Kind regards,
David Wilson
CNS,CLS, LINUX+, CLA, DCTS, LPIC3
LinuxTech CC t/a DcData
CK number: 2001/058368/23
Website: http://www.dcdata.co.za
Support: +27(0)860-1-LINUX
Mobile: +27(0)824147413
Tel: +27(0)333446100
Fax: +27(0)866878971 On 12/03/2012 01:07 PM, Dan Kenigsberg wrote:
On Mon, Dec 03, 2012 at 04:37:01AM -0500, Yeela Kaplan wrote:
Glad to hear it worked out.
When oVirt started the vm it mounted the corrupt disk image that
seemed fine,
but it couldn't find the OS because of the corrupted fs,
and the error caused it to pause the guest. I think we should be a
bit more exact here: a VM without an installed OS
does not pause. The cause of the pause was, most probably, and
attempt
to read from a corrupted qcow. When qemu fails to serve the guest
with
data due to a bug in the underlying storage, qemu stops and waits for
further instructions from management. We use this feature for
automatic
lv-extend (on enospace error). But here, with eother error, a human
intervension is required.
Time to evaluate your email security provider? Watch the video and
take advantage of Mimecast’s first ever limited promotion.
4153
days inactive
4156
days old
9 comments
5 participants
participants (5)
-
Dan Kenigsberg -
David Wilson -
Gianluca Cecchi -
Shu Ming -
Yeela Kaplan