3:17 a.m.
This is a multi-part message in MIME format.
--------------030500050202090506010300
Content-Type: text/plain; charset=WINDOWS-1252; format=flowed
Content-Transfer-Encoding: quoted-printable
Hi everyone,
Sunday fun :)
One of our critical Linux VMs ran out of disk space. I cleared up space=20
by removing old application logs within the guest, rebooted the guest=20
and was presented with the following information message during bootup=20
when the filesystem ext3/4 scan runs:
/Clearing orhpaned inode ...../
Shortly after this message the the guest is paused by oVirt and the "VM=20
has paused due to unknown storage error" message is displayed in the=20
oVirt web GUI.
On the physical node itself, I found the following error=20
in/var/log/vdsm/vdsm.log:
/libvirtEventLoop::INFO::2012-12-02=20
09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)=20
vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device=20
ide0-0-1 error eother/
Any ideas on how to get more informationon what exactly the error is?
I've tried booting the guest into rescue mode, run a manual fsck on the=20
logical volume but each time the guest is automaticall paused due to the=20
"unknown storage error".
We are using a FC SAN for storage, sowhat I've decided to try is export=20
the disk using "dd" on the physical node and then try repairing it by=20
mounting it as a loop filesystem on another physical server, and then=20
use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--=20
Get important Linux and industry-related news at: facebook.com/dcdata=20
<http://facebook.com/dcdata
Kind regards,
David Wilson
CNS,CLS, LINUX+, CLA, DCTS, LPIC3
*LinuxTech CC t/a DcData*
CK number: 2001/058368/23
*Website:* =09http://www.dcdata.co.za
*Support:* =09+27(0)860-1-LINUX
*Mobile:* =09+27(0)824147413
*Tel:* =09+27(0)333446100
*Fax:* =09+27(0)866878971
--------------030500050202090506010300
Content-Type: text/html; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable
<html <head
<meta http-equiv=3D"content-type" content=3D"text/html;
charset=3DISO-8=
859-1" </head <body bgcolor=3D"#FFFFFF"
text=3D"#000000"
<font size=3D"-1"><font size=3D"-1">H</font>i
everyone,<br <br <font size=3D"-1"><font
size=3D"-1">Sunday fun :<font size=3D"-1">)<b=
r <br
</font>One of our cr<font size=3D"-1">itical <font
size=3D"-1">Li=
nux
VMs</font> ran out of </font>disk space. I cleared up
space by removing old application logs within the guest,
rebooted the g<font size=3D"-1">uest and was presented with the
following information message during bootup when the
filesystem <font size=3D"-1">ext3/<font
size=3D"-1">4</font></f=
ont scan runs:<br
<font size=3D"-1"><i>Clearing orhpaned inode
.....</i><br <font
size=3D"-1"><font size=3D"-1">Shortly after this messag=
e
the the guest is p<font size=3D"-1">aused by oVirt and
the "VM has paused due to unknown storage error"
message is displayed in the <font size=3D"-1">oVirt
web <font size=3D"-1">GUI.<br <br
<font size=3D"-1">On the physical node itself, <fon=
t
size=3D"-1">I fo<font
size=3D"-1">und t<font
size=3D"-1">he following error in<font
size=3D"-1"> /var/log/vdsm/vd<font
size=3D"-1">sm.log:<br <font
size=3D"-1"><i>libvirtEventLoop::=
<a class=3D"moz-txt-link-freetext"
href=3D"INFO::2012-12-02">INFO::2012-12-=
02</a
09:05:50,296::libvirtvm::1965::vm.V=
m::(_onAbnormalStop)
vmId=3D`23b9212c-1e25-4003-aa18-b1e=
819bf6bb1`::abnormal
vm stop device ide0-0-1 error
eother</i><br <br <font
size=3D"-1">Any ideas on how
to get more inf<font
size=3D"-1">or=
mation<font
size=3D"-1"> on what exactly
the <font size=3D"-1">error
is?<br
</font></font></font><br <font
size=3D"-1">I've <font
size=3D"-1">tried booting the
guest <font size=3D"-1">into
r<font
size=3D"-1">es<font
size=3D"-1">cue
mode<font
size=3D"-1">, run a
manual fsck <font
size=3D"-1">on the
logical v<font
size=3D"-1">olume
but each</font></fo=
nt></font></font></font></font></font></font time the guest is
automaticall
paused due to the "unknown
storage error".<br <br We are using a FC SAN
for s<font
size=3D"-1">torage</font></font>,
s<font size=3D"-1">o<font
size=3D"-1"=
what I've decided
to</font></font=
try is expo<font
size=3D"-1">rt the
disk using "dd" <font
size=3D"-1">o=
n
the physical node </font>and
t<font size=3D"-1">hen try
repairing it <font
size=3D"-1">by
<font size=3D"-1">mounting it
as a loop filesystem on <font
size=3D"-1">a<font
size=3D"-1">nother
physical</font></font></f=
ont></font></font></font
ser<font size=3D"-1">ver<font
size=3D"-1">, and then use
"dd"
to import it back in again.</font=
><br
<br
<font size=3D"-1">Is this the
right approach to take to get
the system back online?<br <br
</font></font></font></font></font>=
</font></font></font></font></font></font></font></font></font></font></fon=
t></font></font><br
<br </font
<div class=3D"moz-signature">-- <br
<br <br Get
important Linux and industry-related news at: <a
href=3D"http://facebook.com/dcdata">facebook.com/dcdata</... <br
<font color=3D"#000000"><font face=3D"arial"
size=3D"2">Kind regards,=
</font><br <br <font
color=3D"#000000"><font face=3D"arial"><font
size=3D"4"=
<font color=3D"#000000"><font
face=3D"arial"><font size=3D"1"=
<font color=3D"#0779b5"><font
face=3D"arial"><font size=3D"3"=
<font color=3D"#0779b5"><font
face=3D"arial"><font size=3D"1"=
<table border=3D"0" <tbody <tr
<td width=3D"75"><font color=3D"#000000"><font
face=3D"arial"=
<td><font
color=3D"#000000"><font face=3D"arial" </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial" <td><font
color=3D"#000000"><font face=3D"arial" </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial" <td><font
color=3D"#000000"><font face=3D"arial" </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial" <td><font
color=3D"#000000"><font face=3D"arial" </tr
<tr <td><font
color=3D"#000000"><font face=3D"arial" <td><font
color=3D"#000000"><font face=3D"arial" </tr
</tbody </table </font></div =20
<br><br <p style=3D"font-family:
Verdana; font-size:8pt; color:#666666;" align=3D"j=
ustify"
Time to evaluate your email security provider? <a href=3D"http://www.mimeca=
st.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast=
=92s first ever limited promotion.
</p </body
</html
--------------030500050202090506010300--
David
Wilson</font></font></font><brCNS,CLS,
LINUX+, CLA, DCTS,
LPIC3</font></font></font><br<b>LinuxTech
CC t/a
DcData</b></font></font></font><brCK
number:
2001/058368/23</font></font></font><br<font
size=3D"2"><b>Website:</b></font></font></font></td<font
size=3D"2"><a
class=3D"moz-txt-link-freetext" href=3D=
"http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></font><=
/td<font
size=3D"2"><b>Support:</b></font></font></font></td<font
size=3D"2">+27(0)860-1-LINUX</font></font></font></td=
<font
size=3D"2"><b>Mobile:</b></font></font></font></td<font
size=3D"2">+27(0)824147413</font></font></font></td<font
size=3D"2"><b>Tel:</b></font></font></font></td<font
size=3D"2">+27(0)333446100</font></font></font></td<font
size=3D"2"><b>Fax:</b></font></font></font></td<font
size=3D"2">+27(0)866878971</font></font></font></td