This is a multi-part message in MIME format. --------------030500050202090506010300 Content-Type: text/plain; charset=WINDOWS-1252; format=flowed Content-Transfer-Encoding: quoted-printable Hi everyone, Sunday fun :) One of our critical Linux VMs ran out of disk space. I cleared up space=20 by removing old application logs within the guest, rebooted the guest=20 and was presented with the following information message during bootup=20 when the filesystem ext3/4 scan runs: /Clearing orhpaned inode ...../ Shortly after this message the the guest is paused by oVirt and the "VM=20 has paused due to unknown storage error" message is displayed in the=20 oVirt web GUI. On the physical node itself, I found the following error=20 in/var/log/vdsm/vdsm.log: /libvirtEventLoop::INFO::2012-12-02=20 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)=20 vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device=20 ide0-0-1 error eother/ Any ideas on how to get more informationon what exactly the error is? I've tried booting the guest into rescue mode, run a manual fsck on the=20 logical volume but each time the guest is automaticall paused due to the=20 "unknown storage error". We are using a FC SAN for storage, sowhat I've decided to try is export=20 the disk using "dd" on the physical node and then try repairing it by=20 mounting it as a loop filesystem on another physical server, and then=20 use "dd" to import it back in again. Is this the right approach to take to get the system back online? --=20 Get important Linux and industry-related news at: facebook.com/dcdata=20 <http://facebook.com/dcdata> Kind regards, David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* =09http://www.dcdata.co.za *Support:* =09+27(0)860-1-LINUX *Mobile:* =09+27(0)824147413 *Tel:* =09+27(0)333446100 *Fax:* =09+27(0)866878971 --------------030500050202090506010300 Content-Type: text/html; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"content-type" content=3D"text/html; charset=3DISO-8= 859-1"> </head> <body bgcolor=3D"#FFFFFF" text=3D"#000000"> <font size=3D"-1"><font size=3D"-1">H</font>i everyone,<br> <br> <font size=3D"-1"><font size=3D"-1">Sunday fun :<font size=3D"-1">)<b= r> <br> </font>One of our cr<font size=3D"-1">itical <font size=3D"-1">Li= nux VMs</font> ran out of </font>disk space. I cleared up space by removing old application logs within the guest, rebooted the g<font size=3D"-1">uest and was presented with the following information message during bootup when the filesystem <font size=3D"-1">ext3/<font size=3D"-1">4</font></f= ont> scan runs:<br> <font size=3D"-1"><i>Clearing orhpaned inode .....</i><br> <font size=3D"-1"><font size=3D"-1">Shortly after this messag= e the the guest is p<font size=3D"-1">aused by oVirt and the "VM has paused due to unknown storage error" message is displayed in the <font size=3D"-1">oVirt web <font size=3D"-1">GUI.<br> <br> <font size=3D"-1">On the physical node itself, <fon= t size=3D"-1">I fo<font size=3D"-1">und t<font size=3D"-1">he following error in<font size=3D"-1"> /var/log/vdsm/vd<font size=3D"-1">sm.log:<br> <font size=3D"-1"><i>libvirtEventLoop::= <a class=3D"moz-txt-link-freetext" href=3D"INFO::2012-12-02">INFO::2012-12-= 02</a> 09:05:50,296::libvirtvm::1965::vm.V= m::(_onAbnormalStop) vmId=3D`23b9212c-1e25-4003-aa18-b1e= 819bf6bb1`::abnormal vm stop device ide0-0-1 error eother</i><br> <br> <font size=3D"-1">Any ideas on how to get more inf<font size=3D"-1">or= mation<font size=3D"-1"> on what exactly the <font size=3D"-1">error is?<br> </font></font></font><br> <font size=3D"-1">I've <font size=3D"-1">tried booting the guest <font size=3D"-1">into r<font size=3D"-1">es<font size=3D"-1">cue mode<font size=3D"-1">, run a manual fsck <font size=3D"-1">on the logical v<font size=3D"-1">olume but each</font></fo= nt></font></font></font></font></font></font> time the guest is automaticall paused due to the "unknown storage error".<br> <br> We are using a FC SAN for s<font size=3D"-1">torage</font></font>, s<font size=3D"-1">o<font size=3D"-1"=
what I've decided to</font></font=
try is expo<font size=3D"-1">rt the disk using "dd" <font size=3D"-1">o= n the physical node </font>and t<font size=3D"-1">hen try repairing it <font size=3D"-1">by <font size=3D"-1">mounting it as a loop filesystem on <font size=3D"-1">a<font size=3D"-1">nother physical</font></font></f= ont></font></font></font> ser<font size=3D"-1">ver<font size=3D"-1">, and then use "dd" to import it back in again.</font=
<br> <br> <font size=3D"-1">Is this the right approach to take to get the system back online?<br> <br> </font></font></font></font></font>= </font></font></font></font></font></font></font></font></font></font></fon= t></font></font><br> <br> </font> <div class=3D"moz-signature">-- <br> <br> <br> Get important Linux and industry-related news at: <a href=3D"http://facebook.com/dcdata">facebook.com/dcdata</a><br> <br> <font color=3D"#000000"><font face=3D"arial" size=3D"2">Kind regards,= </font><br> <br> <font color=3D"#000000"><font face=3D"arial"><font size=3D"4"= David Wilson</font></font></font><br> <font color=3D"#000000"><font face=3D"arial"><font size=3D"1"= CNS,CLS, LINUX+, CLA, DCTS, LPIC3</font></font></font><br> <font color=3D"#0779b5"><font face=3D"arial"><font size=3D"3"= <b>LinuxTech CC t/a DcData</b></font></font></font><br> <font color=3D"#0779b5"><font face=3D"arial"><font size=3D"1"= CK number: 2001/058368/23</font></font></font><br> <table border=3D"0"> <tbody> <tr> <td width=3D"75"><font color=3D"#000000"><font face=3D"arial"= <font size=3D"2"><b>Website:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><a class=3D"moz-txt-link-freetext" href=3D= "http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></font><= /td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Support:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)860-1-LINUX</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Mobile:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)824147413</font></font></font></td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Tel:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)333446100</font></font></font></td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Fax:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)866878971</font></font></font></td> </tr> </tbody> </table> </font></div> =20 <br><br> <p style=3D"font-family: Verdana; font-size:8pt; color:#666666;" align=3D"j= ustify"> Time to evaluate your email security provider? <a href=3D"http://www.mimeca= st.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast= =92s first ever limited promotion. </p> </body> </html> --------------030500050202090506010300--