This is a multi-part message in MIME format. --------------070609000607040903050604 Content-Type: text/plain; charset=WINDOWS-1252; format=flowed Content-Transfer-Encoding: quoted-printable Hi everyone, Some feedbackon this error, and a question I hope that someone could=20 possibly assist with. I managed to get the guest running again byusing dd and qemu-img to=20 export the filesystem image and base image, convert to raw format and=20 run a fsck on the looped partition. When mounted as a loop filesystem, fsck easily repaired it. Any ideas why oVirt kept pausing the guest and did not allow fsck to=20 repair the filesystem? Get important Linux and industry-related news at: facebook.com/dcdata=20 <http://facebook.com/dcdata> Kind regards, David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* =09http://www.dcdata.co.za *Support:* =09+27(0)860-1-LINUX *Mobile:* =09+27(0)824147413 *Tel:* =09+27(0)333446100 *Fax:* =09+27(0)866878971 On 12/02/2012 11:17 AM, David Wilson wrote:
Hi everyone,
Sunday fun :)
One of our critical Linux VMs ran out of disk space. I cleared up=20 space by removing old application logs within the guest, rebooted the=20 guest and was presented with the following information message during=20 bootup when the filesystem ext3/4 scan runs: /Clearing orhpaned inode ...../ Shortly after this message the the guest is paused by oVirt and the=20 "VM has paused due to unknown storage error" message is displayed in=20 the oVirt web GUI.
On the physical node itself, I found the following error=20 in/var/log/vdsm/vdsm.log: /libvirtEventLoop::INFO::2012-12-02=20 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)=20 vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device=20 ide0-0-1 error eother/
Any ideas on how to get more informationon what exactly the error is?
I've tried booting the guest into rescue mode, run a manual fsck on=20 the logical volume but each time the guest is automaticall paused due=20 to the "unknown storage error".
We are using a FC SAN for storage, sowhat I've decided to try is=20 export the disk using "dd" on the physical node and then try repairing=20 it by mounting it as a loop filesystem on another physical server, and=20 then use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--=20
Get important Linux and industry-related news at: facebook.com/dcdata=20 <http://facebook.com/dcdata>
Kind regards,
David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* =09http://www.dcdata.co.za *Support:* =09+27(0)860-1-LINUX *Mobile:* =09+27(0)824147413 *Tel:* =09+27(0)333446100 *Fax:* =09+27(0)866878971
--------------070609000607040903050604 Content-Type: text/html; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable <html> <head> <meta content=3D"text/html; charset=3DISO-8859-1" http-equiv=3D"Content-Type"> </head> <body bgcolor=3D"#FFFFFF" text=3D"#000000"> <div class=3D"moz-cite-prefix"><font size=3D"-1">H<font size=3D"-1">i</= font> everyone,<br> <br> <font size=3D"-1"><font size=3D"-1">Some feedback<font size=3D"-1">= on this error<font size=3D"-1">, and a <font size=3D"-1">questio= n I hope that someone could possibly assist with.</font></f= ont><br> <font size=3D"-1">I managed to <font size=3D"-1">get the guest running again by<font size=3D"-1"> using dd <font size=3D"-1">and q<font size=3D"-1">emu-img to export the filesystem image and base image<font size=3D"-1">, convert to raw format and run a fsck on the <font size=3D"-1">loop<font size=3D"-1">ed part<font size=3D"-1">ition.<b= r> <font size=3D"-1">When mounted as a loop filesystem, fsck easily repaired <font size=3D"-1">it<font size=3D"-1">.</font=
</font><br> <br> </font><font size=3D"-1">Any ideas why oVirt kept pausing the guest and did no<font size=3D"-1">t allow fsck to <font size=3D"-1">repair the <font size=3D"-1">filesystem?<br> <br> </font></font></font></font></font></= font></font></font></font></font></font></font></font></font></font></font>= <br> </font> <div class=3D"moz-signature"> <br> <br> Get important Linux and industry-related news at: <a href=3D"http://facebook.com/dcdata">facebook.com/dcdata</a><br> <br> <font color=3D"#000000"><font face=3D"arial" size=3D"2">Kind regard= s,</font><br> <br> <font color=3D"#000000"><font face=3D"arial"><font size=3D"= 4">David Wilson</font></font></font><br> <font color=3D"#000000"><font face=3D"arial"><font size=3D"= 1">CNS,CLS, LINUX+, CLA, DCTS, LPIC3</font></font></font><br> <font color=3D"#0779b5"><font face=3D"arial"><font size=3D"= 3"><b>LinuxTech CC t/a DcData</b></font></font></font><br> <font color=3D"#0779b5"><font face=3D"arial"><font size=3D"= 1">CK number: 2001/058368/23</font></font></font><br> <table border=3D"0"> <tbody> <tr> <td width=3D"75"><font color=3D"#000000"><font face=3D"aria= l"><font size=3D"2"><b>Website:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><a class=3D"moz-txt-link-freetext" href= =3D"http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></fon= t></td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Support:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)860-1-LINUX</font></font></font></= td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Mobile:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)824147413</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Tel:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)333446100</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Fax:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)866878971</font></font></font></td=
</tr> </tbody> </table> </font></div> On 12/02/2012 11:17 AM, David Wilson wrote:<br> </div> <blockquote cite=3D"mid:50BB1CB7.8020007@dcdata.co.za" type=3D"cite"> <meta http-equiv=3D"content-type" content=3D"text/html; charset=3DISO-8859-1"> <font size=3D"-1"><font size=3D"-1">H</font>i everyone,<br> <br> <font size=3D"-1"><font size=3D"-1">Sunday fun :<font size=3D"-1">)= <br> <br> </font>One of our cr<font size=3D"-1">itical <font size=3D"-1">= Linux VMs</font> ran out of </font>disk space. I cleared up space by removing old application logs within the guest, rebooted the g<font size=3D"-1">uest and was presented with the following information message during bootup when the filesystem <font size=3D"-1">ext3/<font size=3D"-1">4</font><= /font> scan runs:<br> <font size=3D"-1"><i>Clearing orhpaned inode .....</i><br> <font size=3D"-1"><font size=3D"-1">Shortly after this message the the guest is p<font size=3D"-1">aused by oVirt and the "VM has paused due to unknown storage error" message is displayed in the <font size=3D"-1">oVirt web <font size=3D"-1">GUI.<br> <br> <font size=3D"-1">On the physical node itself, <f= ont size=3D"-1">I fo<font size=3D"-1">und t<font size=3D"-1">he following error in<font size=3D"-1"> /var/log/vdsm/vd<font size=3D"-1">sm.log:<br> <font size=3D"-1"><i>libvirtEventLoop= ::<a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"INFO::2012-12-02">INFO:= :2012-12-02</a> 09:05:50,296::libvirtvm::1965::vm= .Vm::(_onAbnormalStop) vmId=3D`23b9212c-1e25-4003-aa18-b= 1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother</i><br> <br> <font size=3D"-1">Any ideas on how to get more inf<font size=3D"-1">= ormation<font size=3D"-1"> on what exactly the <font size=3D"-1">error is?<br> </font></font></font><br> <font size=3D"-1">I've <font size=3D"-1">tried booting the guest <font size=3D"-1">i= nto r<font size=3D"-1">es<font size=3D"-1">cue mode<fo= nt size=3D"-1">, run a manual fsck <font size=3D"-1">on the logical v<font size=3D"-1">olume but each</font></= font></font></font></font></font></font></font> time the guest is automaticall paused due to the "unknown storage error".<br> <br> We are using a FC SAN for s<font size=3D"-1">torage</font></font=
, s<font size=3D"-1">o<font size=3D"-1"> what I've decided to</font></font> try is expo<fo= nt size=3D"-1">rt the disk using "dd" <font size=3D"-1">on the physical node </font>and t<font size=3D"-1">hen try repairing it <font size=3D"-1">by <font size=3D"-1">mounting it as a loop filesystem on <font size=3D"-1">a<font size=3D"-1">nother physical</font></font><= /font></font></font></font> ser<font size=3D"-1">ver<font size=3D"-1">, and then use "dd" to import it back in again.</font><br> <br> <font size=3D"-1">Is this the right approach to take to get the system back online?<br> <br> </font></font></font></font></fon= t></font></font></font></font></font></font></font></font></font></font></f= ont></font></font><br> <br> </font> <div class=3D"moz-signature">-- <br> <br> <br> Get important Linux and industry-related news at: <a moz-do-not-send=3D"true" href=3D"http://facebook.com/dcdata">face= book.com/dcdata</a><br> <br> <font color=3D"#000000"><font face=3D"arial" size=3D"2">Kind regard= s,</font><br> <br> <font color=3D"#000000"><font face=3D"arial"><font size=3D"= 4">David Wilson</font></font></font><br> <font color=3D"#000000"><font face=3D"arial"><font size=3D"= 1">CNS,CLS,
LINUX+, CLA, DCTS, LPIC3</font></font></font><br> <font color=3D"#0779b5"><font face=3D"arial"><font size=3D"= 3"><b>LinuxTech CC t/a DcData</b></font></font></font><br> <font color=3D"#0779b5"><font face=3D"arial"><font size=3D"= 1">CK number: 2001/058368/23</font></font></font><br> <table border=3D"0"> <tbody> <tr> <td width=3D"75"><font color=3D"#000000"><font face=3D"aria= l"><font size=3D"2"><b>Website:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http://www.dcdata.co.za">http://www.dcdat= a.co.za</a></font></font></font></td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Support:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)860-1-LINUX</font></font></font></= td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Mobile:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)824147413</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Tel:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)333446100</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Fax:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)866878971</font></font></font></td=
</tr> </tbody> </table> </font></div> </blockquote> <br> =20 <br><br> <p style=3D"font-family: Verdana; font-size:8pt; color:#666666;" align=3D"j= ustify"> Time to evaluate your email security provider? <a href=3D"http://www.mimeca= st.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast= =92s first ever limited promotion. </p> </body> </html> --------------070609000607040903050604--