This is a multi-part message in MIME format. --------------040205020003020503010804 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit David Wilson:
Hi everyone,
Some feedbackon this error, and a question I hope that someone could possibly assist with. I managed to get the guest running again byusing dd and qemu-img to export the filesystem image and base image, convert to raw format and run a fsck on the looped partition. I am happy you make that work. Can you describe more details about how you dd the guest image? And how did you use qemu-img to export the filesystem image and base image.
When mounted as a loop filesystem, fsck easily repaired it.
Any ideas why oVirt kept pausing the guest and did not allow fsck to repair the filesystem?
Get important Linux and industry-related news at: facebook.com/dcdata <http://facebook.com/dcdata>
Kind regards,
David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* http://www.dcdata.co.za *Support:* +27(0)860-1-LINUX *Mobile:* +27(0)824147413 *Tel:* +27(0)333446100 *Fax:* +27(0)866878971
On 12/02/2012 11:17 AM, David Wilson wrote:
Hi everyone,
Sunday fun :)
One of our critical Linux VMs ran out of disk space. I cleared up space by removing old application logs within the guest, rebooted the guest and was presented with the following information message during bootup when the filesystem ext3/4 scan runs: /Clearing orhpaned inode ...../ Shortly after this message the the guest is paused by oVirt and the "VM has paused due to unknown storage error" message is displayed in the oVirt web GUI.
On the physical node itself, I found the following error in/var/log/vdsm/vdsm.log: /libvirtEventLoop::INFO::2012-12-02 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop) vmId=`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother/
Any ideas on how to get more informationon what exactly the error is?
I've tried booting the guest into rescue mode, run a manual fsck on the logical volume but each time the guest is automaticall paused due to the "unknown storage error".
We are using a FC SAN for storage, sowhat I've decided to try is export the disk using "dd" on the physical node and then try repairing it by mounting it as a loop filesystem on another physical server, and then use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--
Get important Linux and industry-related news at: facebook.com/dcdata <http://facebook.com/dcdata>
Kind regards,
David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* http://www.dcdata.co.za *Support:* +27(0)860-1-LINUX *Mobile:* +27(0)824147413 *Tel:* +27(0)333446100 *Fax:* +27(0)866878971
Time to evaluate your email security provider? Watch the video <http://www.mimecast.com/emailsecurity/> and take advantage of Mimecast's first ever limited promotion.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- --- ?? Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming@cn.ibm.com or shuming@linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC --------------040205020003020503010804 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">David Wilson:<br> </div> <blockquote cite="mid:50BC6909.7050602@dcdata.co.za" type="cite"> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> <div class="moz-cite-prefix"><font size="-1">H<font size="-1">i</font> everyone,<br> <br> <font size="-1"><font size="-1">Some feedback<font size="-1"> on this error<font size="-1">, and a <font size="-1">question I hope that someone could possibly assist with.</font></font><br> <font size="-1">I managed to <font size="-1">get the guest running again by<font size="-1"> using dd <font size="-1">and q<font size="-1">emu-img to export the filesystem image and base image<font size="-1">, convert to raw format and run a fsck on the <font size="-1">loop<font size="-1">ed part<font size="-1">ition.<br> </font></font></font></font></font></font></font></font></font></font></font></font></font></div> </blockquote> I am happy you make that work. Can you describe more details about how you dd the guest image? And how did you use qemu-img to export the filesystem image and base image.<br> <br> <blockquote cite="mid:50BC6909.7050602@dcdata.co.za" type="cite"> <div class="moz-cite-prefix"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"> <font size="-1">When mounted as a loop filesystem, fsck easily repaired <font size="-1">it<font size="-1">.</font></font><br> <br> </font><font size="-1">Any ideas why oVirt kept pausing the guest and did no<font size="-1">t allow fsck to <font size="-1">repair the <font size="-1">filesystem?<br> <br> </font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font><br> </font> <div class="moz-signature"> <br> <br> Get important Linux and industry-related news at: <a moz-do-not-send="true" href="http://facebook.com/dcdata">facebook.com/dcdata</a><br> <br> <font color="#000000"><font face="arial" size="2">Kind regards,</font><br> <br> <font color="#000000"><font face="arial"><font size="4">David Wilson</font></font></font><br> <font color="#000000"><font face="arial"><font size="1">CNS,CLS, LINUX+, CLA, DCTS, LPIC3</font></font></font><br> <font color="#0779b5"><font face="arial"><font size="3"><b>LinuxTech CC t/a DcData</b></font></font></font><br> <font color="#0779b5"><font face="arial"><font size="1">CK number: 2001/058368/23</font></font></font><br> <table border="0"> <tbody> <tr> <td width="75"><font color="#000000"><font face="arial"><font size="2"><b>Website:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2"><a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Support:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)860-1-LINUX</font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Mobile:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)824147413</font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Tel:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)333446100</font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Fax:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)866878971</font></font></font></td> </tr> </tbody> </table> </font></div> On 12/02/2012 11:17 AM, David Wilson wrote:<br> </div> <blockquote cite="mid:50BB1CB7.8020007@dcdata.co.za" type="cite"> <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> <font size="-1"><font size="-1">H</font>i everyone,<br> <br> <font size="-1"><font size="-1">Sunday fun :<font size="-1">)<br> <br> </font>One of our cr<font size="-1">itical <font size="-1">Linux VMs</font> ran out of </font>disk space. I cleared up space by removing old application logs within the guest, rebooted the g<font size="-1">uest and was presented with the following information message during bootup when the filesystem <font size="-1">ext3/<font size="-1">4</font></font> scan runs:<br> <font size="-1"><i>Clearing orhpaned inode .....</i><br> <font size="-1"><font size="-1">Shortly after this message the the guest is p<font size="-1">aused by oVirt and the "VM has paused due to unknown storage error" message is displayed in the <font size="-1">oVirt web <font size="-1">GUI.<br> <br> <font size="-1">On the physical node itself, <font size="-1">I fo<font size="-1">und t<font size="-1">he following error in<font size="-1"> /var/log/vdsm/vd<font size="-1">sm.log:<br> <font size="-1"><i>libvirtEventLoop::<a moz-do-not-send="true" class="moz-txt-link-freetext" href="INFO::2012-12-02">INFO::2012-12-02</a> 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop) vmId=`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother</i><br> <br> <font size="-1">Any ideas on how to get more inf<font size="-1">ormation<font size="-1"> on what exactly the <font size="-1">error is?<br> </font></font></font><br> <font size="-1">I've <font size="-1">tried booting the guest <font size="-1">into r<font size="-1">es<font size="-1">cue mode<font size="-1">, run a manual fsck <font size="-1">on the logical v<font size="-1">olume but each</font></font></font></font></font></font></font></font> time the guest is automaticall paused due to the "unknown storage error".<br> <br> We are using a FC SAN for s<font size="-1">torage</font></font>, s<font size="-1">o<font size="-1"> what I've decided to</font></font> try is expo<font size="-1">rt the disk using "dd" <font size="-1">on the physical node </font>and t<font size="-1">hen try repairing it <font size="-1">by <font size="-1">mounting it as a loop filesystem on <font size="-1">a<font size="-1">nother physical</font></font></font></font></font></font> ser<font size="-1">ver<font size="-1">, and then use "dd" to import it back in again.</font><br> <br> <font size="-1">Is this the right approach to take to get the system back online?<br> <br> </font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font><br> <br> </font> <div class="moz-signature">-- <br> <br> <br> Get important Linux and industry-related news at: <a moz-do-not-send="true" href="http://facebook.com/dcdata">facebook.com/dcdata</a><br> <br> <font color="#000000"><font face="arial" size="2">Kind regards,</font><br> <br> <font color="#000000"><font face="arial"><font size="4">David Wilson</font></font></font><br> <font color="#000000"><font face="arial"><font size="1">CNS,CLS, LINUX+, CLA, DCTS, LPIC3</font></font></font><br> <font color="#0779b5"><font face="arial"><font size="3"><b>LinuxTech CC t/a DcData</b></font></font></font><br> <font color="#0779b5"><font face="arial"><font size="1">CK number: 2001/058368/23</font></font></font><br> <table border="0"> <tbody> <tr> <td width="75"><font color="#000000"><font face="arial"><font size="2"><b>Website:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2"><a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Support:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)860-1-LINUX</font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Mobile:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)824147413</font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Tel:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)333446100</font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Fax:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)866878971</font></font></font></td> </tr> </tbody> </table> </font></div> </blockquote> <br> <br> <br> <p style="font-family: Verdana; font-size:8pt; color:#666666;" align="justify"> Time to evaluate your email security provider? <a moz-do-not-send="true" href="http://www.mimecast.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast’s first ever limited promotion. </p> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <pre class="moz-signature" cols="72">-- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: <a class="moz-txt-link-abbreviated" href="mailto:shuming@cn.ibm.com">shuming@cn.ibm.com</a> or <a class="moz-txt-link-abbreviated" href="mailto:shuming@linux.vnet.ibm.com">shuming@linux.vnet.ibm.com</a> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC</pre> </body> </html> --------------040205020003020503010804--