This is a multi-part message in MIME format. --------------050904000702030906090307 Content-Type: text/plain; charset=WINDOWS-1252; format=flowed Content-Transfer-Encoding: quoted-printable Hi everyone, Thank you for your responses. Yeela: />When oVirt started the vm it mounted the corrupt disk image that=20 seemed fine, but it couldn't find the OS because of the corrupted fs,=20 and the error caused it to pause the guest./ To clear things up it was the /var mount that had the corrupted=20 filesystem, not the guest's root filesystem. oVirt still continued to pause the guest even when I had booted the=20 guest off a CentOS DVD ISO, run rescue mode and manually activated the=20 /var logical volume. In this case I did not activate or mount the guests=20 root filesystem and only activated the guests /var filesystem so that I=20 could fsck it. fsck would run for around 5-10 minutes with the message=20 "Deleting orphaned inode......" and then oVirt would simply pause the=20 entire guest. The only information I could find was on the physical host's vdsm.log,=20 which specified the following: /libvirtEventLoop::INFO::2012-12-02=20 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)=20 vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device=20 ide0-0-1 error eother /Perhaps there was another log I should have examined to see if more=20 information was provided about why oVirt was pausing the guest? // Shu: This is what I did to dd the images off, and to work around the problem: 1.) On the physical host: Created an NFS mount to another temporary=20 Linux system that had sufficient storage for the 500GB filesystem 2.) On the physical host: Used 'dd' to dump the /var filesystem's=20 logical volume to an image filevia NFS on the temporary Linux system. 3.) On the temporary Linux system that now contained the filesystem=20 image file, I ran "qemu-img info" and noticed that thefilesystem image=20 was qcow2 type and specified a backing file. 4.) On the physical host: Used 'dd' to dump the logical volume specified=20 as a backing file, to an image file via NFS on the temporary Linux system. 5.) On the temporary Linux system: Used 'qemu-img rebase' to change the=20 backing file to the local copyof the backing file image. 6.) On the temporary Linux system: Used 'qemu-img commit' to commit the=20 changes stored in the filesystem image file to the backing file image. 7.) On the temporary Linux system: Used 'qemu-img convert' to convert=20 the backing file image to raw format. 8.) Onthe temporary Linux system: Used 'losetup', 'kpart'and 'fsck' to=20 repair the backing file image. Fsck displayed the same 'Deleting=20 orphaned inode....' message but managed to continue and completed ok. 9.) On the temporary Linux system: Mounted the loop filesystem and=20 confirmed that the data was intact and was current. 10.) In the oVirt GUI: Deactivated the faulty Virtual Disk attached to=20 the guest. 11.) In the oVirt GUI: Created a new 'preallocated' Virtual Disk of=20 sufficient sizefor the guest. 12.) On the physical host: Used 'dd' to upload the raw backing file=20 image from (7) to the new logical volume. 13.) I then configured the guest to boot from the CentOS DVD ISO into=20 rescue modeto confirm that the logical volume for the guest's /var=20 filesystem was accessible and mountable. 14.) Reconfigured the guest to bootfrom it's primary Virtual Disk and=20 started up the guest. // Get important Linux and industry-related news at: facebook.com/dcdata=20 <http://facebook.com/dcdata> Kind regards, David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* =09http://www.dcdata.co.za *Support:* =09+27(0)860-1-LINUX *Mobile:* =09+27(0)824147413 *Tel:* =09+27(0)333446100 *Fax:* =09+27(0)866878971 On 12/03/2012 01:07 PM, Dan Kenigsberg wrote:
On Mon, Dec 03, 2012 at 04:37:01AM -0500, Yeela Kaplan wrote:
Glad to hear it worked out.
When oVirt started the vm it mounted the corrupt disk image that seemed = fine, but it couldn't find the OS because of the corrupted fs, and the error caused it to pause the guest. I think we should be a bit more exact here: a VM without an installed OS does not pause. The cause of the pause was, most probably, and attempt to read from a corrupted qcow. When qemu fails to serve the guest with data due to a bug in the underlying storage, qemu stops and waits for further instructions from management. We use this feature for automatic lv-extend (on enospace error). But here, with eother error, a human intervension is required.
--------------050904000702030906090307 Content-Type: text/html; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable <html> <head> <meta content=3D"text/html; charset=3Dwindows-1252" http-equiv=3D"Content-Type"> </head> <body bgcolor=3D"#FFFFFF" text=3D"#000000"> <div class=3D"moz-cite-prefix"><font size=3D"-1">Hi everyone,<br> <br> <font size=3D"-1">Thank you for your responses.<br> <br> <font size=3D"-1">Yeela:</font></font></font><br> <small><i>>When oVirt started the vm it mounted the corrupt disk image that seemed fine, but it couldn't find the OS because of the corrupted fs, and the error caused it to pause the guest.</i><br> <br> To clear things up it was the /var mount that had the corrupted filesystem, not the guest's root filesystem.<br> oVirt still continued to pause the guest even when I had booted the guest off a CentOS DVD ISO, run rescue mode and manually activated the /var logical volume. In this case I did not activate or mount the guests root filesystem and only activated the guests /var filesystem so that I could fsck it. fsck would run for around 5-10 minutes with the message "Deleting orphaned inode......" and then oVirt would simply pause the entire guest.<br=
The only information I could find was on the physical host's vdsm.log, which specified the following:<br> </small><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D= "-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"=
<font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><i>libv= irtEventLoop::<a class=3D"moz-txt-link-freetext" href=3D"INFO::2012-12-02">INFO::2012-12-02<= /a> 09:05:50,296::libvirtvm::1965::vm= .Vm::(_onAbnormalStop)
vmId=3D`23b9212c-1e25-4003-aa18-b= 1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother<br> <br> </i><font size=3D"-1"><font size=3D"-1">Perh<font size=3D"-1">aps there was another log I should have examined to see if more i<fon= t size=3D"-1">nforma<font size=3D"-1">tion was provided about wh<font size=3D"-1">y oVirt was pausing the guest?<br> <br> </font></font></font></fo= nt></font></font><i></i><font size=3D"-1"><font size=3D"-1"><br=
<br> Shu:<br> <font size=3D"-1">This is what I did to dd the images off, and to <font size=3D"-1">work around the problem<font size=3D"-1">:<br> </font></font><font size=3D"-1">1.) <font size=3D"-1">O<font size=3D"-1">n the physical host<font size=3D"-1">:</font> <font size=3D"-1">C</fo= nt></font>reated an NFS mount to another temporary Linux system that had sufficient storage for the 500GB filesystem<br> <font size=3D"-1">2.<font size=3D"-1">) O<font size=3D"-1">n</font=
the physical host<fon= t size=3D"-1">: <font size=3D"-1">U</fo= nt></font>sed 'dd' to <font size=3D"-1">dump the /var filesystem's logical vol<font size=3D"-1">ume to an image file<font size=3D"-1"> <fon= t size=3D"-1">via NFS on t<font size=3D"-1">he temporar<font size=3D"-1">y</fo= nt> Linux <font size=3D"-1">syste= m</font></font></font>.</font></font></font></font></font><br> <font size=3D"-1">3.) On the temporary Linux s<font size=3D"-1">yste= m that now contained the filesystem image file, I ran "qemu-img info</font>= </font>" and noticed that the<font size=3D"-1"> fi<font size=3D"-1">lesystem<= /font> image was <font size=3D"-1">qc<font size=3D"-1">ow2 <fo= nt size=3D"-1">type<= /font> and</font></font> specified a ba<font size=3D"-1">cking file.<br> <font size=3D"-1">4.) On t<font size=3D"-1">he physical host<fon= t size=3D"-1">: Used <font size=3D"-1">'dd' to dump <font size=3D"-1">the lo<font size=3D"-1">gical volume specified as a backing file, to an image file </font></fon= t></font></font></font></font></font></font></font></font></font></font></f= ont></font></font></font></font></font></font></font></font></font></font><= /font></font></font></font></font></font></font><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><= font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"=
<font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><fo= nt size=3D"-1"><font size=3D"-1"= <font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1">via NFS on t<font size=3D"-1">he temporar<font size=3D"-1">y</fo= nt> Linux <font size=3D"-1">syste= m.<br> <font size=3D"-1">5.<fo= nt size=3D"-1">) On the <font size=3D"-1">tempo= rary Linux system: Used '<font size=3D"-1">qemu-= img reba<font size=3D"-1">se</f= ont>' to change the backing file <fon= t size=3D"-1">to the local copy<fo= nt size=3D"-1"> of the back<font size=3D"-1">ing file image.</font= </font></font></font></font></font></font><br> <font size=3D"-1">6.) On the temporary Linux system: Used 'qemu-img commit' to commit the changes stored in the filesystem image file to the backing file image.<br> <font size=3D"-1">7.) O<font size=3D"-1">n the temporary Linux sy<font size=3D"-1">ste<f= ont size=3D"-1">m<fon= t size=3D"-1">: U<f= ont size=3D"-1">sed 'qemu-img convert' to convert the backing file image to raw format.<br> <font size=3D"-1">8.) On<font size=3D"-1"> <fon= t size=3D"-1">the <font size=3D"-1">tempo= rary Linux system: <font size=3D"-1">Used 'losetup', <font size=3D"-1">'kpar= t'<font size=3D"-1"><font size=3D"-1"> and </font>'fsck'</fo= nt></font></font></font></font></font></font> to <font size=3D"-1">repai= r the backing file image. Fsck displayed t<font size=3D"-1">he same 'Del<font size=3D"-1">eting <font size=3D"-1">orpha= ned in<font size=3D"-1">ode<f= ont size=3D"-1">....' <font size=3D"-1">messa= ge</font></font></font></font></font></font></font> <font size=3D"-1">but managed to <font size=3D"-1">con<f= ont size=3D"-1">tinue and completed ok.<br> <font size=3D"-1">9.) O<font size=3D"-1">n the tempo<font size=3D"-1">rary Linux system: <font size=3D"-1">Mo<fo= nt size=3D"-1">unted the loop filesystem and confirmed that <font size=3D"-1">the <font size=3D"-1">data was intact and was current.</fon= t></font></font></font></font></font></font><br> </font></font></f= ont><font size=3D"-1"><font size=3D"-1">10</f= ont>.) I<font size=3D"-1">n <fo= nt size=3D"-1">the oVirt GUI: Deactivated the faulty Virtual <font size=3D"-1">D</fo= nt>isk attached to the guest.</font>= </font></font><br> <font size=3D"-1"><font size=3D"-1">11</f= ont>.) In the oVirt GUI: Created a new 'preallocated<fon= t size=3D"-1">' <fo= nt size=3D"-1">Virtu= al Disk <font size=3D"-1">of sufficient size<font size=3D"-1"> for the guest.<br> <font size=3D"-1">1<fon= t size=3D"-1">2</fo= nt>.) O<font size=3D"-1">n the physical host: Used <font size=3D"-1">'dd' to upload the <font size=3D"-1">raw ba<font size=3D"-1">cking file image from (7) to the new lo<font size=3D"-1">gical volume.</font></f= ont></font></font></font></font><br> <font size=3D"-1">1<fon= t size=3D"-1">3</fo= nt>.) <font size=3D"-1">I then conf<font size=3D"-1">igure= d the guest to boot from the CentOS <font size=3D"-1">D<fon= t size=3D"-1">VD ISO into res<font size=3D"-1">cue mode<font size=3D"-1"> to confirm that the <font size=3D"-1">lo<fo= nt size=3D"-1">gical <font size=3D"-1">volum= e <font size=3D"-1"><font size=3D"-1">for th<font size=3D"-1">e <fo= nt size=3D"-1"><font size=3D"-1">guest= <font size=3D"-1">'s /var </font></fon= t>filesystem</font></font> was accessi<font size=3D"-1">ble and mountable.<br=
<font size=3D"-1">1<fon= t size=3D"-1">4</fo= nt>.) <font size=3D"-1">Recon= figured the guest to boo<font size=3D"-1">t<fon= t size=3D"-1"> from it's <font size=3D"-1">prima= ry <font size=3D"-1">Virtu= al Disk</font></font=
and sta<font size=3D"-1">rted up the guest.</fo= nt></font></font></font></font><br> </font></font></f= ont></font></font></font></font></font></font></font></font></font></font><= /font></font></font></font></font></font></font></font></font></font></font=
<br> </font><br> </font></font></f= ont></font></font></font></font></font></font></font></font></font></font><= /font></font></font></font></font></font></font></font></font></font></font= </font></font></font></font></font></font></font></font></font></font></fo= nt></font></font></font></font></font></font></font></font><i></i></font></= font></font></font></font></font></font></font></font></font></font></font>= </font></font></font></font></font><br> <br> <br> <br> <br> <div class=3D"moz-signature"> <br> <br> Get important Linux and industry-related news at: <a href=3D"http://facebook.com/dcdata">facebook.com/dcdata</a><br> <br> <font color=3D"#000000"><font face=3D"arial" size=3D"2">Kind regard= s,</font><br> <br> =A0<font color=3D"#000000"><font face=3D"arial"><font size=3D"4">= David Wilson</font></font></font><br> =A0<font color=3D"#000000"><font face=3D"arial"><font size=3D"1">= CNS,CLS, LINUX+, CLA, DCTS, LPIC3</font></font></font><br> =A0<font color=3D"#0779b5"><font face=3D"arial"><font size=3D"3">= <b>LinuxTech CC t/a DcData</b></font></font></font><br> =A0<font color=3D"#0779b5"><font face=3D"arial"><font size=3D"1">= CK number: 2001/058368/23</font></font></font><br> <table border=3D"0"> <tbody> <tr> <td width=3D"75"><font color=3D"#000000"><font face=3D"aria= l"><font size=3D"2"><b>Website:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><a class=3D"moz-txt-link-freetext" href= =3D"http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></fon= t></td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Support:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)860-1-LINUX</font></font></font></= td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Mobile:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)824147413</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Tel:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)333446100</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Fax:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)866878971</font></font></font></td=
</tr> </tbody> </table> </font></div> On 12/03/2012 01:07 PM, Dan Kenigsberg wrote:<br> </div> <blockquote cite=3D"mid:20121203110737.GD20719@redhat.com" type=3D"cite= "> <pre wrap=3D"">On Mon, Dec 03, 2012 at 04:37:01AM -0500, Yeela Kaplan= wrote: </pre> <blockquote type=3D"cite"> <pre wrap=3D"">Glad to hear it worked out. When oVirt started the vm it mounted the corrupt disk image that seemed fin= e, =20 but it couldn't find the OS because of the corrupted fs, and the error caused it to pause the guest. </pre> </blockquote> <pre wrap=3D""> I think we should be a bit more exact here: a VM without an installed OS does not pause. The cause of the pause was, most probably, and attempt to read from a corrupted qcow. When qemu fails to serve the guest with data due to a bug in the underlying storage, qemu stops and waits for further instructions from management. We use this feature for automatic lv-extend (on enospace error). But here, with eother error, a human intervension is required. </pre> </blockquote> <br> =20 <br><br> <p style=3D"font-family: Verdana; font-size:8pt; color:#666666;" align=3D"j= ustify"> Time to evaluate your email security provider? <a href=3D"http://www.mimeca= st.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast= =92s first ever limited promotion. </p> </body> </html> --------------050904000702030906090307--