[Users] VM has paused due to unknown storage error
This is a multi-part message in MIME format. --------------030500050202090506010300 Content-Type: text/plain; charset=WINDOWS-1252; format=flowed Content-Transfer-Encoding: quoted-printable Hi everyone, Sunday fun :) One of our critical Linux VMs ran out of disk space. I cleared up space=20 by removing old application logs within the guest, rebooted the guest=20 and was presented with the following information message during bootup=20 when the filesystem ext3/4 scan runs: /Clearing orhpaned inode ...../ Shortly after this message the the guest is paused by oVirt and the "VM=20 has paused due to unknown storage error" message is displayed in the=20 oVirt web GUI. On the physical node itself, I found the following error=20 in/var/log/vdsm/vdsm.log: /libvirtEventLoop::INFO::2012-12-02=20 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)=20 vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device=20 ide0-0-1 error eother/ Any ideas on how to get more informationon what exactly the error is? I've tried booting the guest into rescue mode, run a manual fsck on the=20 logical volume but each time the guest is automaticall paused due to the=20 "unknown storage error". We are using a FC SAN for storage, sowhat I've decided to try is export=20 the disk using "dd" on the physical node and then try repairing it by=20 mounting it as a loop filesystem on another physical server, and then=20 use "dd" to import it back in again. Is this the right approach to take to get the system back online? --=20 Get important Linux and industry-related news at: facebook.com/dcdata=20 <http://facebook.com/dcdata> Kind regards, David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* =09http://www.dcdata.co.za *Support:* =09+27(0)860-1-LINUX *Mobile:* =09+27(0)824147413 *Tel:* =09+27(0)333446100 *Fax:* =09+27(0)866878971 --------------030500050202090506010300 Content-Type: text/html; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"content-type" content=3D"text/html; charset=3DISO-8= 859-1"> </head> <body bgcolor=3D"#FFFFFF" text=3D"#000000"> <font size=3D"-1"><font size=3D"-1">H</font>i everyone,<br> <br> <font size=3D"-1"><font size=3D"-1">Sunday fun :<font size=3D"-1">)<b= r> <br> </font>One of our cr<font size=3D"-1">itical <font size=3D"-1">Li= nux VMs</font> ran out of </font>disk space. I cleared up space by removing old application logs within the guest, rebooted the g<font size=3D"-1">uest and was presented with the following information message during bootup when the filesystem <font size=3D"-1">ext3/<font size=3D"-1">4</font></f= ont> scan runs:<br> <font size=3D"-1"><i>Clearing orhpaned inode .....</i><br> <font size=3D"-1"><font size=3D"-1">Shortly after this messag= e the the guest is p<font size=3D"-1">aused by oVirt and the "VM has paused due to unknown storage error" message is displayed in the <font size=3D"-1">oVirt web <font size=3D"-1">GUI.<br> <br> <font size=3D"-1">On the physical node itself, <fon= t size=3D"-1">I fo<font size=3D"-1">und t<font size=3D"-1">he following error in<font size=3D"-1"> /var/log/vdsm/vd<font size=3D"-1">sm.log:<br> <font size=3D"-1"><i>libvirtEventLoop::= <a class=3D"moz-txt-link-freetext" href=3D"INFO::2012-12-02">INFO::2012-12-= 02</a> 09:05:50,296::libvirtvm::1965::vm.V= m::(_onAbnormalStop) vmId=3D`23b9212c-1e25-4003-aa18-b1e= 819bf6bb1`::abnormal vm stop device ide0-0-1 error eother</i><br> <br> <font size=3D"-1">Any ideas on how to get more inf<font size=3D"-1">or= mation<font size=3D"-1"> on what exactly the <font size=3D"-1">error is?<br> </font></font></font><br> <font size=3D"-1">I've <font size=3D"-1">tried booting the guest <font size=3D"-1">into r<font size=3D"-1">es<font size=3D"-1">cue mode<font size=3D"-1">, run a manual fsck <font size=3D"-1">on the logical v<font size=3D"-1">olume but each</font></fo= nt></font></font></font></font></font></font> time the guest is automaticall paused due to the "unknown storage error".<br> <br> We are using a FC SAN for s<font size=3D"-1">torage</font></font>, s<font size=3D"-1">o<font size=3D"-1"=
what I've decided to</font></font=
try is expo<font size=3D"-1">rt the disk using "dd" <font size=3D"-1">o= n the physical node </font>and t<font size=3D"-1">hen try repairing it <font size=3D"-1">by <font size=3D"-1">mounting it as a loop filesystem on <font size=3D"-1">a<font size=3D"-1">nother physical</font></font></f= ont></font></font></font> ser<font size=3D"-1">ver<font size=3D"-1">, and then use "dd" to import it back in again.</font=
<br> <br> <font size=3D"-1">Is this the right approach to take to get the system back online?<br> <br> </font></font></font></font></font>= </font></font></font></font></font></font></font></font></font></font></fon= t></font></font><br> <br> </font> <div class=3D"moz-signature">-- <br> <br> <br> Get important Linux and industry-related news at: <a href=3D"http://facebook.com/dcdata">facebook.com/dcdata</a><br> <br> <font color=3D"#000000"><font face=3D"arial" size=3D"2">Kind regards,= </font><br> <br> <font color=3D"#000000"><font face=3D"arial"><font size=3D"4"= David Wilson</font></font></font><br> <font color=3D"#000000"><font face=3D"arial"><font size=3D"1"= CNS,CLS, LINUX+, CLA, DCTS, LPIC3</font></font></font><br> <font color=3D"#0779b5"><font face=3D"arial"><font size=3D"3"= <b>LinuxTech CC t/a DcData</b></font></font></font><br> <font color=3D"#0779b5"><font face=3D"arial"><font size=3D"1"= CK number: 2001/058368/23</font></font></font><br> <table border=3D"0"> <tbody> <tr> <td width=3D"75"><font color=3D"#000000"><font face=3D"arial"= <font size=3D"2"><b>Website:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><a class=3D"moz-txt-link-freetext" href=3D= "http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></font><= /td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Support:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)860-1-LINUX</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Mobile:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)824147413</font></font></font></td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Tel:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)333446100</font></font></font></td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Fax:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)866878971</font></font></font></td> </tr> </tbody> </table> </font></div> =20 <br><br> <p style=3D"font-family: Verdana; font-size:8pt; color:#666666;" align=3D"j= ustify"> Time to evaluate your email security provider? <a href=3D"http://www.mimeca= st.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast= =92s first ever limited promotion. </p> </body> </html> --------------030500050202090506010300--
It seems that your guest file system is corrupted. The dd approach won't help cause this is a virtual disk, you need to fix the virtual disk inside the lv, that's why any operations on the lv won't help. You can try shutting down the vm (not paused, but literally down) and add the guest vm disk as a second disk to another vm. In the second vm try to run fsck on the disk and see if it helps. In any case before trying to manipulate the disk image please save a backup!!! let me know how it goes... ----- Original Message -----
From: "David Wilson" <dave@dcdata.co.za> To: users@ovirt.org Sent: Sunday, December 2, 2012 11:17:43 AM Subject: [Users] VM has paused due to unknown storage error
H i everyone,
Sunday fun : )
One of our cr itical Linux VMs ran out of disk space. I cleared up space by removing old application logs within the guest, rebooted the g uest and was presented with the following information message during bootup when the filesystem ext3/ 4 scan runs: Clearing orhpaned inode ..... Shortly after this message the the guest is p aused by oVirt and the "VM has paused due to unknown storage error" message is displayed in the oVirt web GUI.
On the physical node itself, I fo und t he following error in /var/log/vdsm/vd sm.log: libvirtEventLoop:: INFO::2012-12-02 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop) vmId=`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother
Any ideas on how to get more inf ormation on what exactly the error is?
I've tried booting the guest into r es cue mode , run a manual fsck on the logical v olume but each time the guest is automaticall paused due to the "unknown storage error".
We are using a FC SAN for s torage , s o what I've decided to try is expo rt the disk using "dd" on the physical node and t hen try repairing it by mounting it as a loop filesystem on a nother physical ser ver , and then use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--
Get important Linux and industry-related news at: facebook.com/dcdata
Kind regards,
David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 LinuxTech CC t/a DcData CK number: 2001/058368/23
Website: http://www.dcdata.co.za Support: +27(0)860-1-LINUX Mobile: +27(0)824147413 Tel: +27(0)333446100 Fax: +27(0)866878971
Time to evaluate your email security provider? Watch the video and take advantage of Mimecast’s first ever limited promotion. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
It seems that your guest file system is corrupted. The dd approach won't help cause this is a virtual disk, you need to fix the virtual disk inside the lv, that's why any operations= on the lv won't help. You can try shutting down the vm (not paused, but literally down) and add=
This is a multi-part message in MIME format. --------------000007070806030102080207 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Thanks for your reply Yeela. I appreciate your feedback. Please see my feedback to the list earlier..... weird that I was able to=20 repair the disk using another physical system.Butwhen attached to the=20 guestoVirt kept pausing the guest which prevented fsck from completing. Get important Linux and industry-related news at: facebook.com/dcdata=20 <http://facebook.com/dcdata> Kind regards, David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* =09http://www.dcdata.co.za *Support:* =09+27(0)860-1-LINUX *Mobile:* =09+27(0)824147413 *Tel:* =09+27(0)333446100 *Fax:* =09+27(0)866878971 On 12/03/2012 10:36 AM, Yeela Kaplan wrote: the guest vm disk as
a second disk to another vm. In the second vm try to run fsck on the disk and see if it helps. In any case before trying to manipulate the disk image please save a back= up!!! let me know how it goes...
----- Original Message -----
From: "David Wilson" <dave@dcdata.co.za> To: users@ovirt.org Sent: Sunday, December 2, 2012 11:17:43 AM Subject: [Users] VM has paused due to unknown storage error
H i everyone,
Sunday fun : )
One of our cr itical Linux VMs ran out of disk space. I cleared up space by removing old application logs within the guest, rebooted the g uest and was presented with the following information message during bootup when the filesystem ext3/ 4 scan runs: Clearing orhpaned inode ..... Shortly after this message the the guest is p aused by oVirt and the "VM has paused due to unknown storage error" message is displayed in the oVirt web GUI.
On the physical node itself, I fo und t he following error in /var/log/vdsm/vd sm.log: libvirtEventLoop:: INFO::2012-12-02 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop) vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother
Any ideas on how to get more inf ormation on what exactly the error is?
I've tried booting the guest into r es cue mode , run a manual fsck on the logical v olume but each time the guest is automaticall paused due to the "unknown storage error".
We are using a FC SAN for s torage , s o what I've decided to try is expo rt the disk using "dd" on the physical node and t hen try repairing it by mounting it as a loop filesystem on a nother physical ser ver , and then use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--
Get important Linux and industry-related news at: facebook.com/dcdata
Kind regards,
David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 LinuxTech CC t/a DcData CK number: 2001/058368/23
=09Website: =09http://www.dcdata.co.za =09Support: =09+27(0)860-1-LINUX =09Mobile: =09+27(0)824147413 =09Tel: =09+27(0)333446100 =09Fax: =09+27(0)866878971
Time to evaluate your email security provider? Watch the video and take advantage of Mimecast=E2=80=99s first ever limited promotion. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------000007070806030102080207 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
<html> <head> <meta content=3D"text/html; charset=3DUTF-8" http-equiv=3D"Content-Type= "> </head> <body bgcolor=3D"#FFFFFF" text=3D"#000000"> <div class=3D"moz-cite-prefix"><font size=3D"-1">Thanks for your reply Yeela. I appreciate your feedback.<br> <br> <font size=3D"-1">Please see my <font size=3D"-1">feedback to the list earlier..... weird that I was a<font size=3D"-1">ble to repair the disk using another physical system<font size=3D"-1"><font size=3D"-1">.<font size=3D"-1"> Bu<font size=3D"-1">t<font size=3D"-1"> when attached to the guest</font></font></font></font><font size=3D"-1"> oVirt kept pausing the guest which prevented fsck from completing<font size=3D"-1">.<br> <font size=3D"-1"><font size=3D"-1"></font></font><br> <br> </font></font></font></font></font></font><br> </font> <div class=3D"moz-signature"> <br> <br> Get important Linux and industry-related news at: <a href=3D"http://facebook.com/dcdata">facebook.com/dcdata</a><br> <br> <font color=3D"#000000"><font face=3D"arial" size=3D"2">Kind regard= s,</font><br> <br> =C2=A0<font color=3D"#000000"><font face=3D"arial"><font size=3D"= 4">David Wilson</font></font></font><br> =C2=A0<font color=3D"#000000"><font face=3D"arial"><font size=3D"= 1">CNS,CLS, LINUX+, CLA, DCTS, LPIC3</font></font></font><br> =C2=A0<font color=3D"#0779b5"><font face=3D"arial"><font size=3D"= 3"><b>LinuxTech CC t/a DcData</b></font></font></font><br> =C2=A0<font color=3D"#0779b5"><font face=3D"arial"><font size=3D"= 1">CK number: 2001/058368/23</font></font></font><br> <table border=3D"0"> <tbody> <tr> <td width=3D"75"><font color=3D"#000000"><font face=3D"aria= l"><font size=3D"2"><b>Website:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><a class=3D"moz-txt-link-freetext" href= =3D"http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></fon= t></td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Support:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)860-1-LINUX</font></font></font></= td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Mobile:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)824147413</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Tel:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)333446100</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Fax:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)866878971</font></font></font></td=
</tr> </tbody> </table> </font></div> On 12/03/2012 10:36 AM, Yeela Kaplan wrote:<br> </div> <blockquote cite=3D"mid:636934950.27906854.1354523800172.JavaMail.root@redhat.com= " type=3D"cite"> <pre wrap=3D"">It seems that your guest file system is corrupted. The dd approach won't help cause this is a virtual disk, you need to fix the virtual disk inside the lv, that's why any operations o= n the lv won't help.=20 You can try shutting down the vm (not paused, but literally down) and add t= he guest vm disk as a second disk to another vm. In the second vm try to run fsck on the disk and see if it helps. In any case before trying to manipulate the disk image please save a backup= !!! let me know how it goes... ----- Original Message ----- </pre> <blockquote type=3D"cite"> <pre wrap=3D"">From: "David Wilson" <a class=3D"moz-txt-link-rfc239= 6E" href=3D"mailto:dave@dcdata.co.za"><dave@dcdata.co.za></a> To: <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:users@ovirt.org">u= sers@ovirt.org</a> Sent: Sunday, December 2, 2012 11:17:43 AM Subject: [Users] VM has paused due to unknown storage error H i everyone, Sunday fun : ) One of our cr itical Linux VMs ran out of disk space. I cleared up space by removing old application logs within the guest, rebooted the g uest and was presented with the following information message during bootup when the filesystem ext3/ 4 scan runs: Clearing orhpaned inode ..... Shortly after this message the the guest is p aused by oVirt and the "VM has paused due to unknown storage error" message is displayed in the oVirt web GUI. On the physical node itself, I fo und t he following error in /var/log/vdsm/vd sm.log: libvirtEventLoop:: <a class=3D"moz-txt-link-freetext" href=3D"INFO::2012-12= -02">INFO::2012-12-02</a> 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop) vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother Any ideas on how to get more inf ormation on what exactly the error is? I've tried booting the guest into r es cue mode , run a manual fsck on the logical v olume but each time the guest is automaticall paused due to the "unknown storage error". We are using a FC SAN for s torage , s o what I've decided to try is expo rt the disk using "dd" on the physical node and t hen try repairing it by mounting it as a loop filesystem on a nother physical ser ver , and then use "dd" to import it back in again. Is this the right approach to take to get the system back online? -- Get important Linux and industry-related news at: facebook.com/dcdata Kind regards, David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 LinuxTech CC t/a DcData CK number: 2001/058368/23 =09Website: =09<a class=3D"moz-txt-link-freetext" href=3D"http://www.dcdata= .co.za">http://www.dcdata.co.za</a> =09Support: =09+27(0)860-1-LINUX =09Mobile: =09+27(0)824147413 =09Tel: =09+27(0)333446100 =09Fax: =09+27(0)866878971 Time to evaluate your email security provider? Watch the video and take advantage of Mimecast=E2=80=99s first ever limited promotion. _______________________________________________ Users mailing list <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org">Users= @ovirt.org</a> <a class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.org/mailman/l= istinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <pre wrap=3D""> </pre> </blockquote> <br> =20 <br><br> <p style=3D"font-family: Verdana; font-size:8pt; color:#666666;" align=3D"j= ustify"> Time to evaluate your email security provider? <a href=3D"http://www.mimeca= st.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast= =E2=80=99s first ever limited promotion. </p> </body> </html> --------------000007070806030102080207--
This is a multi-part message in MIME format. --------------070609000607040903050604 Content-Type: text/plain; charset=WINDOWS-1252; format=flowed Content-Transfer-Encoding: quoted-printable Hi everyone, Some feedbackon this error, and a question I hope that someone could=20 possibly assist with. I managed to get the guest running again byusing dd and qemu-img to=20 export the filesystem image and base image, convert to raw format and=20 run a fsck on the looped partition. When mounted as a loop filesystem, fsck easily repaired it. Any ideas why oVirt kept pausing the guest and did not allow fsck to=20 repair the filesystem? Get important Linux and industry-related news at: facebook.com/dcdata=20 <http://facebook.com/dcdata> Kind regards, David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* =09http://www.dcdata.co.za *Support:* =09+27(0)860-1-LINUX *Mobile:* =09+27(0)824147413 *Tel:* =09+27(0)333446100 *Fax:* =09+27(0)866878971 On 12/02/2012 11:17 AM, David Wilson wrote:
Hi everyone,
Sunday fun :)
One of our critical Linux VMs ran out of disk space. I cleared up=20 space by removing old application logs within the guest, rebooted the=20 guest and was presented with the following information message during=20 bootup when the filesystem ext3/4 scan runs: /Clearing orhpaned inode ...../ Shortly after this message the the guest is paused by oVirt and the=20 "VM has paused due to unknown storage error" message is displayed in=20 the oVirt web GUI.
On the physical node itself, I found the following error=20 in/var/log/vdsm/vdsm.log: /libvirtEventLoop::INFO::2012-12-02=20 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)=20 vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device=20 ide0-0-1 error eother/
Any ideas on how to get more informationon what exactly the error is?
I've tried booting the guest into rescue mode, run a manual fsck on=20 the logical volume but each time the guest is automaticall paused due=20 to the "unknown storage error".
We are using a FC SAN for storage, sowhat I've decided to try is=20 export the disk using "dd" on the physical node and then try repairing=20 it by mounting it as a loop filesystem on another physical server, and=20 then use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--=20
Get important Linux and industry-related news at: facebook.com/dcdata=20 <http://facebook.com/dcdata>
Kind regards,
David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* =09http://www.dcdata.co.za *Support:* =09+27(0)860-1-LINUX *Mobile:* =09+27(0)824147413 *Tel:* =09+27(0)333446100 *Fax:* =09+27(0)866878971
--------------070609000607040903050604 Content-Type: text/html; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable <html> <head> <meta content=3D"text/html; charset=3DISO-8859-1" http-equiv=3D"Content-Type"> </head> <body bgcolor=3D"#FFFFFF" text=3D"#000000"> <div class=3D"moz-cite-prefix"><font size=3D"-1">H<font size=3D"-1">i</= font> everyone,<br> <br> <font size=3D"-1"><font size=3D"-1">Some feedback<font size=3D"-1">= on this error<font size=3D"-1">, and a <font size=3D"-1">questio= n I hope that someone could possibly assist with.</font></f= ont><br> <font size=3D"-1">I managed to <font size=3D"-1">get the guest running again by<font size=3D"-1"> using dd <font size=3D"-1">and q<font size=3D"-1">emu-img to export the filesystem image and base image<font size=3D"-1">, convert to raw format and run a fsck on the <font size=3D"-1">loop<font size=3D"-1">ed part<font size=3D"-1">ition.<b= r> <font size=3D"-1">When mounted as a loop filesystem, fsck easily repaired <font size=3D"-1">it<font size=3D"-1">.</font=
</font><br> <br> </font><font size=3D"-1">Any ideas why oVirt kept pausing the guest and did no<font size=3D"-1">t allow fsck to <font size=3D"-1">repair the <font size=3D"-1">filesystem?<br> <br> </font></font></font></font></font></= font></font></font></font></font></font></font></font></font></font></font>= <br> </font> <div class=3D"moz-signature"> <br> <br> Get important Linux and industry-related news at: <a href=3D"http://facebook.com/dcdata">facebook.com/dcdata</a><br> <br> <font color=3D"#000000"><font face=3D"arial" size=3D"2">Kind regard= s,</font><br> <br> <font color=3D"#000000"><font face=3D"arial"><font size=3D"= 4">David Wilson</font></font></font><br> <font color=3D"#000000"><font face=3D"arial"><font size=3D"= 1">CNS,CLS, LINUX+, CLA, DCTS, LPIC3</font></font></font><br> <font color=3D"#0779b5"><font face=3D"arial"><font size=3D"= 3"><b>LinuxTech CC t/a DcData</b></font></font></font><br> <font color=3D"#0779b5"><font face=3D"arial"><font size=3D"= 1">CK number: 2001/058368/23</font></font></font><br> <table border=3D"0"> <tbody> <tr> <td width=3D"75"><font color=3D"#000000"><font face=3D"aria= l"><font size=3D"2"><b>Website:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><a class=3D"moz-txt-link-freetext" href= =3D"http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></fon= t></td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Support:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)860-1-LINUX</font></font></font></= td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Mobile:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)824147413</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Tel:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)333446100</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Fax:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)866878971</font></font></font></td=
</tr> </tbody> </table> </font></div> On 12/02/2012 11:17 AM, David Wilson wrote:<br> </div> <blockquote cite=3D"mid:50BB1CB7.8020007@dcdata.co.za" type=3D"cite"> <meta http-equiv=3D"content-type" content=3D"text/html; charset=3DISO-8859-1"> <font size=3D"-1"><font size=3D"-1">H</font>i everyone,<br> <br> <font size=3D"-1"><font size=3D"-1">Sunday fun :<font size=3D"-1">)= <br> <br> </font>One of our cr<font size=3D"-1">itical <font size=3D"-1">= Linux VMs</font> ran out of </font>disk space. I cleared up space by removing old application logs within the guest, rebooted the g<font size=3D"-1">uest and was presented with the following information message during bootup when the filesystem <font size=3D"-1">ext3/<font size=3D"-1">4</font><= /font> scan runs:<br> <font size=3D"-1"><i>Clearing orhpaned inode .....</i><br> <font size=3D"-1"><font size=3D"-1">Shortly after this message the the guest is p<font size=3D"-1">aused by oVirt and the "VM has paused due to unknown storage error" message is displayed in the <font size=3D"-1">oVirt web <font size=3D"-1">GUI.<br> <br> <font size=3D"-1">On the physical node itself, <f= ont size=3D"-1">I fo<font size=3D"-1">und t<font size=3D"-1">he following error in<font size=3D"-1"> /var/log/vdsm/vd<font size=3D"-1">sm.log:<br> <font size=3D"-1"><i>libvirtEventLoop= ::<a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"INFO::2012-12-02">INFO:= :2012-12-02</a> 09:05:50,296::libvirtvm::1965::vm= .Vm::(_onAbnormalStop) vmId=3D`23b9212c-1e25-4003-aa18-b= 1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother</i><br> <br> <font size=3D"-1">Any ideas on how to get more inf<font size=3D"-1">= ormation<font size=3D"-1"> on what exactly the <font size=3D"-1">error is?<br> </font></font></font><br> <font size=3D"-1">I've <font size=3D"-1">tried booting the guest <font size=3D"-1">i= nto r<font size=3D"-1">es<font size=3D"-1">cue mode<fo= nt size=3D"-1">, run a manual fsck <font size=3D"-1">on the logical v<font size=3D"-1">olume but each</font></= font></font></font></font></font></font></font> time the guest is automaticall paused due to the "unknown storage error".<br> <br> We are using a FC SAN for s<font size=3D"-1">torage</font></font=
, s<font size=3D"-1">o<font size=3D"-1"> what I've decided to</font></font> try is expo<fo= nt size=3D"-1">rt the disk using "dd" <font size=3D"-1">on the physical node </font>and t<font size=3D"-1">hen try repairing it <font size=3D"-1">by <font size=3D"-1">mounting it as a loop filesystem on <font size=3D"-1">a<font size=3D"-1">nother physical</font></font><= /font></font></font></font> ser<font size=3D"-1">ver<font size=3D"-1">, and then use "dd" to import it back in again.</font><br> <br> <font size=3D"-1">Is this the right approach to take to get the system back online?<br> <br> </font></font></font></font></fon= t></font></font></font></font></font></font></font></font></font></font></f= ont></font></font><br> <br> </font> <div class=3D"moz-signature">-- <br> <br> <br> Get important Linux and industry-related news at: <a moz-do-not-send=3D"true" href=3D"http://facebook.com/dcdata">face= book.com/dcdata</a><br> <br> <font color=3D"#000000"><font face=3D"arial" size=3D"2">Kind regard= s,</font><br> <br> <font color=3D"#000000"><font face=3D"arial"><font size=3D"= 4">David Wilson</font></font></font><br> <font color=3D"#000000"><font face=3D"arial"><font size=3D"= 1">CNS,CLS,
LINUX+, CLA, DCTS, LPIC3</font></font></font><br> <font color=3D"#0779b5"><font face=3D"arial"><font size=3D"= 3"><b>LinuxTech CC t/a DcData</b></font></font></font><br> <font color=3D"#0779b5"><font face=3D"arial"><font size=3D"= 1">CK number: 2001/058368/23</font></font></font><br> <table border=3D"0"> <tbody> <tr> <td width=3D"75"><font color=3D"#000000"><font face=3D"aria= l"><font size=3D"2"><b>Website:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http://www.dcdata.co.za">http://www.dcdat= a.co.za</a></font></font></font></td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Support:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)860-1-LINUX</font></font></font></= td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Mobile:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)824147413</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Tel:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)333446100</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Fax:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)866878971</font></font></font></td=
</tr> </tbody> </table> </font></div> </blockquote> <br> =20 <br><br> <p style=3D"font-family: Verdana; font-size:8pt; color:#666666;" align=3D"j= ustify"> Time to evaluate your email security provider? <a href=3D"http://www.mimeca= st.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast= =92s first ever limited promotion. </p> </body> </html> --------------070609000607040903050604--
Glad to hear it worked out. When oVirt started the vm it mounted the corrupt disk image that seemed fine, but it couldn't find the OS because of the corrupted fs, and the error caused it to pause the guest. ----- Original Message -----
From: "David Wilson" <dave@dcdata.co.za> To: users@ovirt.org Sent: Monday, December 3, 2012 10:55:37 AM Subject: Re: [Users] VM has paused due to unknown storage error
H i everyone,
Some feedback on this error , and a question I hope that someone could possibly assist with. I managed to get the guest running again by using dd and q emu-img to export the filesystem image and base image , convert to raw format and run a fsck on the loop ed part ition. When mounted as a loop filesystem, fsck easily repaired it .
Any ideas why oVirt kept pausing the guest and did no t allow fsck to repair the filesystem?
Get important Linux and industry-related news at: facebook.com/dcdata
Kind regards,
David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 LinuxTech CC t/a DcData CK number: 2001/058368/23
Website: http://www.dcdata.co.za Support: +27(0)860-1-LINUX Mobile: +27(0)824147413 Tel: +27(0)333446100 Fax: +27(0)866878971 On 12/02/2012 11:17 AM, David Wilson wrote:
H i everyone,
Sunday fun : )
One of our cr itical Linux VMs ran out of disk space. I cleared up space by removing old application logs within the guest, rebooted the g uest and was presented with the following information message during bootup when the filesystem ext3/ 4 scan runs: Clearing orhpaned inode ..... Shortly after this message the the guest is p aused by oVirt and the "VM has paused due to unknown storage error" message is displayed in the oVirt web GUI.
On the physical node itself, I fo und t he following error in /var/log/vdsm/vd sm.log: libvirtEventLoop:: INFO::2012-12-02 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop) vmId=`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother
Any ideas on how to get more inf ormation on what exactly the error is?
I've tried booting the guest into r es cue mode , run a manual fsck on the logical v olume but each time the guest is automaticall paused due to the "unknown storage error".
We are using a FC SAN for s torage , s o what I've decided to try is expo rt the disk using "dd" on the physical node and t hen try repairing it by mounting it as a loop filesystem on a nother physical ser ver , and then use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--
Get important Linux and industry-related news at: facebook.com/dcdata
Kind regards,
David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 LinuxTech CC t/a DcData CK number: 2001/058368/23
Website: http://www.dcdata.co.za Support: +27(0)860-1-LINUX Mobile: +27(0)824147413 Tel: +27(0)333446100 Fax: +27(0)866878971
Time to evaluate your email security provider? Watch the video and take advantage of Mimecast’s first ever limited promotion. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mon, Dec 3, 2012 at 10:37 AM, Yeela Kaplan wrote:
Glad to hear it worked out.
When oVirt started the vm it mounted the corrupt disk image that seemed fine, but it couldn't find the OS because of the corrupted fs, and the error caused it to pause the guest.
But if this is the case, I think this should be considered as a bug, correct? The VM should start and give an error itself, such as stopping at grub, error on pivot_root, panic, ecc... without impact on its ability to power on itself... or not?
On Mon, Dec 03, 2012 at 04:37:01AM -0500, Yeela Kaplan wrote:
Glad to hear it worked out.
When oVirt started the vm it mounted the corrupt disk image that seemed fine, but it couldn't find the OS because of the corrupted fs, and the error caused it to pause the guest.
I think we should be a bit more exact here: a VM without an installed OS does not pause. The cause of the pause was, most probably, and attempt to read from a corrupted qcow. When qemu fails to serve the guest with data due to a bug in the underlying storage, qemu stops and waits for further instructions from management. We use this feature for automatic lv-extend (on enospace error). But here, with eother error, a human intervension is required.
This is a multi-part message in MIME format. --------------050904000702030906090307 Content-Type: text/plain; charset=WINDOWS-1252; format=flowed Content-Transfer-Encoding: quoted-printable Hi everyone, Thank you for your responses. Yeela: />When oVirt started the vm it mounted the corrupt disk image that=20 seemed fine, but it couldn't find the OS because of the corrupted fs,=20 and the error caused it to pause the guest./ To clear things up it was the /var mount that had the corrupted=20 filesystem, not the guest's root filesystem. oVirt still continued to pause the guest even when I had booted the=20 guest off a CentOS DVD ISO, run rescue mode and manually activated the=20 /var logical volume. In this case I did not activate or mount the guests=20 root filesystem and only activated the guests /var filesystem so that I=20 could fsck it. fsck would run for around 5-10 minutes with the message=20 "Deleting orphaned inode......" and then oVirt would simply pause the=20 entire guest. The only information I could find was on the physical host's vdsm.log,=20 which specified the following: /libvirtEventLoop::INFO::2012-12-02=20 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop)=20 vmId=3D`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device=20 ide0-0-1 error eother /Perhaps there was another log I should have examined to see if more=20 information was provided about why oVirt was pausing the guest? // Shu: This is what I did to dd the images off, and to work around the problem: 1.) On the physical host: Created an NFS mount to another temporary=20 Linux system that had sufficient storage for the 500GB filesystem 2.) On the physical host: Used 'dd' to dump the /var filesystem's=20 logical volume to an image filevia NFS on the temporary Linux system. 3.) On the temporary Linux system that now contained the filesystem=20 image file, I ran "qemu-img info" and noticed that thefilesystem image=20 was qcow2 type and specified a backing file. 4.) On the physical host: Used 'dd' to dump the logical volume specified=20 as a backing file, to an image file via NFS on the temporary Linux system. 5.) On the temporary Linux system: Used 'qemu-img rebase' to change the=20 backing file to the local copyof the backing file image. 6.) On the temporary Linux system: Used 'qemu-img commit' to commit the=20 changes stored in the filesystem image file to the backing file image. 7.) On the temporary Linux system: Used 'qemu-img convert' to convert=20 the backing file image to raw format. 8.) Onthe temporary Linux system: Used 'losetup', 'kpart'and 'fsck' to=20 repair the backing file image. Fsck displayed the same 'Deleting=20 orphaned inode....' message but managed to continue and completed ok. 9.) On the temporary Linux system: Mounted the loop filesystem and=20 confirmed that the data was intact and was current. 10.) In the oVirt GUI: Deactivated the faulty Virtual Disk attached to=20 the guest. 11.) In the oVirt GUI: Created a new 'preallocated' Virtual Disk of=20 sufficient sizefor the guest. 12.) On the physical host: Used 'dd' to upload the raw backing file=20 image from (7) to the new logical volume. 13.) I then configured the guest to boot from the CentOS DVD ISO into=20 rescue modeto confirm that the logical volume for the guest's /var=20 filesystem was accessible and mountable. 14.) Reconfigured the guest to bootfrom it's primary Virtual Disk and=20 started up the guest. // Get important Linux and industry-related news at: facebook.com/dcdata=20 <http://facebook.com/dcdata> Kind regards, David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* =09http://www.dcdata.co.za *Support:* =09+27(0)860-1-LINUX *Mobile:* =09+27(0)824147413 *Tel:* =09+27(0)333446100 *Fax:* =09+27(0)866878971 On 12/03/2012 01:07 PM, Dan Kenigsberg wrote:
On Mon, Dec 03, 2012 at 04:37:01AM -0500, Yeela Kaplan wrote:
Glad to hear it worked out.
When oVirt started the vm it mounted the corrupt disk image that seemed = fine, but it couldn't find the OS because of the corrupted fs, and the error caused it to pause the guest. I think we should be a bit more exact here: a VM without an installed OS does not pause. The cause of the pause was, most probably, and attempt to read from a corrupted qcow. When qemu fails to serve the guest with data due to a bug in the underlying storage, qemu stops and waits for further instructions from management. We use this feature for automatic lv-extend (on enospace error). But here, with eother error, a human intervension is required.
--------------050904000702030906090307 Content-Type: text/html; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable <html> <head> <meta content=3D"text/html; charset=3Dwindows-1252" http-equiv=3D"Content-Type"> </head> <body bgcolor=3D"#FFFFFF" text=3D"#000000"> <div class=3D"moz-cite-prefix"><font size=3D"-1">Hi everyone,<br> <br> <font size=3D"-1">Thank you for your responses.<br> <br> <font size=3D"-1">Yeela:</font></font></font><br> <small><i>>When oVirt started the vm it mounted the corrupt disk image that seemed fine, but it couldn't find the OS because of the corrupted fs, and the error caused it to pause the guest.</i><br> <br> To clear things up it was the /var mount that had the corrupted filesystem, not the guest's root filesystem.<br> oVirt still continued to pause the guest even when I had booted the guest off a CentOS DVD ISO, run rescue mode and manually activated the /var logical volume. In this case I did not activate or mount the guests root filesystem and only activated the guests /var filesystem so that I could fsck it. fsck would run for around 5-10 minutes with the message "Deleting orphaned inode......" and then oVirt would simply pause the entire guest.<br=
The only information I could find was on the physical host's vdsm.log, which specified the following:<br> </small><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D= "-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"=
<font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><i>libv= irtEventLoop::<a class=3D"moz-txt-link-freetext" href=3D"INFO::2012-12-02">INFO::2012-12-02<= /a> 09:05:50,296::libvirtvm::1965::vm= .Vm::(_onAbnormalStop)
vmId=3D`23b9212c-1e25-4003-aa18-b= 1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother<br> <br> </i><font size=3D"-1"><font size=3D"-1">Perh<font size=3D"-1">aps there was another log I should have examined to see if more i<fon= t size=3D"-1">nforma<font size=3D"-1">tion was provided about wh<font size=3D"-1">y oVirt was pausing the guest?<br> <br> </font></font></font></fo= nt></font></font><i></i><font size=3D"-1"><font size=3D"-1"><br=
<br> Shu:<br> <font size=3D"-1">This is what I did to dd the images off, and to <font size=3D"-1">work around the problem<font size=3D"-1">:<br> </font></font><font size=3D"-1">1.) <font size=3D"-1">O<font size=3D"-1">n the physical host<font size=3D"-1">:</font> <font size=3D"-1">C</fo= nt></font>reated an NFS mount to another temporary Linux system that had sufficient storage for the 500GB filesystem<br> <font size=3D"-1">2.<font size=3D"-1">) O<font size=3D"-1">n</font=
the physical host<fon= t size=3D"-1">: <font size=3D"-1">U</fo= nt></font>sed 'dd' to <font size=3D"-1">dump the /var filesystem's logical vol<font size=3D"-1">ume to an image file<font size=3D"-1"> <fon= t size=3D"-1">via NFS on t<font size=3D"-1">he temporar<font size=3D"-1">y</fo= nt> Linux <font size=3D"-1">syste= m</font></font></font>.</font></font></font></font></font><br> <font size=3D"-1">3.) On the temporary Linux s<font size=3D"-1">yste= m that now contained the filesystem image file, I ran "qemu-img info</font>= </font>" and noticed that the<font size=3D"-1"> fi<font size=3D"-1">lesystem<= /font> image was <font size=3D"-1">qc<font size=3D"-1">ow2 <fo= nt size=3D"-1">type<= /font> and</font></font> specified a ba<font size=3D"-1">cking file.<br> <font size=3D"-1">4.) On t<font size=3D"-1">he physical host<fon= t size=3D"-1">: Used <font size=3D"-1">'dd' to dump <font size=3D"-1">the lo<font size=3D"-1">gical volume specified as a backing file, to an image file </font></fon= t></font></font></font></font></font></font></font></font></font></font></f= ont></font></font></font></font></font></font></font></font></font></font><= /font></font></font></font></font></font></font><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><= font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"=
<font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><fo= nt size=3D"-1"><font size=3D"-1"= <font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1"><font size=3D"-1">via NFS on t<font size=3D"-1">he temporar<font size=3D"-1">y</fo= nt> Linux <font size=3D"-1">syste= m.<br> <font size=3D"-1">5.<fo= nt size=3D"-1">) On the <font size=3D"-1">tempo= rary Linux system: Used '<font size=3D"-1">qemu-= img reba<font size=3D"-1">se</f= ont>' to change the backing file <fon= t size=3D"-1">to the local copy<fo= nt size=3D"-1"> of the back<font size=3D"-1">ing file image.</font= </font></font></font></font></font></font><br> <font size=3D"-1">6.) On the temporary Linux system: Used 'qemu-img commit' to commit the changes stored in the filesystem image file to the backing file image.<br> <font size=3D"-1">7.) O<font size=3D"-1">n the temporary Linux sy<font size=3D"-1">ste<f= ont size=3D"-1">m<fon= t size=3D"-1">: U<f= ont size=3D"-1">sed 'qemu-img convert' to convert the backing file image to raw format.<br> <font size=3D"-1">8.) On<font size=3D"-1"> <fon= t size=3D"-1">the <font size=3D"-1">tempo= rary Linux system: <font size=3D"-1">Used 'losetup', <font size=3D"-1">'kpar= t'<font size=3D"-1"><font size=3D"-1"> and </font>'fsck'</fo= nt></font></font></font></font></font></font> to <font size=3D"-1">repai= r the backing file image. Fsck displayed t<font size=3D"-1">he same 'Del<font size=3D"-1">eting <font size=3D"-1">orpha= ned in<font size=3D"-1">ode<f= ont size=3D"-1">....' <font size=3D"-1">messa= ge</font></font></font></font></font></font></font> <font size=3D"-1">but managed to <font size=3D"-1">con<f= ont size=3D"-1">tinue and completed ok.<br> <font size=3D"-1">9.) O<font size=3D"-1">n the tempo<font size=3D"-1">rary Linux system: <font size=3D"-1">Mo<fo= nt size=3D"-1">unted the loop filesystem and confirmed that <font size=3D"-1">the <font size=3D"-1">data was intact and was current.</fon= t></font></font></font></font></font></font><br> </font></font></f= ont><font size=3D"-1"><font size=3D"-1">10</f= ont>.) I<font size=3D"-1">n <fo= nt size=3D"-1">the oVirt GUI: Deactivated the faulty Virtual <font size=3D"-1">D</fo= nt>isk attached to the guest.</font>= </font></font><br> <font size=3D"-1"><font size=3D"-1">11</f= ont>.) In the oVirt GUI: Created a new 'preallocated<fon= t size=3D"-1">' <fo= nt size=3D"-1">Virtu= al Disk <font size=3D"-1">of sufficient size<font size=3D"-1"> for the guest.<br> <font size=3D"-1">1<fon= t size=3D"-1">2</fo= nt>.) O<font size=3D"-1">n the physical host: Used <font size=3D"-1">'dd' to upload the <font size=3D"-1">raw ba<font size=3D"-1">cking file image from (7) to the new lo<font size=3D"-1">gical volume.</font></f= ont></font></font></font></font><br> <font size=3D"-1">1<fon= t size=3D"-1">3</fo= nt>.) <font size=3D"-1">I then conf<font size=3D"-1">igure= d the guest to boot from the CentOS <font size=3D"-1">D<fon= t size=3D"-1">VD ISO into res<font size=3D"-1">cue mode<font size=3D"-1"> to confirm that the <font size=3D"-1">lo<fo= nt size=3D"-1">gical <font size=3D"-1">volum= e <font size=3D"-1"><font size=3D"-1">for th<font size=3D"-1">e <fo= nt size=3D"-1"><font size=3D"-1">guest= <font size=3D"-1">'s /var </font></fon= t>filesystem</font></font> was accessi<font size=3D"-1">ble and mountable.<br=
<font size=3D"-1">1<fon= t size=3D"-1">4</fo= nt>.) <font size=3D"-1">Recon= figured the guest to boo<font size=3D"-1">t<fon= t size=3D"-1"> from it's <font size=3D"-1">prima= ry <font size=3D"-1">Virtu= al Disk</font></font=
and sta<font size=3D"-1">rted up the guest.</fo= nt></font></font></font></font><br> </font></font></f= ont></font></font></font></font></font></font></font></font></font></font><= /font></font></font></font></font></font></font></font></font></font></font=
<br> </font><br> </font></font></f= ont></font></font></font></font></font></font></font></font></font></font><= /font></font></font></font></font></font></font></font></font></font></font= </font></font></font></font></font></font></font></font></font></font></fo= nt></font></font></font></font></font></font></font></font><i></i></font></= font></font></font></font></font></font></font></font></font></font></font>= </font></font></font></font></font><br> <br> <br> <br> <br> <div class=3D"moz-signature"> <br> <br> Get important Linux and industry-related news at: <a href=3D"http://facebook.com/dcdata">facebook.com/dcdata</a><br> <br> <font color=3D"#000000"><font face=3D"arial" size=3D"2">Kind regard= s,</font><br> <br> =A0<font color=3D"#000000"><font face=3D"arial"><font size=3D"4">= David Wilson</font></font></font><br> =A0<font color=3D"#000000"><font face=3D"arial"><font size=3D"1">= CNS,CLS, LINUX+, CLA, DCTS, LPIC3</font></font></font><br> =A0<font color=3D"#0779b5"><font face=3D"arial"><font size=3D"3">= <b>LinuxTech CC t/a DcData</b></font></font></font><br> =A0<font color=3D"#0779b5"><font face=3D"arial"><font size=3D"1">= CK number: 2001/058368/23</font></font></font><br> <table border=3D"0"> <tbody> <tr> <td width=3D"75"><font color=3D"#000000"><font face=3D"aria= l"><font size=3D"2"><b>Website:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><a class=3D"moz-txt-link-freetext" href= =3D"http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></fon= t></td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Support:</b></font></font></font></td=
<td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)860-1-LINUX</font></font></font></= td> </tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Mobile:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)824147413</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Tel:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)333446100</font></font></font></td=
</tr> <tr> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2"><b>Fax:</b></font></font></font></td> <td><font color=3D"#000000"><font face=3D"arial"><font size=3D"2">+27(0)866878971</font></font></font></td=
</tr> </tbody> </table> </font></div> On 12/03/2012 01:07 PM, Dan Kenigsberg wrote:<br> </div> <blockquote cite=3D"mid:20121203110737.GD20719@redhat.com" type=3D"cite= "> <pre wrap=3D"">On Mon, Dec 03, 2012 at 04:37:01AM -0500, Yeela Kaplan= wrote: </pre> <blockquote type=3D"cite"> <pre wrap=3D"">Glad to hear it worked out. When oVirt started the vm it mounted the corrupt disk image that seemed fin= e, =20 but it couldn't find the OS because of the corrupted fs, and the error caused it to pause the guest. </pre> </blockquote> <pre wrap=3D""> I think we should be a bit more exact here: a VM without an installed OS does not pause. The cause of the pause was, most probably, and attempt to read from a corrupted qcow. When qemu fails to serve the guest with data due to a bug in the underlying storage, qemu stops and waits for further instructions from management. We use this feature for automatic lv-extend (on enospace error). But here, with eother error, a human intervension is required. </pre> </blockquote> <br> =20 <br><br> <p style=3D"font-family: Verdana; font-size:8pt; color:#666666;" align=3D"j= ustify"> Time to evaluate your email security provider? <a href=3D"http://www.mimeca= st.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast= =92s first ever limited promotion. </p> </body> </html> --------------050904000702030906090307--
Hi David, You can also check the libvirt log. Please send me the vdsm and libvirt logs so I can look at it too. thanks. ----- Original Message -----
From: "David Wilson" <dave@dcdata.co.za> To: "Dan Kenigsberg" <danken@redhat.com> Cc: "Yeela Kaplan" <ykaplan@redhat.com>, users@ovirt.org Sent: Monday, December 3, 2012 5:51:26 PM Subject: Re: [Users] VM has paused due to unknown storage error
Hi everyone,
Thank you for your responses.
Yeela:
When oVirt started the vm it mounted the corrupt disk image that seemed fine, but it couldn't find the OS because of the corrupted fs, and the error caused it to pause the guest.
To clear things up it was the /var mount that had the corrupted filesystem, not the guest's root filesystem. oVirt still continued to pause the guest even when I had booted the guest off a CentOS DVD ISO, run rescue mode and manually activated the /var logical volume. In this case I did not activate or mount the guests root filesystem and only activated the guests /var filesystem so that I could fsck it. fsck would run for around 5-10 minutes with the message "Deleting orphaned inode......" and then oVirt would simply pause the entire guest. The only information I could find was on the physical host's vdsm.log, which specified the following: libvirtEventLoop:: INFO::2012-12-02 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop) vmId=`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother
Perh aps there was another log I should have examined to see if more i nforma tion was provided about wh y oVirt was pausing the guest?
Shu: This is what I did to dd the images off, and to work around the problem : 1.) O n the physical host : C reated an NFS mount to another temporary Linux system that had sufficient storage for the 500GB filesystem 2. ) O n the physical host : U sed 'dd' to dump the /var filesystem's logical vol ume to an image file via NFS on t he temporar y Linux system . 3.) On the temporary Linux s ystem that now contained the filesystem image file, I ran "qemu-img info " and noticed that the fi lesystem image was qc ow2 type and specified a ba cking file. 4.) On t he physical host : Used 'dd' to dump the lo gical volume specified as a backing file, to an image file via NFS on t he temporar y Linux system. 5. ) On the temporary Linux system: Used ' qemu-img reba se ' to change the backing file to the local copy of the back ing file image. 6.) On the temporary Linux system: Used 'qemu-img commit' to commit the changes stored in the filesystem image file to the backing file image. 7.) O n the temporary Linux sy ste m : U sed 'qemu-img convert' to convert the backing file image to raw format. 8.) On the temporary Linux system: Used 'losetup', 'kpart' and 'fsck' to repair the backing file image. Fsck displayed t he same 'Del eting orphaned in ode ....' message but managed to con tinue and completed ok. 9.) O n the tempo rary Linux system: Mo unted the loop filesystem and confirmed that the data was intact and was current. 10 .) I n the oVirt GUI: Deactivated the faulty Virtual D isk attached to the guest. 11 .) In the oVirt GUI: Created a new 'preallocated ' Virtual Disk of sufficient size for the guest. 1 2 .) O n the physical host: Used 'dd' to upload the raw ba cking file image from (7) to the new lo gical volume. 1 3 .) I then conf igured the guest to boot from the CentOS D VD ISO into res cue mode to confirm that the lo gical volume for th e guest 's /var filesystem was accessi ble and mountable. 1 4 .) Reconfigured the guest to boo t from it's primary Virtual Disk and sta rted up the guest.
Get important Linux and industry-related news at: facebook.com/dcdata
Kind regards,
David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 LinuxTech CC t/a DcData CK number: 2001/058368/23
Website: http://www.dcdata.co.za Support: +27(0)860-1-LINUX Mobile: +27(0)824147413 Tel: +27(0)333446100 Fax: +27(0)866878971 On 12/03/2012 01:07 PM, Dan Kenigsberg wrote:
On Mon, Dec 03, 2012 at 04:37:01AM -0500, Yeela Kaplan wrote:
Glad to hear it worked out.
When oVirt started the vm it mounted the corrupt disk image that seemed fine, but it couldn't find the OS because of the corrupted fs, and the error caused it to pause the guest. I think we should be a bit more exact here: a VM without an installed OS does not pause. The cause of the pause was, most probably, and attempt to read from a corrupted qcow. When qemu fails to serve the guest with data due to a bug in the underlying storage, qemu stops and waits for further instructions from management. We use this feature for automatic lv-extend (on enospace error). But here, with eother error, a human intervension is required.
Time to evaluate your email security provider? Watch the video and take advantage of Mimecast’s first ever limited promotion.
This is a multi-part message in MIME format. --------------040205020003020503010804 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit David Wilson:
Hi everyone,
Some feedbackon this error, and a question I hope that someone could possibly assist with. I managed to get the guest running again byusing dd and qemu-img to export the filesystem image and base image, convert to raw format and run a fsck on the looped partition. I am happy you make that work. Can you describe more details about how you dd the guest image? And how did you use qemu-img to export the filesystem image and base image.
When mounted as a loop filesystem, fsck easily repaired it.
Any ideas why oVirt kept pausing the guest and did not allow fsck to repair the filesystem?
Get important Linux and industry-related news at: facebook.com/dcdata <http://facebook.com/dcdata>
Kind regards,
David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* http://www.dcdata.co.za *Support:* +27(0)860-1-LINUX *Mobile:* +27(0)824147413 *Tel:* +27(0)333446100 *Fax:* +27(0)866878971
On 12/02/2012 11:17 AM, David Wilson wrote:
Hi everyone,
Sunday fun :)
One of our critical Linux VMs ran out of disk space. I cleared up space by removing old application logs within the guest, rebooted the guest and was presented with the following information message during bootup when the filesystem ext3/4 scan runs: /Clearing orhpaned inode ...../ Shortly after this message the the guest is paused by oVirt and the "VM has paused due to unknown storage error" message is displayed in the oVirt web GUI.
On the physical node itself, I found the following error in/var/log/vdsm/vdsm.log: /libvirtEventLoop::INFO::2012-12-02 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop) vmId=`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother/
Any ideas on how to get more informationon what exactly the error is?
I've tried booting the guest into rescue mode, run a manual fsck on the logical volume but each time the guest is automaticall paused due to the "unknown storage error".
We are using a FC SAN for storage, sowhat I've decided to try is export the disk using "dd" on the physical node and then try repairing it by mounting it as a loop filesystem on another physical server, and then use "dd" to import it back in again.
Is this the right approach to take to get the system back online?
--
Get important Linux and industry-related news at: facebook.com/dcdata <http://facebook.com/dcdata>
Kind regards,
David Wilson CNS,CLS, LINUX+, CLA, DCTS, LPIC3 *LinuxTech CC t/a DcData* CK number: 2001/058368/23 *Website:* http://www.dcdata.co.za *Support:* +27(0)860-1-LINUX *Mobile:* +27(0)824147413 *Tel:* +27(0)333446100 *Fax:* +27(0)866878971
Time to evaluate your email security provider? Watch the video <http://www.mimecast.com/emailsecurity/> and take advantage of Mimecast's first ever limited promotion.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- --- ?? Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming@cn.ibm.com or shuming@linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC --------------040205020003020503010804 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">David Wilson:<br> </div> <blockquote cite="mid:50BC6909.7050602@dcdata.co.za" type="cite"> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> <div class="moz-cite-prefix"><font size="-1">H<font size="-1">i</font> everyone,<br> <br> <font size="-1"><font size="-1">Some feedback<font size="-1"> on this error<font size="-1">, and a <font size="-1">question I hope that someone could possibly assist with.</font></font><br> <font size="-1">I managed to <font size="-1">get the guest running again by<font size="-1"> using dd <font size="-1">and q<font size="-1">emu-img to export the filesystem image and base image<font size="-1">, convert to raw format and run a fsck on the <font size="-1">loop<font size="-1">ed part<font size="-1">ition.<br> </font></font></font></font></font></font></font></font></font></font></font></font></font></div> </blockquote> I am happy you make that work. Can you describe more details about how you dd the guest image? And how did you use qemu-img to export the filesystem image and base image.<br> <br> <blockquote cite="mid:50BC6909.7050602@dcdata.co.za" type="cite"> <div class="moz-cite-prefix"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"><font size="-1"> <font size="-1">When mounted as a loop filesystem, fsck easily repaired <font size="-1">it<font size="-1">.</font></font><br> <br> </font><font size="-1">Any ideas why oVirt kept pausing the guest and did no<font size="-1">t allow fsck to <font size="-1">repair the <font size="-1">filesystem?<br> <br> </font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font><br> </font> <div class="moz-signature"> <br> <br> Get important Linux and industry-related news at: <a moz-do-not-send="true" href="http://facebook.com/dcdata">facebook.com/dcdata</a><br> <br> <font color="#000000"><font face="arial" size="2">Kind regards,</font><br> <br> <font color="#000000"><font face="arial"><font size="4">David Wilson</font></font></font><br> <font color="#000000"><font face="arial"><font size="1">CNS,CLS, LINUX+, CLA, DCTS, LPIC3</font></font></font><br> <font color="#0779b5"><font face="arial"><font size="3"><b>LinuxTech CC t/a DcData</b></font></font></font><br> <font color="#0779b5"><font face="arial"><font size="1">CK number: 2001/058368/23</font></font></font><br> <table border="0"> <tbody> <tr> <td width="75"><font color="#000000"><font face="arial"><font size="2"><b>Website:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2"><a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Support:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)860-1-LINUX</font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Mobile:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)824147413</font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Tel:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)333446100</font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Fax:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)866878971</font></font></font></td> </tr> </tbody> </table> </font></div> On 12/02/2012 11:17 AM, David Wilson wrote:<br> </div> <blockquote cite="mid:50BB1CB7.8020007@dcdata.co.za" type="cite"> <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> <font size="-1"><font size="-1">H</font>i everyone,<br> <br> <font size="-1"><font size="-1">Sunday fun :<font size="-1">)<br> <br> </font>One of our cr<font size="-1">itical <font size="-1">Linux VMs</font> ran out of </font>disk space. I cleared up space by removing old application logs within the guest, rebooted the g<font size="-1">uest and was presented with the following information message during bootup when the filesystem <font size="-1">ext3/<font size="-1">4</font></font> scan runs:<br> <font size="-1"><i>Clearing orhpaned inode .....</i><br> <font size="-1"><font size="-1">Shortly after this message the the guest is p<font size="-1">aused by oVirt and the "VM has paused due to unknown storage error" message is displayed in the <font size="-1">oVirt web <font size="-1">GUI.<br> <br> <font size="-1">On the physical node itself, <font size="-1">I fo<font size="-1">und t<font size="-1">he following error in<font size="-1"> /var/log/vdsm/vd<font size="-1">sm.log:<br> <font size="-1"><i>libvirtEventLoop::<a moz-do-not-send="true" class="moz-txt-link-freetext" href="INFO::2012-12-02">INFO::2012-12-02</a> 09:05:50,296::libvirtvm::1965::vm.Vm::(_onAbnormalStop) vmId=`23b9212c-1e25-4003-aa18-b1e819bf6bb1`::abnormal vm stop device ide0-0-1 error eother</i><br> <br> <font size="-1">Any ideas on how to get more inf<font size="-1">ormation<font size="-1"> on what exactly the <font size="-1">error is?<br> </font></font></font><br> <font size="-1">I've <font size="-1">tried booting the guest <font size="-1">into r<font size="-1">es<font size="-1">cue mode<font size="-1">, run a manual fsck <font size="-1">on the logical v<font size="-1">olume but each</font></font></font></font></font></font></font></font> time the guest is automaticall paused due to the "unknown storage error".<br> <br> We are using a FC SAN for s<font size="-1">torage</font></font>, s<font size="-1">o<font size="-1"> what I've decided to</font></font> try is expo<font size="-1">rt the disk using "dd" <font size="-1">on the physical node </font>and t<font size="-1">hen try repairing it <font size="-1">by <font size="-1">mounting it as a loop filesystem on <font size="-1">a<font size="-1">nother physical</font></font></font></font></font></font> ser<font size="-1">ver<font size="-1">, and then use "dd" to import it back in again.</font><br> <br> <font size="-1">Is this the right approach to take to get the system back online?<br> <br> </font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font></font><br> <br> </font> <div class="moz-signature">-- <br> <br> <br> Get important Linux and industry-related news at: <a moz-do-not-send="true" href="http://facebook.com/dcdata">facebook.com/dcdata</a><br> <br> <font color="#000000"><font face="arial" size="2">Kind regards,</font><br> <br> <font color="#000000"><font face="arial"><font size="4">David Wilson</font></font></font><br> <font color="#000000"><font face="arial"><font size="1">CNS,CLS, LINUX+, CLA, DCTS, LPIC3</font></font></font><br> <font color="#0779b5"><font face="arial"><font size="3"><b>LinuxTech CC t/a DcData</b></font></font></font><br> <font color="#0779b5"><font face="arial"><font size="1">CK number: 2001/058368/23</font></font></font><br> <table border="0"> <tbody> <tr> <td width="75"><font color="#000000"><font face="arial"><font size="2"><b>Website:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2"><a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.dcdata.co.za">http://www.dcdata.co.za</a></font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Support:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)860-1-LINUX</font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Mobile:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)824147413</font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Tel:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)333446100</font></font></font></td> </tr> <tr> <td><font color="#000000"><font face="arial"><font size="2"><b>Fax:</b></font></font></font></td> <td><font color="#000000"><font face="arial"><font size="2">+27(0)866878971</font></font></font></td> </tr> </tbody> </table> </font></div> </blockquote> <br> <br> <br> <p style="font-family: Verdana; font-size:8pt; color:#666666;" align="justify"> Time to evaluate your email security provider? <a moz-do-not-send="true" href="http://www.mimecast.com/emailsecurity/">Watch the video</a> and take advantage of Mimecast’s first ever limited promotion. </p> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <pre class="moz-signature" cols="72">-- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: <a class="moz-txt-link-abbreviated" href="mailto:shuming@cn.ibm.com">shuming@cn.ibm.com</a> or <a class="moz-txt-link-abbreviated" href="mailto:shuming@linux.vnet.ibm.com">shuming@linux.vnet.ibm.com</a> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC</pre> </body> </html> --------------040205020003020503010804--
participants (5)
-
Dan Kenigsberg -
David Wilson -
Gianluca Cecchi -
Shu Ming -
Yeela Kaplan