Re: [ovirt-users] disaster recovery

Monday, 22 June 2015

This is a multi-part message in MIME format.
--------------050507020202060701050903
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable

Sure that ovirt is not directly concerned but this has been done=20
following the foreman integration feature. So a warning doing this in a=20
fc environnement could be very preventive.
My question was rather : now is there a possibility to recover this=20
disk? After all, it is only a LVM disk with one VG per LUN and as much=20
LV as much as existing VMs. Recovering LVM metadata might be enough?

Le 21/06/2015 20:23, Dan Yasny a =C3=A9crit :
...

 Anaconda and Kickstart are dangerous in FC environments and have been=20
 known to wipe all kinds of data if zoning wasn't properly done prior=20
 to os deployment to new hosts.

 Nothing to do with oVirt, it's a common mistake people make at least=20
 once before they step on this specific rake.

 On Jun 21, 2015 2:09 PM, "Nathana=C3=ABl Blanchet" <blanchet(a)abes.fr=20
 <mailto:blanchet@abes.fr>> wrote:

     In a traditional installation of an ovirt host, it seems that
     provisionning the OS is the first step before talking about vdsm.
     I didn't go further, the problem is that this host saw the
     production LUN and the kickstart took all the lun into the same VG
     with a XFS format.
     Yes it is an error from myself for not having dissociated the lun.
     Yes it is an error from myself for not having taken care of having
     snapshot lun level backup.
     But now what should I do?
     My idea was to use pvreduce to unlabel the LUN, and then using the
     /etc/lvm/backup/lun for recovering physical volume metadata on
     that lun (http://www.tldp.org/HOWTO/LVM-HOWTO/recovermetadata.html)

     Le 21/06/2015 12:02, mots a =C3=A9crit :

         Let me get this straight:

         You added a new host and instead of letting VDSM manage the VM
         storage you added it manually, completely independent from
         oVirt, during an unattended installation of the host OS? In a
         production environment?

         Why?
           -----Urspr=C3=BCngliche Nachricht-----

             Von:Nathana=C3=ABl Blanchet <blanchet(a)abes.fr
             <mailto:blanchet@abes.fr>>
             Gesendet: Fre 19 Juni 2015 14:09
             An: users(a)ovirt.org <mailto:users@ovirt.org>
             Betreff: [ovirt-users] disaster recovery

             Hello all,

             Here is what can happen as a nightmare for a sysadmin:

             I installed a new host to my pool of pre existing ovirt
             hosts with a
             kickstart file and a default partitionning. I used to do
             this with vms
             that usually get one local disk.
             But..
             This time this host was attached to several existing ovirt
             (production)
             lun and anaconda gathered all existing disk (local and
             lun) into the
             same VG (lv_home) et formatted them with XFS....
             In the webadmin, the domain storage became unvailaible,
             vms were still
             up (thankgoodness), but it was impossible to interact with
             them. If I
             stopped them, it was impossible to reboot them. If I
             launch lvs command,
             some hosts can still see the ovirt LV, but other see only
             the lv_home
             while /dev/[idofvms] are still present.
             So it was my chance that vms were still present, and I
             began to export
             them with a tar at file system level so as to import them
             in a new
             domain storage.

             It seems that data are still present because the vms are
             still running.
             So my question is really : instead of this difficult
             export step, is
             there a way to recover the initial format of the ovirt lun
             so as to make
             the lvm index come back on the disk?

             Any help or disaster experience would be much appreciated.

             --=20
             Nathana=C3=ABl Blanchet

             Supervision r=C3=A9seau
             P=C3=B4le Infrastrutures Informatiques
             227 avenue Professeur-Jean-Louis-Viala
             34193 MONTPELLIER CEDEX 5
             T=C3=A9l. 33 (0)4 67 54 84 55
             Fax  33 (0)4 67 54 84 14
             blanchet(a)abes.fr <mailto:blanchet@abes.fr>

             _______________________________________________
             Users mailing list
             Users(a)ovirt.org <mailto:Users@ovirt.org>
             http://lists.ovirt.org/mailman/listinfo/users

     _______________________________________________
     Users mailing list
     Users(a)ovirt.org <mailto:Users@ovirt.org>
     http://lists.ovirt.org/mailman/listinfo/users

--=20
Nathana=C3=ABl Blanchet

Supervision r=C3=A9seau
P=C3=B4le Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5 =09
T=C3=A9l. 33 (0)4 67 54 84 55
Fax  33 (0)4 67 54 84 14
blanchet(a)abes.fr

--------------050507020202060701050903
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html>
  <head>
    <meta content=3D"text/html; charset=3Dutf-8"
http-equiv=3D"Content-Ty=
pe">
  </head>
  <body text=3D"#000000" bgcolor=3D"#FFFFFF">
    Sure that ovirt is not directly concerned but this has been done
    following the foreman integration feature. So a warning doing this
    in a fc environnement could be very preventive.<br>
    My question was rather : now is there a possibility to recover this
    disk? After all, it is only a LVM disk with one VG per LUN and as
    much LV as much as existing VMs. Recovering LVM metadata might be
    enough?<br>
    <br>
    <div class=3D"moz-cite-prefix">Le 21/06/2015 20:23, Dan Yasny a
      =C3=A9crit=C2=A0:<br>
    </div>
    <blockquote
cite=3D"mid:CALLXwb6H_apELuw=3Dh+uRA9er_Mn15Kw-46f=3DFx5VsGoTSdyJ6w@mail.=
gmail.com"
      type=3D"cite">
      <p dir=3D"ltr">Anaconda and Kickstart are dangerous in FC
        environments and have been known to wipe all kinds of data if
        zoning wasn't properly done prior to os deployment to new hosts.
      </p>
      <p dir=3D"ltr">Nothing to do with oVirt, it's a common mistake
        people make at least once before they step on this specific
        rake.</p>
      <div class=3D"gmail_quote">On Jun 21, 2015 2:09 PM,
"Nathana=C3=ABl
        Blanchet" &lt;<a moz-do-not-send=3D"true"
          href=3D"mailto:blanchet@abes.fr">blanchet@abes.fr</a>&gt;
wrote=
:<br
          type=3D"attribution">
        <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
          .8ex;border-left:1px #ccc solid;padding-left:1ex">In a
          traditional installation of an ovirt host, it seems that
          provisionning the OS is the first step before talking about
          vdsm. I didn't go further, the problem is that this host saw
          the production LUN and the kickstart took all the lun into the
          same VG with a XFS format.<br>
          Yes it is an error from myself for not having dissociated the
          lun.<br>
          Yes it is an error from myself for not having taken care of
          having snapshot lun level backup.<br>
          But now what should I do?<br>
          My idea was to use pvreduce to unlabel the LUN, and then using
          the /etc/lvm/backup/lun for recovering physical volume
          metadata on that lun (<a moz-do-not-send=3D"true"
            href=3D"http://www.tldp.org/HOWTO/LVM-HOWTO/recovermetadata.h=
tml"
            rel=3D"noreferrer"
target=3D"_blank">http://www.tldp.org/HOWT=
O/LVM-HOWTO/recovermetadata.html</a>)<br>
          <br>
          Le 21/06/2015 12:02, mots a =C3=A9crit :<br>
          <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            Let me get this straight:<br>
            <br>
            You added a new host and instead of letting VDSM manage the
            VM storage you added it manually, completely independent
            from oVirt, during an unattended installation of the host
            OS? In a production environment?<br>
            <br>
            Why?<br>
            =C2=A0 -----Urspr=C3=BCngliche Nachricht-----<br>
            <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              Von:Nathana=C3=ABl Blanchet &lt;<a
moz-do-not-send=3D"true"
                href=3D"mailto:blanchet@abes.fr"
target=3D"_blank">blanch=
et(a)abes.fr&lt;/a&gt;&amp;gt;&lt;br&gt;
              Gesendet: Fre 19 Juni 2015 14:09<br>
              An: <a moz-do-not-send=3D"true"
                href=3D"mailto:users@ovirt.org"
target=3D"_blank">users@o=
virt.org</a><br>
              Betreff: [ovirt-users] disaster recovery<br>
              <br>
              Hello all,<br>
              <br>
              Here is what can happen as a nightmare for a sysadmin:<br>
              <br>
              I installed a new host to my pool of pre existing ovirt
              hosts with a<br>
              kickstart file and a default partitionning. I used to do
              this with vms<br>
              that usually get one local disk.<br>
              But..<br>
              This time this host was attached to several existing ovirt
              (production)<br>
              lun and anaconda gathered all existing disk (local and
              lun) into the<br>
              same VG (lv_home) et formatted them with XFS....<br>
              In the webadmin, the domain storage became unvailaible,
              vms were still<br>
              up (thankgoodness), but it was impossible to interact with
              them. If I<br>
              stopped them, it was impossible to reboot them. If I
              launch lvs command,<br>
              some hosts can still see the ovirt LV, but other see only
              the lv_home<br>
              while /dev/[idofvms] are still present.<br>
              So it was my chance that vms were still present, and I
              began to export<br>
              them with a tar at file system level so as to import them
              in a new<br>
              domain storage.<br>
              <br>
              It seems that data are still present because the vms are
              still running.<br>
              So my question is really : instead of this difficult
              export step, is<br>
              there a way to recover the initial format of the ovirt lun
              so as to make<br>
              the lvm index come back on the disk?<br>
              <br>
              Any help or disaster experience would be much appreciated.<=
br>
              <br>
              -- <br>
              Nathana=C3=ABl Blanchet<br>
              <br>
              Supervision r=C3=A9seau<br>
              P=C3=B4le Infrastrutures Informatiques<br>
              227 avenue Professeur-Jean-Louis-Viala<br>
              34193 MONTPELLIER CEDEX 5=C2=A0 =C2=A0 =C2=A0 =C2=A0<br>
              T=C3=A9l. 33 (0)4 67 54 84 55<br>
              Fax=C2=A0 33 (0)4 67 54 84 14<br>
              <a moz-do-not-send=3D"true"
href=3D"mailto:blanchet@abes.fr=
"
                target=3D&quot;_blank&quot;&gt;blanchet(a)abes.fr&lt;/a&gt;&lt;br&gt;
              <br>
              _______________________________________________<br>
              Users mailing list<br>
              <a moz-do-not-send=3D"true"
href=3D"mailto:Users@ovirt.org"
                target=3D&quot;_blank&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br&gt;
              <a moz-do-not-send=3D"true"
                href=3D"http://lists.ovirt.org/mailman/listinfo/users"
                rel=3D"noreferrer"
target=3D"_blank">http://lists.ovirt.o=
rg/mailman/listinfo/users</a><br>
              <br>
            </blockquote>
          </blockquote>
          <br>
          _______________________________________________<br>
          Users mailing list<br>
          <a moz-do-not-send=3D"true"
href=3D"mailto:Users@ovirt.org"
            target=3D&quot;_blank&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br&gt;
          <a moz-do-not-send=3D"true"
            href=3D"http://lists.ovirt.org/mailman/listinfo/users"
            rel=3D"noreferrer"
target=3D"_blank">http://lists.ovirt.org/m=
ailman/listinfo/users</a><br>
        </blockquote>
      </div>
    </blockquote>
    <br>
    <pre class=3D"moz-signature" cols=3D"72">--=20
Nathana=C3=ABl Blanchet

Supervision r=C3=A9seau
P=C3=B4le Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5 =09
T=C3=A9l. 33 (0)4 67 54 84 55
Fax  33 (0)4 67 54 84 14
<a class=3D"moz-txt-link-abbreviated"
href=3D"mailto:blanchet@abes.fr">bl=
anchet(a)abes.fr&lt;/a&gt; </pre>
  </body>
</html>

--------------050507020202060701050903--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] disaster recovery