Re: [ovirt-users] disaster recovery
------=_Part_90_1900995351.1434880991903 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Let me get this straight: You added a new host and instead of letting VDSM manage the VM storage you added it manually, completely independent from oVirt, during an unattended installation of the host OS=3F In a production environment=3F Why=3F =20 -----Urspr=C3=BCngliche Nachricht-----
Von:Nathana=C3=ABl Blanchet <blanchet@abes.fr> Gesendet: Fre 19 Juni 2015 14:09 An: users@ovirt.org Betreff: [ovirt-users] disaster recovery =20 Hello all, =20 Here is what can happen as a nightmare for a sysadmin: =20 I installed a new host to my pool of pre existing ovirt hosts with a=20 kickstart file and a default partitionning. I used to do this with vms=20 that usually get one local disk. But.. This time this host was attached to several existing ovirt (production)=20 lun and anaconda gathered all existing disk (local and lun) into the=20 same VG (lv_home) et formatted them with XFS.... In the webadmin, the domain storage became unvailaible, vms were still=20 up (thankgoodness), but it was impossible to interact with them. If I=20 stopped them, it was impossible to reboot them. If I launch lvs command,=20 some hosts can still see the ovirt LV, but other see only the lv_home=20 while /dev/[idofvms] are still present. So it was my chance that vms were still present, and I began to export=20 them with a tar at file system level so as to import them in a new=20 domain storage. =20 It seems that data are still present because the vms are still running. So my question is really : instead of this difficult export step, is=20 there a way to recover the initial format of the ovirt lun so as to make=20 the lvm index come back on the disk=3F =20 Any help or disaster experience would be much appreciated. =20 --=20 Nathana=C3=ABl Blanchet =20 Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr =20 =20 _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users =20
------=_Part_90_1900995351.1434880991903 Content-Type: application/pgp-signature; name=signature.asc Content-Transfer-Encoding: 7bit Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: CIPHERMAIL (2.9.0-0) iQIcBAEBCAAGBQJVhovfAAoJEJ44dql+IcLKmV4QAIVOz1GZYMx5TM1Sd+By3J5W 9UHwzcsM11eFol8QI5iiQZaFrZvAUETTPsOlnelB9CpUkFv4auFjgsvedTRhplU8 41YdWMfV7YzFDX2Jd2Ou3kiV6SXc40P9LwM5rMAmoip8IhgND7jtue7NVkmKsuvO HSLKb7AvG4+lFwLEuPJlRB0Xe1aSbEbrlHF+G2KhUC4i2fclHePqgBFUrbqGUuiV p0uov92/TVuJSkahll3dWPkUGGRG9Z/gOVwZ2yUv1FnDpPns+KtwsQ0rtIWBp69o u88ntyMw4BP+Sq0haX0kUjFDELFUw52Q0YSw6800I+hQbGQGYh3kLTm69rsVI2sr Pqr7qkH8ELEQb+iXRaWqNV+P2onM8ni2WaX51yuHAonC+3CT31qYrDAiFCzAZ9DD fV1hHI0a+tGLrvju4jMyplzDchAfAUaKOAyP0EUyKKcljAQcBJcCWiireYfWM4IU 39nhb8DRvBQWovUjz6nvdAERNtM+R1ZsONKQYdrgAD6AzZrIvTF8feJnMzErdHEo HNSopYQZxv6gZkafnFG5aCA/SfvwKtzfP57hrVgr6RnOYApyHoTH05ii1iaKErWA 5FHzs1tJ7k7RnKcsswNVMijQJ4Qo5YdMyLeqh5QFWuQmEMWvrm85Uc451rAOxHJJ hVZ22QUIBejbQpXgM2ds =QG8e -----END PGP SIGNATURE----- ------=_Part_90_1900995351.1434880991903--
In a traditional installation of an ovirt host, it seems that provisionning the OS is the first step before talking about vdsm. I didn't go further, the problem is that this host saw the production LUN and the kickstart took all the lun into the same VG with a XFS format. Yes it is an error from myself for not having dissociated the lun. Yes it is an error from myself for not having taken care of having snapshot lun level backup. But now what should I do? My idea was to use pvreduce to unlabel the LUN, and then using the /etc/lvm/backup/lun for recovering physical volume metadata on that lun (http://www.tldp.org/HOWTO/LVM-HOWTO/recovermetadata.html) Le 21/06/2015 12:02, mots a écrit :
Let me get this straight:
You added a new host and instead of letting VDSM manage the VM storage you added it manually, completely independent from oVirt, during an unattended installation of the host OS? In a production environment?
Why?
-----Ursprüngliche Nachricht-----
Von:Nathanaël Blanchet <blanchet@abes.fr> Gesendet: Fre 19 Juni 2015 14:09 An: users@ovirt.org Betreff: [ovirt-users] disaster recovery
Hello all,
Here is what can happen as a nightmare for a sysadmin:
I installed a new host to my pool of pre existing ovirt hosts with a kickstart file and a default partitionning. I used to do this with vms that usually get one local disk. But.. This time this host was attached to several existing ovirt (production) lun and anaconda gathered all existing disk (local and lun) into the same VG (lv_home) et formatted them with XFS.... In the webadmin, the domain storage became unvailaible, vms were still up (thankgoodness), but it was impossible to interact with them. If I stopped them, it was impossible to reboot them. If I launch lvs command, some hosts can still see the ovirt LV, but other see only the lv_home while /dev/[idofvms] are still present. So it was my chance that vms were still present, and I began to export them with a tar at file system level so as to import them in a new domain storage.
It seems that data are still present because the vms are still running. So my question is really : instead of this difficult export step, is there a way to recover the initial format of the ovirt lun so as to make the lvm index come back on the disk?
Any help or disaster experience would be much appreciated.
-- Nathanaël Blanchet
Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Anaconda and Kickstart are dangerous in FC environments and have been known to wipe all kinds of data if zoning wasn't properly done prior to os deployment to new hosts. Nothing to do with oVirt, it's a common mistake people make at least once before they step on this specific rake. On Jun 21, 2015 2:09 PM, "Nathanaël Blanchet" <blanchet@abes.fr> wrote:
In a traditional installation of an ovirt host, it seems that provisionning the OS is the first step before talking about vdsm. I didn't go further, the problem is that this host saw the production LUN and the kickstart took all the lun into the same VG with a XFS format. Yes it is an error from myself for not having dissociated the lun. Yes it is an error from myself for not having taken care of having snapshot lun level backup. But now what should I do? My idea was to use pvreduce to unlabel the LUN, and then using the /etc/lvm/backup/lun for recovering physical volume metadata on that lun ( http://www.tldp.org/HOWTO/LVM-HOWTO/recovermetadata.html)
Le 21/06/2015 12:02, mots a écrit :
Let me get this straight:
You added a new host and instead of letting VDSM manage the VM storage you added it manually, completely independent from oVirt, during an unattended installation of the host OS? In a production environment?
Why? -----Ursprüngliche Nachricht-----
Von:Nathanaël Blanchet <blanchet@abes.fr> Gesendet: Fre 19 Juni 2015 14:09 An: users@ovirt.org Betreff: [ovirt-users] disaster recovery
Hello all,
Here is what can happen as a nightmare for a sysadmin:
I installed a new host to my pool of pre existing ovirt hosts with a kickstart file and a default partitionning. I used to do this with vms that usually get one local disk. But.. This time this host was attached to several existing ovirt (production) lun and anaconda gathered all existing disk (local and lun) into the same VG (lv_home) et formatted them with XFS.... In the webadmin, the domain storage became unvailaible, vms were still up (thankgoodness), but it was impossible to interact with them. If I stopped them, it was impossible to reboot them. If I launch lvs command, some hosts can still see the ovirt LV, but other see only the lv_home while /dev/[idofvms] are still present. So it was my chance that vms were still present, and I began to export them with a tar at file system level so as to import them in a new domain storage.
It seems that data are still present because the vms are still running. So my question is really : instead of this difficult export step, is there a way to recover the initial format of the ovirt lun so as to make the lvm index come back on the disk?
Any help or disaster experience would be much appreciated.
-- Nathanaël Blanchet
Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This is a multi-part message in MIME format. --------------050507020202060701050903 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Sure that ovirt is not directly concerned but this has been done=20 following the foreman integration feature. So a warning doing this in a=20 fc environnement could be very preventive. My question was rather : now is there a possibility to recover this=20 disk? After all, it is only a LVM disk with one VG per LUN and as much=20 LV as much as existing VMs. Recovering LVM metadata might be enough? Le 21/06/2015 20:23, Dan Yasny a =C3=A9crit :
Anaconda and Kickstart are dangerous in FC environments and have been=20 known to wipe all kinds of data if zoning wasn't properly done prior=20 to os deployment to new hosts.
Nothing to do with oVirt, it's a common mistake people make at least=20 once before they step on this specific rake.
On Jun 21, 2015 2:09 PM, "Nathana=C3=ABl Blanchet" <blanchet@abes.fr=20 <mailto:blanchet@abes.fr>> wrote:
In a traditional installation of an ovirt host, it seems that provisionning the OS is the first step before talking about vdsm. I didn't go further, the problem is that this host saw the production LUN and the kickstart took all the lun into the same VG with a XFS format. Yes it is an error from myself for not having dissociated the lun. Yes it is an error from myself for not having taken care of having snapshot lun level backup. But now what should I do? My idea was to use pvreduce to unlabel the LUN, and then using the /etc/lvm/backup/lun for recovering physical volume metadata on that lun (http://www.tldp.org/HOWTO/LVM-HOWTO/recovermetadata.html)
Le 21/06/2015 12:02, mots a =C3=A9crit :
Let me get this straight:
You added a new host and instead of letting VDSM manage the VM storage you added it manually, completely independent from oVirt, during an unattended installation of the host OS? In a production environment?
Why? -----Urspr=C3=BCngliche Nachricht-----
Von:Nathana=C3=ABl Blanchet <blanchet@abes.fr <mailto:blanchet@abes.fr>> Gesendet: Fre 19 Juni 2015 14:09 An: users@ovirt.org <mailto:users@ovirt.org> Betreff: [ovirt-users] disaster recovery
Hello all,
Here is what can happen as a nightmare for a sysadmin:
I installed a new host to my pool of pre existing ovirt hosts with a kickstart file and a default partitionning. I used to do this with vms that usually get one local disk. But.. This time this host was attached to several existing ovirt (production) lun and anaconda gathered all existing disk (local and lun) into the same VG (lv_home) et formatted them with XFS.... In the webadmin, the domain storage became unvailaible, vms were still up (thankgoodness), but it was impossible to interact with them. If I stopped them, it was impossible to reboot them. If I launch lvs command, some hosts can still see the ovirt LV, but other see only the lv_home while /dev/[idofvms] are still present. So it was my chance that vms were still present, and I began to export them with a tar at file system level so as to import them in a new domain storage.
It seems that data are still present because the vms are still running. So my question is really : instead of this difficult export step, is there a way to recover the initial format of the ovirt lun so as to make the lvm index come back on the disk?
Any help or disaster experience would be much appreciated.
--=20 Nathana=C3=ABl Blanchet
Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr <mailto:blanchet@abes.fr>
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
--=20 Nathana=C3=ABl Blanchet Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr --------------050507020202060701050903 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html> <head> <meta content=3D"text/html; charset=3Dutf-8" http-equiv=3D"Content-Ty= pe"> </head> <body text=3D"#000000" bgcolor=3D"#FFFFFF"> Sure that ovirt is not directly concerned but this has been done following the foreman integration feature. So a warning doing this in a fc environnement could be very preventive.<br> My question was rather : now is there a possibility to recover this disk? After all, it is only a LVM disk with one VG per LUN and as much LV as much as existing VMs. Recovering LVM metadata might be enough?<br> <br> <div class=3D"moz-cite-prefix">Le 21/06/2015 20:23, Dan Yasny a =C3=A9crit=C2=A0:<br> </div> <blockquote cite=3D"mid:CALLXwb6H_apELuw=3Dh+uRA9er_Mn15Kw-46f=3DFx5VsGoTSdyJ6w@mail.= gmail.com" type=3D"cite"> <p dir=3D"ltr">Anaconda and Kickstart are dangerous in FC environments and have been known to wipe all kinds of data if zoning wasn't properly done prior to os deployment to new hosts. </p> <p dir=3D"ltr">Nothing to do with oVirt, it's a common mistake people make at least once before they step on this specific rake.</p> <div class=3D"gmail_quote">On Jun 21, 2015 2:09 PM, "Nathana=C3=ABl Blanchet" <<a moz-do-not-send=3D"true" href=3D"mailto:blanchet@abes.fr">blanchet@abes.fr</a>> wrote= :<br type=3D"attribution"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">In a traditional installation of an ovirt host, it seems that provisionning the OS is the first step before talking about vdsm. I didn't go further, the problem is that this host saw the production LUN and the kickstart took all the lun into the same VG with a XFS format.<br> Yes it is an error from myself for not having dissociated the lun.<br> Yes it is an error from myself for not having taken care of having snapshot lun level backup.<br> But now what should I do?<br> My idea was to use pvreduce to unlabel the LUN, and then using the /etc/lvm/backup/lun for recovering physical volume metadata on that lun (<a moz-do-not-send=3D"true" href=3D"http://www.tldp.org/HOWTO/LVM-HOWTO/recovermetadata.h= tml" rel=3D"noreferrer" target=3D"_blank">http://www.tldp.org/HOWT= O/LVM-HOWTO/recovermetadata.html</a>)<br> <br> Le 21/06/2015 12:02, mots a =C3=A9crit :<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Let me get this straight:<br> <br> You added a new host and instead of letting VDSM manage the VM storage you added it manually, completely independent from oVirt, during an unattended installation of the host OS? In a production environment?<br> <br> Why?<br> =C2=A0 -----Urspr=C3=BCngliche Nachricht-----<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Von:Nathana=C3=ABl Blanchet <<a moz-do-not-send=3D"true" href=3D"mailto:blanchet@abes.fr" target=3D"_blank">blanch= et@abes.fr</a>><br> Gesendet: Fre 19 Juni 2015 14:09<br> An: <a moz-do-not-send=3D"true" href=3D"mailto:users@ovirt.org" target=3D"_blank">users@o= virt.org</a><br> Betreff: [ovirt-users] disaster recovery<br> <br> Hello all,<br> <br> Here is what can happen as a nightmare for a sysadmin:<br> <br> I installed a new host to my pool of pre existing ovirt hosts with a<br> kickstart file and a default partitionning. I used to do this with vms<br> that usually get one local disk.<br> But..<br> This time this host was attached to several existing ovirt (production)<br> lun and anaconda gathered all existing disk (local and lun) into the<br> same VG (lv_home) et formatted them with XFS....<br> In the webadmin, the domain storage became unvailaible, vms were still<br> up (thankgoodness), but it was impossible to interact with them. If I<br> stopped them, it was impossible to reboot them. If I launch lvs command,<br> some hosts can still see the ovirt LV, but other see only the lv_home<br> while /dev/[idofvms] are still present.<br> So it was my chance that vms were still present, and I began to export<br> them with a tar at file system level so as to import them in a new<br> domain storage.<br> <br> It seems that data are still present because the vms are still running.<br> So my question is really : instead of this difficult export step, is<br> there a way to recover the initial format of the ovirt lun so as to make<br> the lvm index come back on the disk?<br> <br> Any help or disaster experience would be much appreciated.<= br> <br> -- <br> Nathana=C3=ABl Blanchet<br> <br> Supervision r=C3=A9seau<br> P=C3=B4le Infrastrutures Informatiques<br> 227 avenue Professeur-Jean-Louis-Viala<br> 34193 MONTPELLIER CEDEX 5=C2=A0 =C2=A0 =C2=A0 =C2=A0<br> T=C3=A9l. 33 (0)4 67 54 84 55<br> Fax=C2=A0 33 (0)4 67 54 84 14<br> <a moz-do-not-send=3D"true" href=3D"mailto:blanchet@abes.fr= " target=3D"_blank">blanchet@abes.fr</a><br> <br> _______________________________________________<br> Users mailing list<br> <a moz-do-not-send=3D"true" href=3D"mailto:Users@ovirt.org" target=3D"_blank">Users@ovirt.org</a><br> <a moz-do-not-send=3D"true" href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer" target=3D"_blank">http://lists.ovirt.o= rg/mailman/listinfo/users</a><br> <br> </blockquote> </blockquote> <br> _______________________________________________<br> Users mailing list<br> <a moz-do-not-send=3D"true" href=3D"mailto:Users@ovirt.org" target=3D"_blank">Users@ovirt.org</a><br> <a moz-do-not-send=3D"true" href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer" target=3D"_blank">http://lists.ovirt.org/m= ailman/listinfo/users</a><br> </blockquote> </div> </blockquote> <br> <pre class=3D"moz-signature" cols=3D"72">--=20 Nathana=C3=ABl Blanchet Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:blanchet@abes.fr">bl= anchet@abes.fr</a> </pre> </body> </html> --------------050507020202060701050903--
participants (3)
-
Dan Yasny -
mots -
Nathanaël Blanchet