[Users] SOLVED: Recovering VMs when the engine is lost
Hello Itamar and Ayal, I tried to recover the VMs by using the first method (converting the master into an export domain) but after the changes, I couldn't add the export domain as it was "an incompatible version". Therefore I explored the other way of restoring: - First, I copied the master/vms folder from the master storage to my local host (and renamed al the .ovf to .xml so I can double click them and view them in an xml browser on my stupid Windows) - As we store the ovirt NFS export folder on /data/ovirt, I connected to all nodes and moved the $UUID folder from /data/ovirt to /data/old. - I added all the cpu nodes and NFS storages to the new cluster - I recreated the virtual machines based on the xml files in the specs and our internal vm specifications with exactly the same MAC address (we specify our own) and disk sizes, having the disks residing on the same storages as the old disks - Looking at the xml file of the new VMs I figured out the folders and file names of the new disks and then moved the old disks over: cd /data/ovirt/$UUID/images/$FOLDERUUID mv /data/old/images/$OLDFOLDERUUID/$OLDDISKUUID ./$DISKUUID mv /data/old/images/$OLDFOLDERUUID/$OLDDISKUUID.meta ./$DISKUUID.meta mv /data/old/images/$OLDFOLDERUUID/$OLDDISKUUID.lease ./$DISKUUID.lease - After doing this, the new VM starts nicely using the disks with all the content on them Of course, you can also choose other storages for the new disks, but then you have to rsync -avS the sparse files over to the new storage and this takes hours for 600G disks. Caveats: There were a few machines I actually created from a template. In the XML files for those machines, we had the wrong MAC addresses for the nics. I assume, that these were the ones generated during the template clone and not the final ones I set after cloning the VMs. But as we have nice host specifications as yaml files for every host, I was relying more on them than on the XML. In the end, it took me around 3 minutes of manual work to recreate the vm, move the disks over and start the vm. Automating that with a bit of yaml and xml parsing magic and recreating the VMS automatically via script could lead to recreating a vm in under 1 minute (except when moving disks from storage to storage). Kind regards Jörn
-----Original Message----- From: Itamar Heim [mailto:iheim@redhat.com] Sent: Freitag, 23. November 2012 15:34 To: Joern Ott Cc: users@ovirt.org; Ayal Baron Subject: Re: [Users] Recovering VMs when the engine is lost
On 11/23/2012 04:16 PM, Joern Ott wrote:
Hello Itamar,
-----Original Message----- From: Itamar Heim [mailto:iheim@redhat.com] Sent: Freitag, 23. November 2012 14:11 To: Joern Ott Cc: users@ovirt.org; Ayal Baron Subject: Re: [Users] Recovering VMs when the engine is lost
On 11/23/2012 02:58 PM, Joern Ott wrote:
Hey Itamar,
this is an NFS storage.
we don't currently support importing an existing data storage domain. so you need to create a new nfs data domain.
i suggest two 'hacky' options after that 1. less hacky - convert the current nfs data domain to an export domain, and import the VMs from it.
Is there any info on how to do this, I didn't find much info on export domains on the wiki. The last time I tried to move thin disks with ovirt GUI from one storage to another, they were expanded and filled up the destination. In the database, the disks were shown to be on the destination host and were marked as "invalid", so this procedure also has some risks.
check this one, and please wikify for others if you find it helpful: http://comments.gmane.org/gmane.comp.emulators.ovirt.user/4428
2. more hacky - recreate the VMs on the new nfs data domain with exact same details, and copy the disk over the newly created disks.
benefit of the other is you may be able to move the data, instead of full blown copy, while in option 1 (which is much simpler/less error prone), you need to import all vm's again.
The advantage of this would be faster speed as most disks are thin
provisioned disks. So I think, I will go for this option.
just to make it clear - this is risky, and you should know what you are doing when copying/moving files over the new layout, which must be identical to previous definitions.
there is current work to detect disks from a storage domain: http://wiki.ovirt.org/wiki/Features/Domain_Scan
when it will be ready (and better when import existing storage domain will be supported), this would be easier.
KR Jörn
-----Original Message----- From: Itamar Heim [mailto:iheim@redhat.com] Sent: Freitag, 23. November 2012 10:32 To: Joern Ott Cc: users@ovirt.org; Ayal Baron Subject: Re: [Users] Recovering VMs when the engine is lost
On 11/23/2012 11:03 AM, Joern Ott wrote:
Hello everybody,
I managed to re-install the server the oVirt engine was running on in my testlab. All the cpu and storages are still there and the VMs are still running without problems. Is there a way, I can attach the still existing nodes to the new oVirt engine and recover/import the
VMs?
Alternatively, is there a way, I can import the VMs disks? I still know how the data center and clusters were set up and the specs for the VMs are also documented, so I'd like to re-attach the disks, which would save me some time compared to the re-installs.
is this an iscsi or nfs data storage domain?
Kind regards Jörn
On 11/28/2012 07:36 AM, Joern Ott wrote:
Hello Itamar and Ayal,
I tried to recover the VMs by using the first method (converting the master into an export domain) but after the changes, I couldn't add the export domain as it was "an incompatible version". Therefore I explored the other way of restoring: - First, I copied the master/vms folder from the master storage to my local host (and renamed al the .ovf to .xml so I can double click them and view them in an xml browser on my stupid Windows) - As we store the ovirt NFS export folder on /data/ovirt, I connected to all nodes and moved the $UUID folder from /data/ovirt to /data/old. - I added all the cpu nodes and NFS storages to the new cluster - I recreated the virtual machines based on the xml files in the specs and our internal vm specifications with exactly the same MAC address (we specify our own) and disk sizes, having the disks residing on the same storages as the old disks - Looking at the xml file of the new VMs I figured out the folders and file names of the new disks and then moved the old disks over: cd /data/ovirt/$UUID/images/$FOLDERUUID mv /data/old/images/$OLDFOLDERUUID/$OLDDISKUUID ./$DISKUUID mv /data/old/images/$OLDFOLDERUUID/$OLDDISKUUID.meta ./$DISKUUID.meta mv /data/old/images/$OLDFOLDERUUID/$OLDDISKUUID.lease ./$DISKUUID.lease - After doing this, the new VM starts nicely using the disks with all the content on them
Of course, you can also choose other storages for the new disks, but then you have to rsync -avS the sparse files over to the new storage and this takes hours for 600G disks.
Caveats: There were a few machines I actually created from a template. In the XML files for those machines, we had the wrong MAC addresses for the nics. I assume, that these were the ones generated during the template clone and not the final ones I set after cloning the VMs. But as we have nice host specifications as yaml files for every host, I was relying more on them than on the XML.
In the end, it took me around 3 minutes of manual work to recreate the vm, move the disks over and start the vm. Automating that with a bit of yaml and xml parsing magic and recreating the VMS automatically via script could lead to recreating a vm in under 1 minute (except when moving disks from storage to storage).
Kind regards Jörn
-----Original Message----- From: Itamar Heim [mailto:iheim@redhat.com] Sent: Freitag, 23. November 2012 15:34 To: Joern Ott Cc: users@ovirt.org; Ayal Baron Subject: Re: [Users] Recovering VMs when the engine is lost
On 11/23/2012 04:16 PM, Joern Ott wrote:
Hello Itamar,
-----Original Message----- From: Itamar Heim [mailto:iheim@redhat.com] Sent: Freitag, 23. November 2012 14:11 To: Joern Ott Cc: users@ovirt.org; Ayal Baron Subject: Re: [Users] Recovering VMs when the engine is lost
On 11/23/2012 02:58 PM, Joern Ott wrote:
Hey Itamar,
this is an NFS storage.
we don't currently support importing an existing data storage domain. so you need to create a new nfs data domain.
i suggest two 'hacky' options after that 1. less hacky - convert the current nfs data domain to an export domain, and import the VMs from it.
Is there any info on how to do this, I didn't find much info on export domains on the wiki. The last time I tried to move thin disks with ovirt GUI from one storage to another, they were expanded and filled up the destination. In the database, the disks were shown to be on the destination host and were marked as "invalid", so this procedure also has some risks.
check this one, and please wikify for others if you find it helpful: http://comments.gmane.org/gmane.comp.emulators.ovirt.user/4428
2. more hacky - recreate the VMs on the new nfs data domain with exact same details, and copy the disk over the newly created disks.
benefit of the other is you may be able to move the data, instead of full blown copy, while in option 1 (which is much simpler/less error prone), you need to import all vm's again.
The advantage of this would be faster speed as most disks are thin
provisioned disks. So I think, I will go for this option.
just to make it clear - this is risky, and you should know what you are doing when copying/moving files over the new layout, which must be identical to previous definitions.
there is current work to detect disks from a storage domain: http://wiki.ovirt.org/wiki/Features/Domain_Scan
when it will be ready (and better when import existing storage domain will be supported), this would be easier.
KR Jörn
-----Original Message----- From: Itamar Heim [mailto:iheim@redhat.com] Sent: Freitag, 23. November 2012 10:32 To: Joern Ott Cc: users@ovirt.org; Ayal Baron Subject: Re: [Users] Recovering VMs when the engine is lost
On 11/23/2012 11:03 AM, Joern Ott wrote: > Hello everybody, > > I managed to re-install the server the oVirt engine was running on > in my testlab. All the cpu and storages are still there and the > VMs are still running without problems. Is there a way, I can > attach the still existing nodes to the new oVirt engine and > recover/import the
VMs?
> > Alternatively, is there a way, I can import the VMs disks? I still > know how the data center and clusters were set up and the specs > for the VMs are also documented, so I'd like to re-attach the > disks, which would save me some time compared to the re-installs. >
is this an iscsi or nfs data storage domain?
Kind regards Jörn
while hacky, i wonder if worth wiki-fying.
On 11/29/2012 11:52 AM, Itamar Heim wrote:
On 11/28/2012 07:36 AM, Joern Ott wrote:
Hello Itamar and Ayal,
I tried to recover the VMs by using the first method (converting the master into an export domain) but after the changes, I couldn't add the export domain as it was "an incompatible version". Therefore I explored the other way of restoring: - First, I copied the master/vms folder from the master storage to my local host (and renamed al the .ovf to .xml so I can double click them and view them in an xml browser on my stupid Windows) - As we store the ovirt NFS export folder on /data/ovirt, I connected to all nodes and moved the $UUID folder from /data/ovirt to /data/old. - I added all the cpu nodes and NFS storages to the new cluster - I recreated the virtual machines based on the xml files in the specs and our internal vm specifications with exactly the same MAC address (we specify our own) and disk sizes, having the disks residing on the same storages as the old disks - Looking at the xml file of the new VMs I figured out the folders and file names of the new disks and then moved the old disks over: cd /data/ovirt/$UUID/images/$FOLDERUUID mv /data/old/images/$OLDFOLDERUUID/$OLDDISKUUID ./$DISKUUID mv /data/old/images/$OLDFOLDERUUID/$OLDDISKUUID.meta ./$DISKUUID.meta mv /data/old/images/$OLDFOLDERUUID/$OLDDISKUUID.lease ./$DISKUUID.lease - After doing this, the new VM starts nicely using the disks with all the content on them
Of course, you can also choose other storages for the new disks, but then you have to rsync -avS the sparse files over to the new storage and this takes hours for 600G disks.
Caveats: There were a few machines I actually created from a template. In the XML files for those machines, we had the wrong MAC addresses for the nics. I assume, that these were the ones generated during the template clone and not the final ones I set after cloning the VMs. But as we have nice host specifications as yaml files for every host, I was relying more on them than on the XML.
In the end, it took me around 3 minutes of manual work to recreate the vm, move the disks over and start the vm. Automating that with a bit of yaml and xml parsing magic and recreating the VMS automatically via script could lead to recreating a vm in under 1 minute (except when moving disks from storage to storage).
Kind regards Jörn
-----Original Message----- From: Itamar Heim [mailto:iheim@redhat.com] Sent: Freitag, 23. November 2012 15:34 To: Joern Ott Cc: users@ovirt.org; Ayal Baron Subject: Re: [Users] Recovering VMs when the engine is lost
On 11/23/2012 04:16 PM, Joern Ott wrote:
Hello Itamar,
-----Original Message----- From: Itamar Heim [mailto:iheim@redhat.com] Sent: Freitag, 23. November 2012 14:11 To: Joern Ott Cc: users@ovirt.org; Ayal Baron Subject: Re: [Users] Recovering VMs when the engine is lost
On 11/23/2012 02:58 PM, Joern Ott wrote:
Hey Itamar,
this is an NFS storage.
we don't currently support importing an existing data storage domain. so you need to create a new nfs data domain.
i suggest two 'hacky' options after that 1. less hacky - convert the current nfs data domain to an export domain, and import the VMs from it.
Is there any info on how to do this, I didn't find much info on export domains on the wiki. The last time I tried to move thin disks with ovirt GUI from one storage to another, they were expanded and filled up the destination. In the database, the disks were shown to be on the destination host and were marked as "invalid", so this procedure also has some risks.
check this one, and please wikify for others if you find it helpful: http://comments.gmane.org/gmane.comp.emulators.ovirt.user/4428
2. more hacky - recreate the VMs on the new nfs data domain with exact same details, and copy the disk over the newly created disks.
benefit of the other is you may be able to move the data, instead of full blown copy, while in option 1 (which is much simpler/less error prone), you need to import all vm's again.
The advantage of this would be faster speed as most disks are thin
provisioned disks. So I think, I will go for this option.
just to make it clear - this is risky, and you should know what you are doing when copying/moving files over the new layout, which must be identical to previous definitions.
there is current work to detect disks from a storage domain: http://wiki.ovirt.org/wiki/Features/Domain_Scan
when it will be ready (and better when import existing storage domain will be supported), this would be easier.
KR Jörn
> -----Original Message----- > From: Itamar Heim [mailto:iheim@redhat.com] > Sent: Freitag, 23. November 2012 10:32 > To: Joern Ott > Cc: users@ovirt.org; Ayal Baron > Subject: Re: [Users] Recovering VMs when the engine is lost > > On 11/23/2012 11:03 AM, Joern Ott wrote: >> Hello everybody, >> >> I managed to re-install the server the oVirt engine was running on >> in my testlab. All the cpu and storages are still there and the >> VMs are still running without problems. Is there a way, I can >> attach the still existing nodes to the new oVirt engine and >> recover/import the
VMs?
>> >> Alternatively, is there a way, I can import the VMs disks? I still >> know how the data center and clusters were set up and the specs >> for the VMs are also documented, so I'd like to re-attach the >> disks, which would save me some time compared to the re-installs. >> > > is this an iscsi or nfs data storage domain? > >
Kind regards Jörn
while hacky, i wonder if worth wiki-fying. Well it would not hurt to have it in the wiki marking it as a 'hack' This is something what could happen because of some bug in the engine and you'll need to recover. Another thing I am wondering about is, if we should not provide a recovery tool for something like this. It sounds pretty much like something what can be supported for backing up VMs for example.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
participants (3)
-
Itamar Heim -
Joern Ott -
Vinzenz Feenstra