Move VM disks from gluster storage to local storage
Hi, An unknown bug broke our gluster storage (dom_md/ids is corrupted) and oVirt no longer activates the storage(I tried to recover it using the similar issues reported in mailing list but it didn't work). As I checked VM disk images are still accessible when I mount the gluster storage manually. How can we manually move the VM disk images to local storage? (oVirt complains about gluster storage being inactive when using the web interface for move/copy) Thanks, Siavash
On Sun, Aug 14, 2016 at 5:55 PM, Siavash Safi <siavash.safi@gmail.com> wrote:
Hi,
An unknown bug broke our gluster storage (dom_md/ids is corrupted) and oVirt no longer activates the storage(I tried to recover it using the similar issues reported in mailing list but it didn't work).
Can you explain what you did? The best way to fix this is to initialize the corrupt id file and activate the domain.
As I checked VM disk images are still accessible when I mount the gluster storage manually. How can we manually move the VM disk images to local storage? (oVirt complains about gluster storage being inactive when using the web interface for move/copy)
You can easily copy the images to another file based storage (nfs, gluster) like this: 1. activate other storage domain using engine 2. mount gluster domain manually 3. copy the image from gluster domain to the other domain: cp -r gluster-domain-mountpoint/images/image-uuid /rhev/data-center/mnt/server:_path/other-domain-uuid/images/ But the images will not be available since engine does know them. Maybe this can be fixed by modifying engine database. Another solution (if you are using ovirt 4.0), is to upload the images to a new disk, and attach the disk to the vm instead of the missing disk. Nir
On Sun, Aug 14, 2016 at 8:07 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Aug 14, 2016 at 5:55 PM, Siavash Safi <siavash.safi@gmail.com> wrote:
Hi,
An unknown bug broke our gluster storage (dom_md/ids is corrupted) and oVirt no longer activates the storage(I tried to recover it using the similar issues reported in mailing list but it didn't work).
Can you explain what you did?
cd /mnt/4697fbde-45fb-4f91-ac4c-5516bc59f683/dom_md/ rm ids touch ids sanlock direct init -s 4697fbde-45fb-4f91-ac4c-5516bc59f683:0:ids:1048576
The best way to fix this is to initialize the corrupt id file and activate the domain.
This would be great!
As I checked VM disk images are still accessible when I mount the gluster
storage manually. How can we manually move the VM disk images to local storage? (oVirt complains about gluster storage being inactive when using the web interface for move/copy)
You can easily copy the images to another file based storage (nfs, gluster) like this:
1. activate other storage domain using engine 2. mount gluster domain manually 3. copy the image from gluster domain to the other domain:
cp -r gluster-domain-mountpoint/images/image-uuid /rhev/data-center/mnt/server:_path/other-domain-uuid/images/
But the images will not be available since engine does know them. Maybe this can be fixed by modifying engine database.
How complicated is it?
Another solution (if you are using ovirt 4.0), is to upload the images to a new disk, and attach the disk to the vm instead of the missing disk.
We are running 3.6
Nir
On Sun, Aug 14, 2016 at 8:10 PM, Siavash Safi <siavash.safi@gmail.com> wrote:
On Sun, Aug 14, 2016 at 8:07 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Aug 14, 2016 at 5:55 PM, Siavash Safi <siavash.safi@gmail.com> wrote:
Hi,
An unknown bug broke our gluster storage (dom_md/ids is corrupted) and oVirt no longer activates the storage(I tried to recover it using the similar issues reported in mailing list but it didn't work).
Can you explain what you did?
cd /mnt/4697fbde-45fb-4f91-ac4c-5516bc59f683/dom_md/ rm ids touch ids sanlock direct init -s 4697fbde-45fb-4f91-ac4c-5516bc59f683:0:ids:1048576
The offset parameter should be 0, not 1048576: sanlock direct init -s 4697fbde-45fb-4f91-ac4c-5516bc59f683:0:ids:0 See http://lists.ovirt.org/pipermail/users/2016-February/038046.html Please retry this. Also, are you using replica 3? These issues typically happened when people used replica 2 gluster volumes.
The best way to fix this is to initialize the corrupt id file and activate the domain.
This would be great!
As I checked VM disk images are still accessible when I mount the gluster storage manually. How can we manually move the VM disk images to local storage? (oVirt complains about gluster storage being inactive when using the web interface for move/copy)
You can easily copy the images to another file based storage (nfs, gluster) like this:
1. activate other storage domain using engine 2. mount gluster domain manually 3. copy the image from gluster domain to the other domain:
cp -r gluster-domain-mountpoint/images/image-uuid /rhev/data-center/mnt/server:_path/other-domain-uuid/images/
But the images will not be available since engine does know them. Maybe this can be fixed by modifying engine database.
How complicated is it?
I never tried this, lets try the simple way first.
Another solution (if you are using ovirt 4.0), is to upload the images to a new disk, and attach the disk to the vm instead of the missing disk.
We are running 3.6
Maybe consider an upgrade? Nir
On Sun, Aug 14, 2016 at 10:04 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Aug 14, 2016 at 8:10 PM, Siavash Safi <siavash.safi@gmail.com> wrote:
On Sun, Aug 14, 2016 at 8:07 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Aug 14, 2016 at 5:55 PM, Siavash Safi <siavash.safi@gmail.com> wrote:
Hi,
An unknown bug broke our gluster storage (dom_md/ids is corrupted) and oVirt no longer activates the storage(I tried to recover it using the
similar
issues reported in mailing list but it didn't work).
Can you explain what you did?
cd /mnt/4697fbde-45fb-4f91-ac4c-5516bc59f683/dom_md/ rm ids touch ids sanlock direct init -s 4697fbde-45fb-4f91-ac4c-5516bc59f683:0:ids:1048576
The offset parameter should be 0, not 1048576:
sanlock direct init -s 4697fbde-45fb-4f91-ac4c-5516bc59f683:0:ids:0
See http://lists.ovirt.org/pipermail/users/2016-February/038046.html
Please retry this.
I didn't know what the number at the end of the sring does ;)
Also, are you using replica 3? These issues typically happened when people used replica 2 gluster volumes.
Actually we removed one of the broken nodes from gluster and tried to setup local storage. I wiped the storage and added the bricks back to gluster. Thanks Nir, recreating the ids file with correct offset and resizing gluster to replica 3 fixed the issue :)
The best way to fix this is to initialize the corrupt id file and activate the domain.
This would be great!
As I checked VM disk images are still accessible when I mount the gluster storage manually. How can we manually move the VM disk images to local storage? (oVirt complains about gluster storage being inactive when using the web interface for move/copy)
You can easily copy the images to another file based storage (nfs, gluster) like this:
1. activate other storage domain using engine 2. mount gluster domain manually 3. copy the image from gluster domain to the other domain:
cp -r gluster-domain-mountpoint/images/image-uuid /rhev/data-center/mnt/server:_path/other-domain-uuid/images/
But the images will not be available since engine does know them. Maybe this can be fixed by modifying engine database.
How complicated is it?
I never tried this, lets try the simple way first.
Another solution (if you are using ovirt 4.0), is to upload the images to a new disk, and attach the disk to the vm instead of the missing disk.
We are running 3.6
Maybe consider an upgrade?
Nir
participants (2)
-
Nir Soffer -
Siavash Safi