Hi, We have a situation where the HostedEngine was cleaned up and the VMs are no longer running. Looking at the logs we can see the drive files as:

2019-03-26T07:42:46.915838Z qemu-kvm: -drive file=/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685,format=qcow2,if=none,id=drive-ua-b2b872cd-b468-4f14-ae20-555ed823e84b,serial=b2b872cd-b468-4f14-ae20-555ed823e84b,werror=stop,rerror=stop,cache=none,aio=native: 'serial' is deprecated, please use the corresponding option of '-device' instead I assume this is the disk was writing to before it went down. Trying to list the file gives an error and the file is not there; ls -l /rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685 Is there a way we can recover the VM's disk images ? NOTE: No HostedEngine backups -- Regards, Sakhi Hadebe _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FPTPLMY4TC4...

What happened is the engine's root filesystem had filled up. My colleague tried to resize the root lvm. The engine then did not come back. In trying to resolve that he cleaned up the engine and tried to re-install it, no luck in doing that. That brought down all the VMs. All VMs are down. we trying to move them into one of the standalone kvm host. We have been trying to locate the VM disk images, with no luck. According to the of the VM xml configuration file.the disk file is /rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685 Unfortunately we can find it and the solution on teh forum states that we can only find in the associated logical volume, but i think only when teh vm is running. The disk images we have been trying to boot up from are the one's we got from the gluster bricks, but the are far small that real images and can't boot

On Thu, Apr 11, 2019 at 6:13 PM Simone Tiraboschi <stirabos(a)redhat.com&gt; wrote: > > > On Thu, Apr 11, 2019 at 9:46 AM Sakhi Hadebe <sakhi(a)sanren.ac.za&gt; wrote: > >> Hi, >> >> We have a situation where the HostedEngine was cleaned up and the VMs >> are no longer running. Looking at the logs we can see the drive files as: >> > > Do you have any guess on what really happened? > Are you sure that the disks really disappeared? > > Please notice that the symlinks under rhev/data-center/mnt/glusterSD/glustermount... > are created on the fly only when needed. > > Are you sure that your host is correctly connecting the gluster storage > domain? > > >> >> 2019-03-26T07:42:46.915838Z qemu-kvm: -drive >> file=/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685,format=qcow2,if=none,id=drive-ua-b2b872cd-b468-4f14-ae20-555ed823e84b,serial=b2b872cd-b468-4f14-ae20-555ed823e84b,werror=stop,rerror=stop,cache=none,aio=native: >> 'serial' is deprecated, please use the corresponding option of '-device' >> instead >> >> I assume this is the disk was writing to before it went down. Trying to >> list the file gives an error and the file is not there; >> ls -l >> /rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685 >> >> Is there a way we can recover the VM's disk images ? >> >> NOTE: No HostedEngine backups >> >> -- >> Regards, >> Sakhi Hadebe >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/FPTPLMY4TC4... >> > -- Regards, Sakhi Hadebe Engineer: South African National Research Network (SANReN)Competency Area, Meraka, CSIR Tel: +27 12 841 2308 <+27128414213> Fax: +27 12 841 4223 <+27128414223> Cell: +27 71 331 9622 <+27823034657> Email: sakhi(a)sanren.ac.za <shadebe(a)csir.co.za&gt;

Adding to what me and my colleague shared I am able to locate the disk images of the VMs, I copied some of them and tried to boot them from another standalone kvm host, however booting the disk images wasn't succesful as it landed on a rescue mode. The strange part is that the VM disk images are 64MB in size, which doesn't seem to be normal for a disk image(see the below command extract). root@gohan 019a7072-43d5-44b5-bb86-7a7327f02087]# pwd /gluster_bricks/data/data/659de125-5671-4777-b27e-974aec0a4c9c/images/019a7072-43d5-44b5-bb86-7a7327f02087 [root@gohan 019a7072-43d5-44b5-bb86-7a7327f02087]# ll -h total 66M -rw-rw----. 2 vdsm kvm 64M Mar 22 13:48 f5f97478-6ccb-48bc-93b7-2fd5939f40bf -rw-rw----. 2 vdsm kvm 1.0M Mar 4 12:10 f5f97478-6ccb-48bc-93b7-2fd5939f40bf.lease -rw-r--r--. 2 vdsm kvm 317 Mar 22 11:06 f5f97478-6ccb-48bc-93b7-2fd5939f40bf.meta Please share insights on how I can reconstruct the disk image so that it can become bootable on the kvm host. Thanks in advance for the reply.

_______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZIQNXSY5GAK...

This listing is from a gluster mount not from the underlying brick, which should combine all parts from the underlying .glusterfs folder. I believe when you make use of the feature.shard the files should be broken up in peaces according the shard-size. Olaf _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/JIEAGLEN67E...

Basically i want to take out all of the HDD's in the main gluster pool, and replace with SSD's. My thought was to put everything in maintenance, copy the data manually over to a transient storage server. Destroy the gluster volume, swap in all the new drives, build a new gluster volume with the same name / settings, move data back, and be done. Any thoughts on this?

_______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TP6V2OYG5ZX...