[ovirt-users] Re: Data recovery from (now unused, but still mounted) Gluster Volume for a single VM

First off, I have very little hope, you'll be able to recover your data working at gluster level... And then there is a lot of information missing between the lines: I guess you are using a 3 node HCI setup and were adding new disks (/dev/sdb) on all three nodes and trying to move the glusterfs to those new bigger disks? Resizing/moving/adding or removing disks are "natural" operations for Gluster. But oVirt isn't "gluster native" and may not be so forgiving if you just swap device paths on bricks. <rant> Practical guides on how to replace the storage without down time (after all this is a HA solution, right?) are somehow missing from the oVirt documentation, and if I was a rich man, perhaps I'd get myself an RHV support contract and see if RHEL engineers would say anything but "not supported". </rant> The first thing I'd recommend is to create some temporary space. I found using an extra disk as NFS storage on one of the hosts was a good way to gain some maneuvering room e.g. for backups. You can try to attach the disk of the broken VM as a secondary to another good VM to see if the data can be salvaged from there. But before you attach it (and perhaps an automatic fsck ruins it for you), you can perhaps create a copy to the NFS export/backup (domain). If you weren't out of space, you'd just create a local copy and work with that. You can also try exporting the disk image, but there is a lot of untested or slow code in that operation from my experience. If that image happens to be empty (I've seen that happen) or the data on it cannot be recovered, there is little to be gained, by trying to work at the GlusterFS level. The logical disk image file will be chunked into 64MB bits and their order is buried deep either in GlusterFS or in oVirt and perhaps your business is the better place to invest your energy. But there is a good chance the data portion of that disk image still has your data. The fact that oVirt/KVM generally pauses VMs when it has issues with the storage, tends to preserve and protect your data rather better than what happens when physical hosts suffer brown outs or power glitches. I guess you'll have learned that oVirt doesn't protect you from making mistakes, it only tries to offer some resilience against faults. It's good and valuable to report these things, because it helps others to learn, too. I sincerely hope you'll make do!