3:45 a.m.
Hi,
We have a situation where the HostedEngine was cleaned up and the VMs are
no longer running. Looking at the logs we can see the drive files as:
2019-03-26T07:42:46.915838Z qemu-kvm: -drive
file=/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685,format=qcow2,if=none,id=drive-ua-b2b872cd-b468-4f14-ae20-555ed823e84b,serial=b2b872cd-b468-4f14-ae20-555ed823e84b,werror=stop,rerror=stop,cache=none,aio=native:
'serial' is deprecated, please use the corresponding option of '-device'
instead
I assume this is the disk was writing to before it went down. Trying to
list the file gives an error and the file is not there;
ls -l
/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685
Is there a way we can recover the VM's disk images ?
NOTE: No HostedEngine backups
--
Regards,
Sakhi Hadebe
12:13 p.m.
On Thu, Apr 11, 2019 at 9:46 AM Sakhi Hadebe <sakhi(a)sanren.ac.za> wrote:
Hi,
We have a situation where the HostedEngine was cleaned up and the VMs are
no longer running. Looking at the logs we can see the drive files as:
Do you have any guess on what really happened?
Are you sure that the disks really disappeared?
Please notice that the symlinks under
rhev/data-center/mnt/glusterSD/glustermount...
are created on the fly only when needed.
Are you sure that your host is correctly connecting the gluster storage
domain?
2019-03-26T07:42:46.915838Z qemu-kvm: -drive
file=/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685,format=qcow2,if=none,id=drive-ua-b2b872cd-b468-4f14-ae20-555ed823e84b,serial=b2b872cd-b468-4f14-ae20-555ed823e84b,werror=stop,rerror=stop,cache=none,aio=native:
'serial' is deprecated, please use the corresponding option of '-device'
instead
I assume this is the disk was writing to before it went down. Trying to
list the file gives an error and the file is not there;
ls -l
/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685
Is there a way we can recover the VM's disk images ?
NOTE: No HostedEngine backups
--
Regards,
Sakhi Hadebe
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FPTPLMY4TC4...
1:12 p.m.
What happened is the engine's root filesystem had filled up. My colleague
tried to resize the root lvm. The engine then did not come back. In trying
to resolve that he cleaned up the engine and tried to re-install it, no
luck in doing that.
That brought down all the VMs. All VMs are down. we trying to move them
into one of the standalone kvm host. We have been trying to locate the VM
disk images, with no luck.
According to the of the VM xml configuration file.the disk file is
/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685
Unfortunately we can find it and the solution on teh forum states that we
can only find in the associated logical volume, but i think only when teh
vm is running.
The disk images we have been trying to boot up from are the one's we got
from the gluster bricks, but the are far small that real images and can't
boot
On Thu, Apr 11, 2019 at 6:13 PM Simone Tiraboschi <stirabos(a)redhat.com>
wrote:
On Thu, Apr 11, 2019 at 9:46 AM Sakhi Hadebe <sakhi(a)sanren.ac.za> wrote:
> Hi,
>
> We have a situation where the HostedEngine was cleaned up and the VMs are
> no longer running. Looking at the logs we can see the drive files as:
>
Do you have any guess on what really happened?
Are you sure that the disks really disappeared?
Please notice that the symlinks under rhev/data-center/mnt/glusterSD/glustermount...
are created on the fly only when needed.
Are you sure that your host is correctly connecting the gluster storage
domain?
>
> 2019-03-26T07:42:46.915838Z qemu-kvm: -drive
>
file=/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685,format=qcow2,if=none,id=drive-ua-b2b872cd-b468-4f14-ae20-555ed823e84b,serial=b2b872cd-b468-4f14-ae20-555ed823e84b,werror=stop,rerror=stop,cache=none,aio=native:
> 'serial' is deprecated, please use the corresponding option of
'-device'
> instead
>
> I assume this is the disk was writing to before it went down. Trying to
> list the file gives an error and the file is not there;
> ls -l
>
/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685
>
> Is there a way we can recover the VM's disk images ?
>
> NOTE: No HostedEngine backups
>
> --
> Regards,
> Sakhi Hadebe
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FPTPLMY4TC4...
>
--
Regards,
Sakhi Hadebe
Engineer: South African National Research Network (SANReN)Competency
Area, Meraka, CSIR
Tel: +27 12 841 2308 <+27128414213>
Fax: +27 12 841 4223 <+27128414223>
Cell: +27 71 331 9622 <+27823034657>
Email: sakhi(a)sanren.ac.za <shadebe(a)csir.co.za>
5:04 p.m.
Il Gio 11 Apr 2019, 19:12 Sakhi Hadebe <sakhi(a)sanren.ac.za> ha scritto:
What happened is the engine's root filesystem had filled up. My
colleague
tried to resize the root lvm. The engine then did not come back. In trying
to resolve that he cleaned up the engine and tried to re-install it, no
luck in doing that.
That brought down all the VMs. All VMs are down. we trying to move them
into one of the standalone kvm host. We have been trying to locate the VM
disk images, with no luck.
According to the of the VM xml configuration file.the disk file is
/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685
Unfortunately we can find it and the solution on teh forum states that we
can only find in the associated logical volume, but i think only when teh
vm is running.
The disk images we have been trying to boot up from are the one's we got
from the gluster bricks, but the are far small that real images and can't
boot
I'd strongly suggest to deploy a new environment over a new storage domain
and import there the existing hosts; if you configure the cluster in the
new engine for virt+gluster, the engine should be able to automatically
detect the gluster volumes there and then you will be able to import the
existing storage domains into the new engine.
All the definitions of the VMs are periodically (1h) saved also on a
special volume on the storage domain called OVF_STORE so you will find your
existing VMs into the new engine and you will be able to start them from
there.
On Thu, Apr 11, 2019 at 6:13 PM Simone Tiraboschi <stirabos(a)redhat.com>
wrote:
>
>
> On Thu, Apr 11, 2019 at 9:46 AM Sakhi Hadebe <sakhi(a)sanren.ac.za> wrote:
>
>> Hi,
>>
>> We have a situation where the HostedEngine was cleaned up and the VMs
>> are no longer running. Looking at the logs we can see the drive files as:
>>
>
> Do you have any guess on what really happened?
> Are you sure that the disks really disappeared?
>
> Please notice that the symlinks under rhev/data-center/mnt/glusterSD/glustermount...
> are created on the fly only when needed.
>
> Are you sure that your host is correctly connecting the gluster storage
> domain?
>
>
>>
>> 2019-03-26T07:42:46.915838Z qemu-kvm: -drive
>>
file=/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685,format=qcow2,if=none,id=drive-ua-b2b872cd-b468-4f14-ae20-555ed823e84b,serial=b2b872cd-b468-4f14-ae20-555ed823e84b,werror=stop,rerror=stop,cache=none,aio=native:
>> 'serial' is deprecated, please use the corresponding option of
'-device'
>> instead
>>
>> I assume this is the disk was writing to before it went down. Trying to
>> list the file gives an error and the file is not there;
>> ls -l
>>
/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685
>>
>> Is there a way we can recover the VM's disk images ?
>>
>> NOTE: No HostedEngine backups
>>
>> --
>> Regards,
>> Sakhi Hadebe
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FPTPLMY4TC4...
>>
>
--
Regards,
Sakhi Hadebe
Engineer: South African National Research Network (SANReN)Competency Area, Meraka, CSIR
Tel: +27 12 841 2308 <+27128414213>
Fax: +27 12 841 4223 <+27128414223>
Cell: +27 71 331 9622 <+27823034657>
Email: sakhi(a)sanren.ac.za <shadebe(a)csir.co.za>
5:15 a.m.
Adding to what me and my colleague shared
I am able to locate the disk images of the VMs, I copied some of them and tried to boot
them from another standalone kvm host, however booting the disk images wasn't
succesful as it landed on a rescue mode. The strange part is that the VM disk images are
64MB in size, which doesn't seem to be normal for a disk image(see the below command
extract).
root@gohan 019a7072-43d5-44b5-bb86-7a7327f02087]# pwd
/gluster_bricks/data/data/659de125-5671-4777-b27e-974aec0a4c9c/images/019a7072-43d5-44b5-bb86-7a7327f02087
[root@gohan 019a7072-43d5-44b5-bb86-7a7327f02087]# ll -h
total 66M
-rw-rw----. 2 vdsm kvm 64M Mar 22 13:48 f5f97478-6ccb-48bc-93b7-2fd5939f40bf
-rw-rw----. 2 vdsm kvm 1.0M Mar 4 12:10 f5f97478-6ccb-48bc-93b7-2fd5939f40bf.lease
-rw-r--r--. 2 vdsm kvm 317 Mar 22 11:06 f5f97478-6ccb-48bc-93b7-2fd5939f40bf.meta
Please share insights on how I can reconstruct the disk image so that it can become
bootable on the kvm host.
Thanks in advance for the reply.
9:53 a.m.
On Fri, Apr 12, 2019 at 11:16 AM <tau(a)sanren.ac.za> wrote:
Adding to what me and my colleague shared
I am able to locate the disk images of the VMs, I copied some of them and
tried to boot them from another standalone kvm host, however booting the
disk images wasn't succesful as it landed on a rescue mode. The strange
part is that the VM disk images are 64MB in size, which doesn't seem to be
normal for a disk image(see the below command extract).
root@gohan 019a7072-43d5-44b5-bb86-7a7327f02087]# pwd
/gluster_bricks/data/data/659de125-5671-4777-b27e-974aec0a4c9c/images/019a7072-43d5-44b5-bb86-7a7327f02087
[root@gohan 019a7072-43d5-44b5-bb86-7a7327f02087]# ll -h
total 66M
-rw-rw----. 2 vdsm kvm 64M Mar 22 13:48
f5f97478-6ccb-48bc-93b7-2fd5939f40bf
-rw-rw----. 2 vdsm kvm 1.0M Mar 4 12:10
f5f97478-6ccb-48bc-93b7-2fd5939f40bf.lease
-rw-r--r--. 2 vdsm kvm 317 Mar 22 11:06
f5f97478-6ccb-48bc-93b7-2fd5939f40bf.meta
Please share insights on how I can reconstruct the disk image so that it
can become bootable on the kvm host.
Thanks in advance for the reply.
I'd suggest to double check all the gluster logs because 64 M doesn't seem
anyhow reasonable there.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZIQNXSY5GAK...
4:13 p.m.
On Fri, Apr 12, 2019, 12:16 <tau(a)sanren.ac.za> wrote:
Adding to what me and my colleague shared
I am able to locate the disk images of the VMs, I copied some of them and
tried to boot them from another standalone kvm host, however booting the
disk images wasn't succesful as it landed on a rescue mode. The strange
part is that the VM disk images are 64MB in size, which doesn't seem to be
normal for a disk image(see the below command extract).
root@gohan 019a7072-43d5-44b5-bb86-7a7327f02087]# pwd
/gluster_bricks/data/data/659de125-5671-4777-b27e-974aec0a4c9c/images/019a7072-43d5-44b5-bb86-7a7327f02087
[root@gohan 019a7072-43d5-44b5-bb86-7a7327f02087]# ll -h
total 66M
-rw-rw----. 2 vdsm kvm 64M Mar 22 13:48
f5f97478-6ccb-48bc-93b7-2fd5939f40bf
-rw-rw----. 2 vdsm kvm 1.0M Mar 4 12:10
f5f97478-6ccb-48bc-93b7-2fd5939f40bf.lease
-rw-r--r--. 2 vdsm kvm 317 Mar 22 11:06
f5f97478-6ccb-48bc-93b7-2fd5939f40bf.meta
Id suggest to mount the gluster volume and get the disk image from there,
instead directly from the brick.
Please share insights on how I can reconstruct the disk image so that it
can become bootable on the kvm host.
Thanks in advance for the reply.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZIQNXSY5GAK...
10:41 a.m.
It looks like i've got the exact same issue;
drwxr-xr-x. 2 vdsm kvm 4.0K Mar 29 16:01 .
drwxr-xr-x. 22 vdsm kvm 4.0K Mar 29 18:34 ..
-rw-rw----. 1 vdsm kvm 64M Feb 4 01:32 44781cef-173a-4d84-88c5-18f7310037b4
-rw-rw----. 1 vdsm kvm 1.0M Oct 16 2018 44781cef-173a-4d84-88c5-18f7310037b4.lease
-rw-r--r--. 1 vdsm kvm 311 Mar 29 16:00 44781cef-173a-4d84-88c5-18f7310037b4.meta
Within the meta file the image is marked legal and reports a size of SIZE=41943040,
interestingly the format is mark RAW, while it was a thinly created volume.
My suspicion is that something went wrong while the volume was being livemigrated, and
somehow the merging of the images broke the volume.
11:04 a.m.
The data chunks are under .glusterfs folder on bricks now. Not a single
huge file you can easily access from a brick.
Not sure when that change was introduced though.
Fil
On Thu, May 9, 2019, 10:43 AM <olaf.buitelaar(a)gmail.com> wrote:
It looks like i've got the exact same issue;
drwxr-xr-x. 2 vdsm kvm 4.0K Mar 29 16:01 .
drwxr-xr-x. 22 vdsm kvm 4.0K Mar 29 18:34 ..
-rw-rw----. 1 vdsm kvm 64M Feb 4 01:32
44781cef-173a-4d84-88c5-18f7310037b4
-rw-rw----. 1 vdsm kvm 1.0M Oct 16 2018
44781cef-173a-4d84-88c5-18f7310037b4.lease
-rw-r--r--. 1 vdsm kvm 311 Mar 29 16:00
44781cef-173a-4d84-88c5-18f7310037b4.meta
Within the meta file the image is marked legal and reports a size of
SIZE=41943040, interestingly the format is mark RAW, while it was a thinly
created volume.
My suspicion is that something went wrong while the volume was being
livemigrated, and somehow the merging of the images broke the volume.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UDPJ2R5XUFN...
1:55 p.m.
This listing is from a gluster mount not from the underlying brick, which should combine
all parts from the underlying .glusterfs folder. I believe when you make use of the
feature.shard the files should be broken up in peaces according the shard-size.
Olaf
3:06 p.m.
Sorry, it wasn't clear from your post and earlier in the thread tau＠
sanren.ac.za was clearly listing data on the brick, not on a mounted
glusterfs volume.
--
Dmitry Filonov
Linux Administrator
SBGrid Core | Harvard Medical School
250 Longwood Ave, SGM-114
Boston, MA 02115
On Thu, May 9, 2019 at 1:56 PM <olaf.buitelaar(a)gmail.com> wrote:
This listing is from a gluster mount not from the underlying brick,
which
should combine all parts from the underlying .glusterfs folder. I believe
when you make use of the feature.shard the files should be broken up in
peaces according the shard-size.
Olaf
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JIEAGLEN67E...
5:43 a.m.
Hi Dimitry,
Sorry for not being clearer, I've missed the part the ls was from the underlying
brick. Than i've clearly a different issue.
Best Olaf
2 p.m.
New subject: Migrating Domain Storage Gluster
Basically i want to take out all of the HDD's in the main gluster pool,
and replace with SSD's.
My thought was to put everything in maintenance, copy the data manually
over to a transient storage server. Destroy the gluster volume, swap in
all the new drives, build a new gluster volume with the same name /
settings, move data back, and be done.
Any thoughts on this?
12:09 a.m.
New subject: Migrating Domain Storage Gluster
On Thu, May 9, 2019, 20:57 Alex McWhirter <alex(a)triadic.us> wrote:
Basically i want to take out all of the HDD's in the main gluster
pool,
and replace with SSD's.
My thought was to put everything in maintenance, copy the data manually
over to a transient storage server. Destroy the gluster volume, swap in
all the new drives, build a new gluster volume with the same name /
settings, move data back, and be done.
Any thoughts on this?
Seems ok if you can afford downtime. Otherwise you can go one node at a
time.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TP6V2OYG5ZX...
2054
days inactive
2085
days old
13 comments
7 participants
participants (7)
-
Alex K -
Alex McWhirter -
Dmitry Filonov -
olaf.buitelaar＠gmail.com -
Sakhi Hadebe -
Simone Tiraboschi -
tau＠sanren.ac.za