10:14 p.m.
On Sun, Oct 18, 2015 at 7:00 PM, Jaret Garcia <jaret.garcia(a)packet.mx> wrote:
Hi everyone,
Afew weeks ago we had a problem with the SPM and all host in the cluster got
stocked in contending, we restarted hosts one by one, and the issue was
solved. Howerver we didn't notice that one server even it never stop
running, it changed its state some way and then no changes could be done to
the VM, we tried to add more RAM and we saw the message "Cannot run VM. This
VM is not managed by the engine",
I would open a bug about this, and attach engine and vdsm logs showing the
timeframe of this event.
so we ssh the VM an send it to reboot, and
once we did that the VM never came back
Sure, if engine does not know this vm, it will never restart it. The
libvirt vm is not
persistent, engine is keeping the vm info in the engine database, and keeps the
vm up on some host.
, we still see the VM in the engine
administration but it does not show any information regarding to network,
disk, and so.
Please attach engine db dump to the bug, to understand what is "does not show
any information"
We created another VM to replace the services in the one we
lost, however we need to recover the files in the lost VM, we believe the
image should be in the storage but we haven't found a way to recover it,
some time ago we came across a similar situation but at that time it was a
NFS data domain, so it was easier for us to go inside the storage server an
search for the VM ID to scp the image and mount it somewhere else, this time
the storage is iscsi and even we found that the hosts mount the target in
/rhev/data-center/mnt/blockSD/ we only see there the active images for the
cluster, can anyone point us how we can recover the lost image? We know the
VM ID and the Disk ID from Ovirt.
To recover the images, you need the image id. If you don't see it in the engine
ui, you can try to search in the engine database.
(Adding Maor to help with finding the image id in the database)
The pool id can be found on the host in /rhev/data-center - there
should be one directory,
its name is the pool id. If you have more than one, use the one which
is not empty.
# Assuming this value (taken from my test setup)
pool_id = 591475db-6fa9-455d-9c05-7f6e30fb06d5
image_id = 5b10b1b9-ee82-46ee-9f3d-3659d37e4851
Once you found the image id, do:
# Update lvm metadata daemon
pvscan --cache
# Find the volumes
# lvs -o lv_name,vg_name,tags | awk '/IU_<image_id>/ {print $1,$2}'
2782e797-e49a-4364-99d7-d7544a42e939 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318
4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318
Now we know that:
domain_id = 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318
# Activate the lvs
lvchange -ay 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939
lvchange -ay 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12
# Find the top volume by running qemu-img info on all the lvs
# qemu-img info
/dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939
image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 0
cluster_size: 65536
Format specific information:
compat: 0.10
# qemu-img info
/dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12
image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 0
cluster_size: 65536
backing file:
../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939
(actual path:
/dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939)
backing file format: qcow2
Format specific information:
compat: 0.10
The top volume is the one with the largest number of items in the
"backing file" value.
In this case, it is
/dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12
So:
volume_id = 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12
# Prepare the image to create the links in /rhev/data-center
In a perfect wold, we could use the path to the lv /dev/vgname/lvname,
but the relative path
used by qemu is based on the directories and symbolic links created
inside /rhev/data-center
the easier way to created them is by preparing the image.
# vdsClient -s 0 prepareImage 591475db-6fa9-455d-9c05-7f6e30fb06d5
6c77adb1-74fc-4fa9-a0ac-3b5a4b789318
5b10b1b9-ee82-46ee-9f3d-3659d37e4851
4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12
{'domainID': '6c77adb1-74fc-4fa9-a0ac-3b5a4b789318',
'imageID': '5b10b1b9-ee82-46ee-9f3d-3659d37e4851',
'leaseOffset': 113246208,
'leasePath': '/dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/leases',
'path':
'/rhev/data-center/mnt/blockSD/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/images/5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12',
'volType': 'path',
'volumeID': '4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12'}
# Copy the volume data to some file system
I'm using raw, you may like to use qcow2
cd <some mountpoint>
qemu-img convert -p -O raw
/rhev/data-center/mnt/blockSD/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/images/5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12
saved-disk.img
# Teardown the image
# vdsClient -s 0 teardownImage 591475db-6fa9-455d-9c05-7f6e30fb06d5
6c77adb1-74fc-4fa9-a0ac-3b5a4b789318
5b10b1b9-ee82-46ee-9f3d-3659d37e4851
4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12
OK
# Check the saved image
# qemu-img info saved-disk.img
image: saved-disk.img
file format: raw
virtual size: 8.0G (8589934592 bytes)
disk size: 1.2G
You can mount this image and copy files, or copy the data to an empty
disk you created for the new vm.
Nir
Our Setup
ovirt version: 3.5.4 hosted engine
4 supermicro hosts running centos 7.1
1 iscsi storage server running Open-E DSS v7 Lite
Thanks in advance
Jaret
Email sent using Packet Mail - Email, Groupware and Calendaring for the
cloud!
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users