On Sun, Oct 18, 2015 at 10:14 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Oct 18, 2015 at 7:00 PM, Jaret Garcia <jaret.garcia@packet.mx> wrote:
Hi everyone,
Afew weeks ago we had a problem with the SPM and all host in the cluster got stocked in contending, we restarted hosts one by one, and the issue was solved. Howerver we didn't notice that one server even it never stop running, it changed its state some way and then no changes could be done to the VM, we tried to add more RAM and we saw the message "Cannot run VM. This VM is not managed by the engine",
I would open a bug about this, and attach engine and vdsm logs showing the timeframe of this event.
so we ssh the VM an send it to reboot, and once we did that the VM never came back
Sure, if engine does not know this vm, it will never restart it. The libvirt vm is not persistent, engine is keeping the vm info in the engine database, and keeps the vm up on some host.
, we still see the VM in the engine administration but it does not show any information regarding to network, disk, and so.
Please attach engine db dump to the bug, to understand what is "does not show any information"
We created another VM to replace the services in the one we lost, however we need to recover the files in the lost VM, we believe the image should be in the storage but we haven't found a way to recover it, some time ago we came across a similar situation but at that time it was a NFS data domain, so it was easier for us to go inside the storage server an search for the VM ID to scp the image and mount it somewhere else, this time the storage is iscsi and even we found that the hosts mount the target in /rhev/data-center/mnt/blockSD/ we only see there the active images for the cluster, can anyone point us how we can recover the lost image? We know the VM ID and the Disk ID from Ovirt.
To recover the images, you need the image id. If you don't see it in the engine ui, you can try to search in the engine database. (Adding Maor to help with finding the image id in the database)
The pool id can be found on the host in /rhev/data-center - there should be one directory, its name is the pool id. If you have more than one, use the one which is not empty.
# Assuming this value (taken from my test setup)
pool_id = 591475db-6fa9-455d-9c05-7f6e30fb06d5 image_id = 5b10b1b9-ee82-46ee-9f3d-3659d37e4851
Once you found the image id, do:
# Update lvm metadata daemon
pvscan --cache
# Find the volumes
# lvs -o lv_name,vg_name,tags | awk '/IU_<image_id>/ {print $1,$2}' 2782e797-e49a-4364-99d7-d7544a42e939 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318
Now we know that: domain_id = 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318
# Activate the lvs
lvchange -ay 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939 lvchange -ay 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12
# Find the top volume by running qemu-img info on all the lvs
# qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 Format specific information: compat: 0.10
# qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 backing file: ../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939 (actual path: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939) backing file format: qcow2 Format specific information: compat: 0.10
The top volume is the one with the largest number of items in the "backing file" value.
Correction: using the backing file, you can see the parent of each volume. The volume without the backing file is the base volume. The top volume is the volume which is not parent of any other volume. Here is an example with 3 volumes: # qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 Format specific information: compat: 0.10 This is the base volume. # qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 backing file: ../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939 (actual path: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939) backing file format: qcow2 Format specific information: compat: 0.10 This volume parent is 2782e797-e49a-4364-99d7-d7544a42e939 (the base volume) # qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/9de6a73e-49a6-45e6-b1aa-bc85e630bf39 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/9de6a73e-49a6-45e6-b1aa-bc85e630bf39 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 backing file: ../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 (actual path: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12) backing file format: qcow2 Format specific information: compat: 0.10 This volume parent is 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 (the volume above) So this is the top volume, which can be used to copy the volume data with qemu-img convert. Another way to find this info is using the volume metadata in /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/metadata but it may be stale; the canonical source of information is the qcow image reported by qemu-img. An easier way to get the information, is using getVolumesList - but this uses vdsm metadata, which may be stale (in disaster recovery context). # vdsClient -s 0 getVolumesList 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 591475db-6fa9-455d-9c05-7f6e30fb06d5 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 9de6a73e-49a6-45e6-b1aa-bc85e630bf39 : Parent is 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 2782e797-e49a-4364-99d7-d7544a42e939 : {"DiskAlias":"test_Disk1","DiskDescription":""}. 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 : Parent is 2782e797-e49a-4364-99d7-d7544a42e939 In ovirt 3.6, there is even easier way, using vdsm-tool dump-volumes-chains. This is based on getVolumesList, so it vdsm metadata is broken, you should use the lower level qemu-img info. # vdsm-tool dump-volume-chains 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 Images volume chains (base volume first) image: 55b41fbd-5e22-4f9d-b72f-aa7af9d7ccb8 - 78f22775-916c-4e72-8c5b-9917734b26da status: OK, voltype: SHARED, format: COW, legality: LEGAL, type: SPARSE image: 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 - 2782e797-e49a-4364-99d7-d7544a42e939 status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE - 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE - 9de6a73e-49a6-45e6-b1aa-bc85e630bf39 status: OK, voltype: LEAF, format: COW, legality: LEGAL, type: SPARSE image: 7ea9086c-c82d-405b-aabc-2d66f2106f6d - 3f400d56-4412-439c-af43-f379bb5160af status: OK, voltype: LEAF, format: RAW, legality: LEGAL, type: PREALLOCATED image: 8cd92346-555f-4a87-8415-ed681dc7a0a7 - cc56eca9-26c5-4428-8797-b3e7fa7a0c89 status: OK, voltype: LEAF, format: RAW, legality: LEGAL, type: PREALLOCATED
In this case, it is /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12
So: volume_id = 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12
# Prepare the image to create the links in /rhev/data-center
In a perfect wold, we could use the path to the lv /dev/vgname/lvname, but the relative path used by qemu is based on the directories and symbolic links created inside /rhev/data-center the easier way to created them is by preparing the image.
# vdsClient -s 0 prepareImage 591475db-6fa9-455d-9c05-7f6e30fb06d5 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 {'domainID': '6c77adb1-74fc-4fa9-a0ac-3b5a4b789318', 'imageID': '5b10b1b9-ee82-46ee-9f3d-3659d37e4851', 'leaseOffset': 113246208, 'leasePath': '/dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/leases', 'path': '/rhev/data-center/mnt/blockSD/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/images/5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12', 'volType': 'path', 'volumeID': '4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12'}
# Copy the volume data to some file system
I'm using raw, you may like to use qcow2
cd <some mountpoint> qemu-img convert -p -O raw /rhev/data-center/mnt/blockSD/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/images/5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 saved-disk.img
# Teardown the image
# vdsClient -s 0 teardownImage 591475db-6fa9-455d-9c05-7f6e30fb06d5 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 OK
# Check the saved image
# qemu-img info saved-disk.img image: saved-disk.img file format: raw virtual size: 8.0G (8589934592 bytes) disk size: 1.2G
You can mount this image and copy files, or copy the data to an empty disk you created for the new vm.
Nir
Our Setup ovirt version: 3.5.4 hosted engine 4 supermicro hosts running centos 7.1 1 iscsi storage server running Open-E DSS v7 Lite
Thanks in advance
Jaret Email sent using Packet Mail - Email, Groupware and Calendaring for the cloud!
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users