From jaret.garcia at packet.mx Sun Oct 18 12:39:36 2015 Content-Type: multipart/mixed; boundary="===============9135439298608185938==" MIME-Version: 1.0 From: Jaret Garcia To: users at ovirt.org Subject: [ovirt-users] This VM is not managed by the engine Date: Sun, 18 Oct 2015 11:00:02 -0500 Message-ID: --===============9135439298608185938== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --=3D_223255a2b9290ae41a14342812244984 Content-Type: text/plain; charset=3DUTF-8 Content-Transfer-Encoding: quoted-printable Hi everyone,=3D0A=3D0AAfew weeks ago we had a problem with the SPM and all = h=3D ost in the=3D0Acluster got stocked in contending, we restarted hosts one b= =3D y one, and=3D0Athe issue was solved. Howerver we didn't notice that one se= =3D rver even=3D0Ait never stop running, it changed its state some way and the= =3D n no=3D0Achanges could be done to the VM, we tried to add more RAM and we= =3D saw=3D0Athe message "Cannot run VM. This VM is not managed by the engine"= =3D , so=3D0Awe ssh the VM an send it to reboot, and once we did that the VM n= =3D ever=3D0Acame back, we still see the VM in the engine administration but i= =3D t=3D0Adoes not show any information regarding to network, disk, and so. We= =3D =3D0Acreated another VM to replace the services in the one we lost, howeve= =3D r=3D0Awe need to recover the files in the lost VM, we believe the image=3D0= A=3D should be in the storage but we haven't found a way to recover it,=3D0Asom= =3D e time ago we came across a similar situation but at that time it=3D0Awas= =3D a NFS data domain, so it was easier for us to go inside the=3D0Astorage s= =3D erver an search for the VM ID to scp the image and mount it=3D0Asomewhere= =3D else, this time the storage is iscsi and even we found that=3D0Athe hosts= =3D mount the target in /rhev/data-center/mnt/blockSD/=3DC2=3DA0=3DC2=3DA0 we= =3D0Aonl=3D y see there the active images for the cluster, can anyone point us=3D0Ahow= =3D we can recover the lost image?=3DC2=3DA0 We know the VM ID and the Disk ID= =3D =3D0Afrom Ovirt.=3D0A=3D0AOur Setup=3D0Aovirt version: 3.5.4 hosted engine= =3D0A4 s=3D upermicro hosts running centos 7.1=3D0A1 iscsi storage server running Open= =3D -E DSS v7 Lite=3D0A=3D0AThanks in advance=3D0A=3D0AJaret=3D0AEmail sent usi= ng Pack=3D et Mail - Email, Groupware and Calendaring for=3D0Athe cloud! --=3D_223255a2b9290ae41a14342812244984 Content-Type: text/html; charset=3DUTF-8 Content-Transfer-Encoding: quoted-printable Hi everyone,

Afew weeks ago we had a problem with the SPM=3D and all host in the cluster got stocked in contending, we restarted hos=3D ts one by one, and the issue was solved. Howerver we didn't notice that=3D one server even it never stop running, it changed its state some way an=3D d then no changes could be done to the VM, we tried to add more RAM and=3D we saw the message "Cannot run VM. This VM is not managed by the engine=3D ", so we ssh the VM an send it to reboot, and once we did that the VM ne=3D ver came back, we still see the VM in the engine administration but it d=3D oes not show any information regarding to network, disk, and so. We crea=3D ted another VM to replace the services in the one we lost, however we ne=3D ed to recover the files in the lost VM, we believe the image should be i=3D n the storage but we haven't found a way to recover it, some time ago we=3D came across a similar situation but at that time it was a NFS data doma=3D in, so it was easier for us to go inside the storage server an search fo=3D r the VM ID to scp the image and mount it somewhere else, this time the=3D storage is iscsi and even we found that the hosts mount the target in /=3D rhev/data-center/mnt/blockSD/=3DC2=3DA0=3DC2=3DA0 we only see there the act= ive i=3D mages for the cluster, can anyone point us how we can recover the lost i=3D mage?=3DC2=3DA0 We know the VM ID and the Disk ID from Ovirt.

Our Se= t=3D up
ovirt version: 3.5.4 hosted engine
4 supermicro hosts running c=3D entos 7.1
1 iscsi storage server running Open-E DSS v7 Lite

Th=3D anks in advance

Jaret
Email sent using Packet Mail - Email, Gr=3D oupware and Calendaring for the cloud! --=3D_223255a2b9290ae41a14342812244984-- --===============9135439298608185938== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS09XzIyMzI1NWEyYjkyOTBhZTQxYTE0MzQyODEyMjQ0OTg0CkNvbnRlbnQtVHlwZTogdGV4dC9w bGFpbjsgY2hhcnNldD1VVEYtOApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJp bnRhYmxlCgpIaSBldmVyeW9uZSw9MEE9MEFBZmV3IHdlZWtzIGFnbyB3ZSBoYWQgYSBwcm9ibGVt IHdpdGggdGhlIFNQTSBhbmQgYWxsIGg9Cm9zdCBpbiB0aGU9MEFjbHVzdGVyIGdvdCBzdG9ja2Vk IGluIGNvbnRlbmRpbmcsIHdlIHJlc3RhcnRlZCBob3N0cyBvbmUgYj0KeSBvbmUsIGFuZD0wQXRo ZSBpc3N1ZSB3YXMgc29sdmVkLiBIb3dlcnZlciB3ZSBkaWRuJ3Qgbm90aWNlIHRoYXQgb25lIHNl PQpydmVyIGV2ZW49MEFpdCBuZXZlciBzdG9wIHJ1bm5pbmcsIGl0IGNoYW5nZWQgaXRzIHN0YXRl IHNvbWUgd2F5IGFuZCB0aGU9Cm4gbm89MEFjaGFuZ2VzIGNvdWxkIGJlIGRvbmUgdG8gdGhlIFZN LCB3ZSB0cmllZCB0byBhZGQgbW9yZSBSQU0gYW5kIHdlPQogc2F3PTBBdGhlIG1lc3NhZ2UgIkNh bm5vdCBydW4gVk0uIFRoaXMgVk0gaXMgbm90IG1hbmFnZWQgYnkgdGhlIGVuZ2luZSI9Ciwgc289 MEF3ZSBzc2ggdGhlIFZNIGFuIHNlbmQgaXQgdG8gcmVib290LCBhbmQgb25jZSB3ZSBkaWQgdGhh dCB0aGUgVk0gbj0KZXZlcj0wQWNhbWUgYmFjaywgd2Ugc3RpbGwgc2VlIHRoZSBWTSBpbiB0aGUg ZW5naW5lIGFkbWluaXN0cmF0aW9uIGJ1dCBpPQp0PTBBZG9lcyBub3Qgc2hvdyBhbnkgaW5mb3Jt YXRpb24gcmVnYXJkaW5nIHRvIG5ldHdvcmssIGRpc2ssIGFuZCBzby4gV2U9Cj0wQWNyZWF0ZWQg YW5vdGhlciBWTSB0byByZXBsYWNlIHRoZSBzZXJ2aWNlcyBpbiB0aGUgb25lIHdlIGxvc3QsIGhv d2V2ZT0Kcj0wQXdlIG5lZWQgdG8gcmVjb3ZlciB0aGUgZmlsZXMgaW4gdGhlIGxvc3QgVk0sIHdl IGJlbGlldmUgdGhlIGltYWdlPTBBPQpzaG91bGQgYmUgaW4gdGhlIHN0b3JhZ2UgYnV0IHdlIGhh dmVuJ3QgZm91bmQgYSB3YXkgdG8gcmVjb3ZlciBpdCw9MEFzb209CmUgdGltZSBhZ28gd2UgY2Ft ZSBhY3Jvc3MgYSBzaW1pbGFyIHNpdHVhdGlvbiBidXQgYXQgdGhhdCB0aW1lIGl0PTBBd2FzPQog YSBORlMgZGF0YSBkb21haW4sIHNvIGl0IHdhcyBlYXNpZXIgZm9yIHVzIHRvIGdvIGluc2lkZSB0 aGU9MEFzdG9yYWdlIHM9CmVydmVyIGFuIHNlYXJjaCBmb3IgdGhlIFZNIElEIHRvIHNjcCB0aGUg aW1hZ2UgYW5kIG1vdW50IGl0PTBBc29tZXdoZXJlPQogZWxzZSwgdGhpcyB0aW1lIHRoZSBzdG9y YWdlIGlzIGlzY3NpIGFuZCBldmVuIHdlIGZvdW5kIHRoYXQ9MEF0aGUgaG9zdHM9CiBtb3VudCB0 aGUgdGFyZ2V0IGluIC9yaGV2L2RhdGEtY2VudGVyL21udC9ibG9ja1NELz1DMj1BMD1DMj1BMCB3 ZT0wQW9ubD0KeSBzZWUgdGhlcmUgdGhlIGFjdGl2ZSBpbWFnZXMgZm9yIHRoZSBjbHVzdGVyLCBj YW4gYW55b25lIHBvaW50IHVzPTBBaG93PQogd2UgY2FuIHJlY292ZXIgdGhlIGxvc3QgaW1hZ2U/ PUMyPUEwIFdlIGtub3cgdGhlIFZNIElEIGFuZCB0aGUgRGlzayBJRD0KPTBBZnJvbSBPdmlydC49 MEE9MEFPdXIgU2V0dXA9MEFvdmlydCB2ZXJzaW9uOiAzLjUuNCBob3N0ZWQgZW5naW5lPTBBNCBz PQp1cGVybWljcm8gaG9zdHMgcnVubmluZyBjZW50b3MgNy4xPTBBMSBpc2NzaSBzdG9yYWdlIHNl cnZlciBydW5uaW5nIE9wZW49Ci1FIERTUyB2NyBMaXRlPTBBPTBBVGhhbmtzIGluIGFkdmFuY2U9 MEE9MEFKYXJldD0wQUVtYWlsIHNlbnQgdXNpbmcgUGFjaz0KZXQgTWFpbCAtIEVtYWlsLCBHcm91 cHdhcmUgYW5kIENhbGVuZGFyaW5nIGZvcj0wQXRoZSBjbG91ZCEKCi0tPV8yMjMyNTVhMmI5Mjkw YWU0MWExNDM0MjgxMjI0NDk4NApDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD1VVEYt OApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCgo8aHRtbD48Ym9k eSBzdHlsZT0zRCJmb250LWZhbWlseTogSGVsdmV0aWNhLEFyaWFsLHNhbnMtc2VyaWY7IGZvbnQt c2l6ZTo9CiAxMnB4OyI+SGkgZXZlcnlvbmUsPGJyPjxicj5BZmV3IHdlZWtzIGFnbyB3ZSBoYWQg YSBwcm9ibGVtIHdpdGggdGhlIFNQTT0KIGFuZCBhbGwgaG9zdCBpbiB0aGUgY2x1c3RlciBnb3Qg c3RvY2tlZCBpbiBjb250ZW5kaW5nLCB3ZSByZXN0YXJ0ZWQgaG9zPQp0cyBvbmUgYnkgb25lLCBh bmQgdGhlIGlzc3VlIHdhcyBzb2x2ZWQuIEhvd2VydmVyIHdlIGRpZG4ndCBub3RpY2UgdGhhdD0K IG9uZSBzZXJ2ZXIgZXZlbiBpdCBuZXZlciBzdG9wIHJ1bm5pbmcsIGl0IGNoYW5nZWQgaXRzIHN0 YXRlIHNvbWUgd2F5IGFuPQpkIHRoZW4gbm8gY2hhbmdlcyBjb3VsZCBiZSBkb25lIHRvIHRoZSBW TSwgd2UgdHJpZWQgdG8gYWRkIG1vcmUgUkFNIGFuZD0KIHdlIHNhdyB0aGUgbWVzc2FnZSAiQ2Fu bm90IHJ1biBWTS4gVGhpcyBWTSBpcyBub3QgbWFuYWdlZCBieSB0aGUgZW5naW5lPQoiLCBzbyB3 ZSBzc2ggdGhlIFZNIGFuIHNlbmQgaXQgdG8gcmVib290LCBhbmQgb25jZSB3ZSBkaWQgdGhhdCB0 aGUgVk0gbmU9CnZlciBjYW1lIGJhY2ssIHdlIHN0aWxsIHNlZSB0aGUgVk0gaW4gdGhlIGVuZ2lu ZSBhZG1pbmlzdHJhdGlvbiBidXQgaXQgZD0Kb2VzIG5vdCBzaG93IGFueSBpbmZvcm1hdGlvbiBy ZWdhcmRpbmcgdG8gbmV0d29yaywgZGlzaywgYW5kIHNvLiBXZSBjcmVhPQp0ZWQgYW5vdGhlciBW TSB0byByZXBsYWNlIHRoZSBzZXJ2aWNlcyBpbiB0aGUgb25lIHdlIGxvc3QsIGhvd2V2ZXIgd2Ug bmU9CmVkIHRvIHJlY292ZXIgdGhlIGZpbGVzIGluIHRoZSBsb3N0IFZNLCB3ZSBiZWxpZXZlIHRo ZSBpbWFnZSBzaG91bGQgYmUgaT0KbiB0aGUgc3RvcmFnZSBidXQgd2UgaGF2ZW4ndCBmb3VuZCBh IHdheSB0byByZWNvdmVyIGl0LCBzb21lIHRpbWUgYWdvIHdlPQogY2FtZSBhY3Jvc3MgYSBzaW1p bGFyIHNpdHVhdGlvbiBidXQgYXQgdGhhdCB0aW1lIGl0IHdhcyBhIE5GUyBkYXRhIGRvbWE9Cmlu LCBzbyBpdCB3YXMgZWFzaWVyIGZvciB1cyB0byBnbyBpbnNpZGUgdGhlIHN0b3JhZ2Ugc2VydmVy IGFuIHNlYXJjaCBmbz0KciB0aGUgVk0gSUQgdG8gc2NwIHRoZSBpbWFnZSBhbmQgbW91bnQgaXQg c29tZXdoZXJlIGVsc2UsIHRoaXMgdGltZSB0aGU9CiBzdG9yYWdlIGlzIGlzY3NpIGFuZCBldmVu IHdlIGZvdW5kIHRoYXQgdGhlIGhvc3RzIG1vdW50IHRoZSB0YXJnZXQgaW4gLz0Kcmhldi9kYXRh LWNlbnRlci9tbnQvYmxvY2tTRC89QzI9QTA9QzI9QTAgd2Ugb25seSBzZWUgdGhlcmUgdGhlIGFj dGl2ZSBpPQptYWdlcyBmb3IgdGhlIGNsdXN0ZXIsIGNhbiBhbnlvbmUgcG9pbnQgdXMgaG93IHdl IGNhbiByZWNvdmVyIHRoZSBsb3N0IGk9Cm1hZ2U/PUMyPUEwIFdlIGtub3cgdGhlIFZNIElEIGFu ZCB0aGUgRGlzayBJRCBmcm9tIE92aXJ0Ljxicj48YnI+T3VyIFNldD0KdXA8YnI+b3ZpcnQgdmVy c2lvbjogMy41LjQgaG9zdGVkIGVuZ2luZTxicj40IHN1cGVybWljcm8gaG9zdHMgcnVubmluZyBj PQplbnRvcyA3LjE8YnI+MSBpc2NzaSBzdG9yYWdlIHNlcnZlciBydW5uaW5nIE9wZW4tRSBEU1Mg djcgTGl0ZTxicj48YnI+VGg9CmFua3MgaW4gYWR2YW5jZTxicj48YnI+SmFyZXQ8YnI+RW1haWwg c2VudCB1c2luZyBQYWNrZXQgTWFpbCAtIEVtYWlsLCBHcj0Kb3Vwd2FyZSBhbmQgQ2FsZW5kYXJp bmcgZm9yIHRoZSBjbG91ZCE8L2JvZHk+PC9odG1sPgoKLS09XzIyMzI1NWEyYjkyOTBhZTQxYTE0 MzQyODEyMjQ0OTg0LS0KCg== --===============9135439298608185938==-- From nsoffer at redhat.com Sun Oct 18 15:14:17 2015 Content-Type: multipart/mixed; boundary="===============5734509508864322946==" MIME-Version: 1.0 From: Nir Soffer To: users at ovirt.org Subject: Re: [ovirt-users] This VM is not managed by the engine Date: Sun, 18 Oct 2015 22:14:15 +0300 Message-ID: In-Reply-To: c7e4857224a6c89bf7cad81679defbb53019875f@webmail.packet.mx --===============5734509508864322946== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Sun, Oct 18, 2015 at 7:00 PM, Jaret Garcia wr= ote: > Hi everyone, > > Afew weeks ago we had a problem with the SPM and all host in the cluster = got > stocked in contending, we restarted hosts one by one, and the issue was > solved. Howerver we didn't notice that one server even it never stop > running, it changed its state some way and then no changes could be done = to > the VM, we tried to add more RAM and we saw the message "Cannot run VM. T= his > VM is not managed by the engine", I would open a bug about this, and attach engine and vdsm logs showing the timeframe of this event. > so we ssh the VM an send it to reboot, and > once we did that the VM never came back Sure, if engine does not know this vm, it will never restart it. The libvirt vm is not persistent, engine is keeping the vm info in the engine database, and keeps= the vm up on some host. > , we still see the VM in the engine > administration but it does not show any information regarding to network, > disk, and so. Please attach engine db dump to the bug, to understand what is "does not sh= ow any information" > We created another VM to replace the services in the one we > lost, however we need to recover the files in the lost VM, we believe the > image should be in the storage but we haven't found a way to recover it, > some time ago we came across a similar situation but at that time it was a > NFS data domain, so it was easier for us to go inside the storage server = an > search for the VM ID to scp the image and mount it somewhere else, this t= ime > the storage is iscsi and even we found that the hosts mount the target in > /rhev/data-center/mnt/blockSD/ we only see there the active images for = the > cluster, can anyone point us how we can recover the lost image? We know = the > VM ID and the Disk ID from Ovirt. To recover the images, you need the image id. If you don't see it in the en= gine ui, you can try to search in the engine database. (Adding Maor to help with finding the image id in the database) The pool id can be found on the host in /rhev/data-center - there should be one directory, its name is the pool id. If you have more than one, use the one which is not empty. # Assuming this value (taken from my test setup) pool_id =3D 591475db-6fa9-455d-9c05-7f6e30fb06d5 image_id =3D 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 Once you found the image id, do: # Update lvm metadata daemon pvscan --cache # Find the volumes # lvs -o lv_name,vg_name,tags | awk '/IU_/ {print $1,$2}' 2782e797-e49a-4364-99d7-d7544a42e939 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 Now we know that: domain_id =3D 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 # Activate the lvs lvchange -ay 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d= 7544a42e939 lvchange -ay 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0= aaab3f2aa12 # Find the top volume by running qemu-img info on all the lvs # qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e= 939 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7= 544a42e939 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 Format specific information: compat: 0.10 # qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2a= a12 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0a= aab3f2aa12 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 backing file: ../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99= d7-d7544a42e939 (actual path: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/../5b10b1b9-ee82-46= ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939) backing file format: qcow2 Format specific information: compat: 0.10 The top volume is the one with the largest number of items in the "backing file" value. In this case, it is /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2a= a12 So: volume_id =3D 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 # Prepare the image to create the links in /rhev/data-center In a perfect wold, we could use the path to the lv /dev/vgname/lvname, but the relative path used by qemu is based on the directories and symbolic links created inside /rhev/data-center the easier way to created them is by preparing the image. # vdsClient -s 0 prepareImage 591475db-6fa9-455d-9c05-7f6e30fb06d5 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 {'domainID': '6c77adb1-74fc-4fa9-a0ac-3b5a4b789318', 'imageID': '5b10b1b9-ee82-46ee-9f3d-3659d37e4851', 'leaseOffset': 113246208, 'leasePath': '/dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/leases', 'path': '/rhev/data-center/mnt/blockSD/6c77adb1-74fc-4fa9-a0ac-3b5a4b78931= 8/images/5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab= 3f2aa12', 'volType': 'path', 'volumeID': '4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12'} # Copy the volume data to some file system I'm using raw, you may like to use qcow2 cd qemu-img convert -p -O raw /rhev/data-center/mnt/blockSD/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/images/5= b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 saved-disk.img # Teardown the image # vdsClient -s 0 teardownImage 591475db-6fa9-455d-9c05-7f6e30fb06d5 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 OK # Check the saved image # qemu-img info saved-disk.img image: saved-disk.img file format: raw virtual size: 8.0G (8589934592 bytes) disk size: 1.2G You can mount this image and copy files, or copy the data to an empty disk you created for the new vm. Nir > > Our Setup > ovirt version: 3.5.4 hosted engine > 4 supermicro hosts running centos 7.1 > 1 iscsi storage server running Open-E DSS v7 Lite > > Thanks in advance > > Jaret > Email sent using Packet Mail - Email, Groupware and Calendaring for the > cloud! > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > --===============5734509508864322946==-- From mlipchuk at redhat.com Sun Oct 18 15:52:40 2015 Content-Type: multipart/mixed; boundary="===============0104973589395302888==" MIME-Version: 1.0 From: Maor Lipchuk To: users at ovirt.org Subject: Re: [ovirt-users] This VM is not managed by the engine Date: Sun, 18 Oct 2015 15:52:35 -0400 Message-ID: <489488469.56370242.1445197955579.JavaMail.zimbra@redhat.com> In-Reply-To: CAMRbyyt_cxxVsS2nTq4n1kfYJ7oaUp3q3Y17BiqZsaE33b4WCg@mail.gmail.com --===============0104973589395302888== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----- Original Message ----- > From: "Nir Soffer" > To: "Jaret Garcia" > Cc: "users" , "Maor Lipchuk" > Sent: Sunday, October 18, 2015 10:14:15 PM > Subject: Re: [ovirt-users] This VM is not managed by the engine > = > On Sun, Oct 18, 2015 at 7:00 PM, Jaret Garcia = wrote: > > Hi everyone, > > > > Afew weeks ago we had a problem with the SPM and all host in the cluster > > got > > stocked in contending, we restarted hosts one by one, and the issue was > > solved. Howerver we didn't notice that one server even it never stop > > running, it changed its state some way and then no changes could be don= e to > > the VM, we tried to add more RAM and we saw the message "Cannot run VM. > > This > > VM is not managed by the engine", > = > I would open a bug about this, and attach engine and vdsm logs showing the > timeframe of this event. > = > > so we ssh the VM an send it to reboot, and > > once we did that the VM never came back > = > Sure, if engine does not know this vm, it will never restart it. The > libvirt vm is not > persistent, engine is keeping the vm info in the engine database, and kee= ps > the > vm up on some host. > = > > , we still see the VM in the engine > > administration but it does not show any information regarding to networ= k, > > disk, and so. > = > Please attach engine db dump to the bug, to understand what is "does not = show > any information" > = > > We created another VM to replace the services in the one we > > lost, however we need to recover the files in the lost VM, we believe t= he > > image should be in the storage but we haven't found a way to recover it, > > some time ago we came across a similar situation but at that time it wa= s a > > NFS data domain, so it was easier for us to go inside the storage serve= r an > > search for the VM ID to scp the image and mount it somewhere else, this > > time > > the storage is iscsi and even we found that the hosts mount the target = in > > /rhev/data-center/mnt/blockSD/ we only see there the active images for > > the > > cluster, can anyone point us how we can recover the lost image? We know > > the > > VM ID and the Disk ID from Ovirt. > = > To recover the images, you need the image id. If you don't see it in the > engine > ui, you can try to search in the engine database. > (Adding Maor to help with finding the image id in the database) Hi Jaret, If you know the image id and you don't see the disk in the UI you can try t= o register it. Please take a look at http://www.ovirt.org/Features/ImportStorageDomain#Reg= ister_an_unregistered_disk how to add an unregistered disk. Let me know if that helps Regards, Maor > = > The pool id can be found on the host in /rhev/data-center - there > should be one directory, > its name is the pool id. If you have more than one, use the one which > is not empty. > = > # Assuming this value (taken from my test setup) > = > pool_id =3D 591475db-6fa9-455d-9c05-7f6e30fb06d5 > image_id =3D 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 > = > Once you found the image id, do: > = > # Update lvm metadata daemon > = > pvscan --cache > = > # Find the volumes > = > # lvs -o lv_name,vg_name,tags | awk '/IU_/ {print $1,$2}' > 2782e797-e49a-4364-99d7-d7544a42e939 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > = > Now we know that: > domain_id =3D 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > = > # Activate the lvs > = > lvchange -ay > 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e939 > lvchange -ay > 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > = > # Find the top volume by running qemu-img info on all the lvs > = > # qemu-img info > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a4= 2e939 > image: > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a4= 2e939 > file format: qcow2 > virtual size: 8.0G (8589934592 bytes) > disk size: 0 > cluster_size: 65536 > Format specific information: > compat: 0.10 > = > # qemu-img info > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f= 2aa12 > image: > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f= 2aa12 > file format: qcow2 > virtual size: 8.0G (8589934592 bytes) > disk size: 0 > cluster_size: 65536 > backing file: > ../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e= 939 > (actual path: > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/../5b10b1b9-ee82-46ee-9f3d-3659= d37e4851/2782e797-e49a-4364-99d7-d7544a42e939) > backing file format: qcow2 > Format specific information: > compat: 0.10 > = > The top volume is the one with the largest number of items in the > "backing file" value. > In this case, it is > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f= 2aa12 > = > So: > volume_id =3D 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > = > # Prepare the image to create the links in /rhev/data-center > = > In a perfect wold, we could use the path to the lv /dev/vgname/lvname, > but the relative path > used by qemu is based on the directories and symbolic links created > inside /rhev/data-center > the easier way to created them is by preparing the image. > = > # vdsClient -s 0 prepareImage 591475db-6fa9-455d-9c05-7f6e30fb06d5 > 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 > 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > {'domainID': '6c77adb1-74fc-4fa9-a0ac-3b5a4b789318', > 'imageID': '5b10b1b9-ee82-46ee-9f3d-3659d37e4851', > 'leaseOffset': 113246208, > 'leasePath': '/dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/leases', > 'path': > '/rhev/data-center/mnt/blockSD/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/imag= es/5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa1= 2', > 'volType': 'path', > 'volumeID': '4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12'} > = > # Copy the volume data to some file system > = > I'm using raw, you may like to use qcow2 > = > cd > qemu-img convert -p -O raw > /rhev/data-center/mnt/blockSD/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/images= /5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > saved-disk.img > = > # Teardown the image > = > # vdsClient -s 0 teardownImage 591475db-6fa9-455d-9c05-7f6e30fb06d5 > 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 > 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > OK > = > # Check the saved image > = > # qemu-img info saved-disk.img > image: saved-disk.img > file format: raw > virtual size: 8.0G (8589934592 bytes) > disk size: 1.2G > = > You can mount this image and copy files, or copy the data to an empty > disk you created for the new vm. > = > Nir > = > > > > Our Setup > > ovirt version: 3.5.4 hosted engine > > 4 supermicro hosts running centos 7.1 > > 1 iscsi storage server running Open-E DSS v7 Lite > > > > Thanks in advance > > > > Jaret > > Email sent using Packet Mail - Email, Groupware and Calendaring for the > > cloud! > > > > _______________________________________________ > > Users mailing list > > Users(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > > >=20 --===============0104973589395302888==-- From nsoffer at redhat.com Mon Oct 19 02:34:15 2015 Content-Type: multipart/mixed; boundary="===============8109624224788679223==" MIME-Version: 1.0 From: Nir Soffer To: users at ovirt.org Subject: Re: [ovirt-users] This VM is not managed by the engine Date: Mon, 19 Oct 2015 09:34:13 +0300 Message-ID: In-Reply-To: CAMRbyyt_cxxVsS2nTq4n1kfYJ7oaUp3q3Y17BiqZsaE33b4WCg@mail.gmail.com --===============8109624224788679223== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Sun, Oct 18, 2015 at 10:14 PM, Nir Soffer wrote: > On Sun, Oct 18, 2015 at 7:00 PM, Jaret Garcia = wrote: >> Hi everyone, >> >> Afew weeks ago we had a problem with the SPM and all host in the cluster= got >> stocked in contending, we restarted hosts one by one, and the issue was >> solved. Howerver we didn't notice that one server even it never stop >> running, it changed its state some way and then no changes could be done= to >> the VM, we tried to add more RAM and we saw the message "Cannot run VM. = This >> VM is not managed by the engine", > > I would open a bug about this, and attach engine and vdsm logs showing the > timeframe of this event. > >> so we ssh the VM an send it to reboot, and >> once we did that the VM never came back > > Sure, if engine does not know this vm, it will never restart it. The > libvirt vm is not > persistent, engine is keeping the vm info in the engine database, and kee= ps the > vm up on some host. > >> , we still see the VM in the engine >> administration but it does not show any information regarding to network, >> disk, and so. > > Please attach engine db dump to the bug, to understand what is "does not = show > any information" > >> We created another VM to replace the services in the one we >> lost, however we need to recover the files in the lost VM, we believe the >> image should be in the storage but we haven't found a way to recover it, >> some time ago we came across a similar situation but at that time it was= a >> NFS data domain, so it was easier for us to go inside the storage server= an >> search for the VM ID to scp the image and mount it somewhere else, this = time >> the storage is iscsi and even we found that the hosts mount the target in >> /rhev/data-center/mnt/blockSD/ we only see there the active images for= the >> cluster, can anyone point us how we can recover the lost image? We know= the >> VM ID and the Disk ID from Ovirt. > > To recover the images, you need the image id. If you don't see it in the = engine > ui, you can try to search in the engine database. > (Adding Maor to help with finding the image id in the database) > > The pool id can be found on the host in /rhev/data-center - there > should be one directory, > its name is the pool id. If you have more than one, use the one which > is not empty. > > # Assuming this value (taken from my test setup) > > pool_id =3D 591475db-6fa9-455d-9c05-7f6e30fb06d5 > image_id =3D 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 > > Once you found the image id, do: > > # Update lvm metadata daemon > > pvscan --cache > > # Find the volumes > > # lvs -o lv_name,vg_name,tags | awk '/IU_/ {print $1,$2}' > 2782e797-e49a-4364-99d7-d7544a42e939 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > > Now we know that: > domain_id =3D 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > > # Activate the lvs > > lvchange -ay 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7= -d7544a42e939 > lvchange -ay 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0= -0aaab3f2aa12 > > # Find the top volume by running qemu-img info on all the lvs > > # qemu-img info > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a4= 2e939 > image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-= d7544a42e939 > file format: qcow2 > virtual size: 8.0G (8589934592 bytes) > disk size: 0 > cluster_size: 65536 > Format specific information: > compat: 0.10 > > # qemu-img info > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f= 2aa12 > image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-= 0aaab3f2aa12 > file format: qcow2 > virtual size: 8.0G (8589934592 bytes) > disk size: 0 > cluster_size: 65536 > backing file: ../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-= 99d7-d7544a42e939 > (actual path: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/../5b10b1b9-ee82-= 46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939) > backing file format: qcow2 > Format specific information: > compat: 0.10 > > The top volume is the one with the largest number of items in the > "backing file" value. Correction: using the backing file, you can see the parent of each volume. The volume without the backing file is the base volume. The top volume is the volume which is not parent of any other volume. Here is an example with 3 volumes: # qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7544a42e= 939 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/2782e797-e49a-4364-99d7-d7= 544a42e939 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 Format specific information: compat: 0.10 This is the base volume. # qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f2a= a12 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0a= aab3f2aa12 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 backing file: ../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/2782e797-e49a-4364-99= d7-d7544a42e939 (actual path: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/../5b10b1b9-ee82-46= ee-9f3d-3659d37e4851/2782e797-e49a-4364-99d7-d7544a42e939) backing file format: qcow2 Format specific information: compat: 0.10 This volume parent is 2782e797-e49a-4364-99d7-d7544a42e939 (the base volume) # qemu-img info /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/9de6a73e-49a6-45e6-b1aa-bc85e630b= f39 image: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/9de6a73e-49a6-45e6-b1aa-bc= 85e630bf39 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 0 cluster_size: 65536 backing file: ../5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2= d0-0aaab3f2aa12 (actual path: /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/../5b10b1b9-ee82-46= ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12) backing file format: qcow2 Format specific information: compat: 0.10 This volume parent is 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 (the volume abov= e) So this is the top volume, which can be used to copy the volume data with qemu-img convert. Another way to find this info is using the volume metadata in /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/metadata but it may be stale; the canonical source of information is the qcow image reported by qemu-img. An easier way to get the information, is using getVolumesList - but this uses vdsm metadata, which may be stale (in disaster recovery context). # vdsClient -s 0 getVolumesList 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 591475db-6fa9-455d-9c05-7f6e30fb06d5 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 9de6a73e-49a6-45e6-b1aa-bc85e630bf39 : Parent is 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 2782e797-e49a-4364-99d7-d7544a42e939 : {"DiskAlias":"test_Disk1","DiskDescription":""}. 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 : Parent is 2782e797-e49a-4364-99d7-d7544a42e939 In ovirt 3.6, there is even easier way, using vdsm-tool dump-volumes-chains. This is based on getVolumesList, so it vdsm metadata is broken, you should use the lower level qemu-img info. # vdsm-tool dump-volume-chains 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 Images volume chains (base volume first) image: 55b41fbd-5e22-4f9d-b72f-aa7af9d7ccb8 - 78f22775-916c-4e72-8c5b-9917734b26da status: OK, voltype: SHARED, format: COW, legality: LEGAL, type: SPARSE image: 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 - 2782e797-e49a-4364-99d7-d7544a42e939 status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE - 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE - 9de6a73e-49a6-45e6-b1aa-bc85e630bf39 status: OK, voltype: LEAF, format: COW, legality: LEGAL, type: SPARSE image: 7ea9086c-c82d-405b-aabc-2d66f2106f6d - 3f400d56-4412-439c-af43-f379bb5160af status: OK, voltype: LEAF, format: RAW, legality: LEGAL, type: PREALLOCATED image: 8cd92346-555f-4a87-8415-ed681dc7a0a7 - cc56eca9-26c5-4428-8797-b3e7fa7a0c89 status: OK, voltype: LEAF, format: RAW, legality: LEGAL, type: PREALLOCATED > In this case, it is > /dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/4bc34865-64b8-4a6c-b2d0-0aaab3f= 2aa12 > > So: > volume_id =3D 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > > # Prepare the image to create the links in /rhev/data-center > > In a perfect wold, we could use the path to the lv /dev/vgname/lvname, > but the relative path > used by qemu is based on the directories and symbolic links created > inside /rhev/data-center > the easier way to created them is by preparing the image. > > # vdsClient -s 0 prepareImage 591475db-6fa9-455d-9c05-7f6e30fb06d5 > 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 > 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > {'domainID': '6c77adb1-74fc-4fa9-a0ac-3b5a4b789318', > 'imageID': '5b10b1b9-ee82-46ee-9f3d-3659d37e4851', > 'leaseOffset': 113246208, > 'leasePath': '/dev/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/leases', > 'path': '/rhev/data-center/mnt/blockSD/6c77adb1-74fc-4fa9-a0ac-3b5a4b789= 318/images/5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aa= ab3f2aa12', > 'volType': 'path', > 'volumeID': '4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12'} > > # Copy the volume data to some file system > > I'm using raw, you may like to use qcow2 > > cd > qemu-img convert -p -O raw > /rhev/data-center/mnt/blockSD/6c77adb1-74fc-4fa9-a0ac-3b5a4b789318/images= /5b10b1b9-ee82-46ee-9f3d-3659d37e4851/4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > saved-disk.img > > # Teardown the image > > # vdsClient -s 0 teardownImage 591475db-6fa9-455d-9c05-7f6e30fb06d5 > 6c77adb1-74fc-4fa9-a0ac-3b5a4b789318 > 5b10b1b9-ee82-46ee-9f3d-3659d37e4851 > 4bc34865-64b8-4a6c-b2d0-0aaab3f2aa12 > OK > > # Check the saved image > > # qemu-img info saved-disk.img > image: saved-disk.img > file format: raw > virtual size: 8.0G (8589934592 bytes) > disk size: 1.2G > > You can mount this image and copy files, or copy the data to an empty > disk you created for the new vm. > > Nir > >> >> Our Setup >> ovirt version: 3.5.4 hosted engine >> 4 supermicro hosts running centos 7.1 >> 1 iscsi storage server running Open-E DSS v7 Lite >> >> Thanks in advance >> >> Jaret >> Email sent using Packet Mail - Email, Groupware and Calendaring for the >> cloud! >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> --===============8109624224788679223==--