[ovirt-users] Unregistered disks - snapshot images present(Was Re: Import storage domain - disks not listed)

There is this bug https://bugzilla.redhat.com/1270562 which forces OVF_STORE update but it is not yet implemented. On Tue, May 3, 2016 at 8:52 AM, Sahina Bose <sabose(a)redhat.com&gt; wrote: > > On 05/02/2016 09:36 PM, Maor Lipchuk wrote: >> On Mon, May 2, 2016 at 5:45 PM, Sahina Bose <sabose(a)redhat.com&gt; wrote: >>> >>> >>> On 05/02/2016 05:57 PM, Maor Lipchuk wrote: >>> >>> >>> >>> On Mon, May 2, 2016 at 1:08 PM, Sahina Bose <sabose(a)redhat.com&gt; wrote: >>>> >>>> >>>> On 05/02/2016 03:15 PM, Maor Lipchuk wrote: >>>> >>>> >>>> >>>> On Mon, May 2, 2016 at 12:29 PM, Sahina Bose <sabose(a)redhat.com&gt; wrote: >>>>> >>>>> >>>>> On 05/01/2016 05:33 AM, Maor Lipchuk wrote: >>>>> >>>>> Hi Sahina, >>>>> >>>>> The disks with snapshots should be part of the VMs, once you will >>>>> register those VMs you should see those disks in the disks sub tab. >>>>> >>>>> >>>>> Maor, >>>>> >>>>> I was unable to import VM which prompted question - I assumed we had to >>>>> register disks first. So maybe I need to troubleshoot why I could not import >>>>> VMs from the domain first. >>>>> It fails with an error "Image does not exist". Where does it look for >>>>> volume IDs to pass to GetImageInfoVDSCommand - the OVF disk? >>>>> >>>>> >>>>> In engine.log >>>>> >>>>> 2016-05-02 04:15:14,812 ERROR >>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] >>>>> (ajp-/127.0.0.1:8702-1) [32f0b27c] Ir >>>>> sBroker::getImageInfo::Failed getting image info >>>>> imageId='6f4da17a-05a2-4d77-8091-d2fca3bbea1c' does not exist on >>>>> domainName='sahinasl >>>>> ave', domainId='5e1a37cf-933d-424c-8e3d-eb9e40b690a7', error code: >>>>> 'VolumeDoesNotExist', message: Volume does not exist: (u'6f4da17a-0 >>>>> 5a2-4d77-8091-d2fca3bbea1c',) >>>>> 2016-05-02 04:15:14,814 WARN >>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] >>>>> (ajp-/127.0.0.1:8702-1) [32f0b27c] >>>>> executeIrsBrokerCommand: getImageInfo on >>>>> '6f4da17a-05a2-4d77-8091-d2fca3bbea1c' threw an exception - assuming image >>>>> doesn't exist: IRS >>>>> GenericException: IRSErrorException: VolumeDoesNotExist >>>>> 2016-05-02 04:15:14,814 INFO >>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] >>>>> (ajp-/127.0.0.1:8702-1) [32f0b27c] >>>>> FINISH, DoesImageExistVDSCommand, return: false, log id: 3366f39b >>>>> 2016-05-02 04:15:14,814 WARN >>>>> [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] >>>>> (ajp-/127.0.0.1:8702-1) [32f0b27c] CanDoAction of action >>>>> 'ImportVmFromConfiguration' failed for user admin@internal. Reasons: >>>>> VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_IMAGE_DOES_NOT_EXIST >>>>> >>>>> >>>>> >>>>> jsonrpc.Executor/2::DEBUG::2016-05-02 >>>>> 13:45:13,903::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling >>>>> 'Volume.getInfo' in >>>>> bridge with {u'imageID': u'c52e4e02-dc6c-4a77-a184-9fcab88106c2', >>>>> u'storagepoolID': u'46ac4975-a84e-4e76-8e73-7971d0dadf0b', u'volumeI >>>>> D': u'6f4da17a-05a2-4d77-8091-d2fca3bbea1c', u'storagedomainID': >>>>> u'5e1a37cf-933d-424c-8e3d-eb9e40b690a7'} >>>>> >>>>> jsonrpc.Executor/2::DEBUG::2016-05-02 >>>>> 13:45:13,910::fileVolume::535::Storage.Volume::(validateVolumePath) validate >>>>> path for 6f4da17a-05a2-4d77-8091-d2fca3bbea1c >>>>> jsonrpc.Executor/2::ERROR::2016-05-02 >>>>> 13:45:13,914::task::866::Storage.TaskManager.Task::(_setError) >>>>> Task=`94dba16f-f7eb-439e-95e2-a04b34b92f84`::Unexpected error >>>>> Traceback (most recent call last): >>>>> File "/usr/share/vdsm/storage/task.py", line 873, in _run >>>>> return fn(*args, **kargs) >>>>> File "/usr/share/vdsm/logUtils.py", line 49, in wrapper >>>>> res = f(*args, **kwargs) >>>>> File "/usr/share/vdsm/storage/hsm.py", line 3162, in getVolumeInfo >>>>> volUUID=volUUID).getInfo() >>>>> File "/usr/share/vdsm/storage/sd.py", line 457, in produceVolume >>>>> volUUID) >>>>> File "/usr/share/vdsm/storage/glusterVolume.py", line 16, in >>>>> __init__ >>>>> volUUID) >>>>> File "/usr/share/vdsm/storage/fileVolume.py", line 58, in __init__ >>>>> volume.Volume.__init__(self, repoPath, sdUUID, imgUUID, volUUID) >>>>> File "/usr/share/vdsm/storage/volume.py", line 181, in __init__ >>>>> self.validate() >>>>> File "/usr/share/vdsm/storage/volume.py", line 194, in validate >>>>> self.validateVolumePath() >>>>> File "/usr/share/vdsm/storage/fileVolume.py", line 540, in >>>>> validateVolumePath >>>>> raise se.VolumeDoesNotExist(self.volUUID) >>>>> VolumeDoesNotExist: Volume does not exist: >>>>> (u'6f4da17a-05a2-4d77-8091-d2fca3bbea1c',) >>>>> >>>>> When I look at the tree output - there's no >>>>> 6f4da17a-05a2-4d77-8091-d2fca3bbea1c file. >>>>> >>>>> >>>>> ├── c52e4e02-dc6c-4a77-a184-9fcab88106c2 >>>>> │ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659 >>>>> │ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659.lease >>>>> │ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659.meta >>>>> │ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2 >>>>> │ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2.lease >>>>> │ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2.meta >>>>> │ │ │ ├── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa >>>>> │ │ │ ├── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa.lease >>>>> │ │ │ └── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa.meta >>>> >>>> >>>> Usually the "image does not exists" message is prompted once the VM's >>>> disk is managed in a different storage domain which were not imported yet. >>>> >>>> Few questions: >>>> 1. Were there any other Storage Domain which are not present in the >>>> setup? >>>> >>>> >>>> In the original RHEV instance - there were 3 storage domain >>>> i) Hosted engine storage domain: engine >>>> ii) Master data domain: vmstore >>>> iii) An export domain: expVol (no data here) >>>> >>>> To my backup RHEV server, I only imported vmstore. >>> >>> Just to be sure, can you look into the server which the engine storage >>> domain resides on. >>> Is there a chance that 6f4da17a-05a2-4d77-8091-d2fca3bbea1c could be >>> there? >>> >>> Also do you have an old engine log, is there a chance to look for image >>> 6f4da17a-05a2-4d77-8091-d2fca3bbea1c in there, to see when it was created >>> and if there were any problems in the process. >>> >>> >>> >>> I checked the engine logs and it seems that image >>> 6f4da17a-05a2-4d77-8091-d2fca3bbea1c was removed as part of snapshot merge. >>> Could it be that the OVF was not updated? >> >> >> The OVF_STORE is being update every 60 minutes or when a Storage >> Domain is moved to maintenance. >> If the DR process was done before the OVF_STORE got updated and the >> setup was destroyed then that might be the reason for the missing >> image in the OVF. >> >> The problem is that this image id is already part of the images chain >> configured in the VM's OVF. >> As a work around maybe you can create a new volume with VdsClient >> createVolume on the Host with the same UUID. >> or maybe to merge all the disk's snapshots through the vdsClient and >> once there will be only one volume you can register this disk as >> floating disk. > > Is there a way via API to update OVF_STORE? > This way before the DR process we can ensure that the OVF_STORE is referring > to the correct images. > > >> >>> engine.log-20160426.gz:2016-04-26 01:54:28,576 INFO >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] >>> (pool-5-thread-1) [6f94943f] START, MergeVDSCommand(HostName = rhsdev9, >>> MergeVDSCommandParameters:{runAsync='true', >>> hostId='b9a662bf-e05e-4e6a-9dfe-ec1be76d48e7', >>> vmId='e2b89e45-6f99-465d-aa08-fc4f746f0dd0', >>> storagePoolId='00000001-0001-0001-0001-000000000305', >>> storageDomainId='5e1a37cf-933d-424c-8e3d-eb9e40b690a7', >>> imageGroupId='c52e4e02-dc6c-4a77-a184-9fcab88106c2', >>> imageId='6f4da17a-05a2-4d77-8091-d2fca3bbea1c', >>> baseImageId='90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa', >>> topImageId='6f4da17a-05a2-4d77-8091-d2fca3bbea1c', bandwidth='0'}), log id: >>> 5a65a864 >>> engine.log-20160426.gz:2016-04-26 01:55:03,629 INFO >>> [org.ovirt.engine.core.bll.MergeCommandCallback] >>> (DefaultQuartzScheduler_Worker-57) [6f94943f] Merge command has completed >>> for images >>> '90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa'..'6f4da17a-05a2-4d77-8091-d2fca3bbea1c' >>> engine.log-20160426.gz:2016-04-26 01:55:06,707 INFO >>> [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-5-thread-2) [3c72e9f8] >>> Successfully removed volume(s): [6f4da17a-05a2-4d77-8091-d2fca3bbea1c] >>> >>> >>> >>>> 2. Can you look for the image id 6f4da17a-05a2-4d77-8091-d2fca3bbea1c in >>>> your storage server (Search on all the rest of the storage domains)? >>>> >>>> >>>> No - there was no file with this id (checked both in the original RHEV >>>> instance) >>>> [root@dhcp42-105 mnt]# pwd >>>> /rhev/data-center/mnt >>>> [root@dhcp42-105 mnt]# find . -name >>>> 6f4da17a-05a2-4d77-8091-d2fca3bbea1c >>>> [root@dhcp42-105 mnt]# >>>> >>>> >>>> Were there any operations being done on the VM before the recovery such >>>> as remove disk, move disk, or a new creation of a disk? >>>> >>>> >>>> No. The only operation done was creation of a snapshot - and it >>>> completed before the recovery was attempted. >>>> >>>> >>>> Regards, >>>> Maor >>>> >>>>> Regarding floating disks (without snapshots), you can register them >>>>> through REST. >>>>> If you are working on the master branch there should be a sub tab >>>>> dedicated for those also. >>>>> >>>>> Regards, >>>>> Maor >>>>> >>>>> On Tue, Apr 26, 2016 at 1:44 PM, Sahina Bose <sabose(a)redhat.com&gt; wrote: >>>>>> Hi all, >>>>>> >>>>>> I have a gluster volume used as data storage domain which is >>>>>> replicated to a slave gluster volume (say, slavevol) using gluster's >>>>>> geo-replication feature. >>>>>> >>>>>> Now, in a new oVirt instance, I use the import storage domain to >>>>>> import the slave gluster volume. The "VM Import" tab correctly lists the VMs >>>>>> that were present in my original gluster volume. However the "Disks" tab is >>>>>> empty. >>>>>> >>>>>> GET >>>>>> https://new-ovitt/api/storagedomains/5e1a37cf-933d-424c-8e3d-eb9e40b690a7... >>>>>> --> >>>>>> <disks/> >>>>>> >>>>>> >>>>>> In the code GetUnregisteredDiskQuery - if volumesList.size() != 1 - >>>>>> the image is skipped with a comment that we can't deal with snapshots. >>>>>> >>>>>> How do I recover the disks/images in this case? >>>>>> >>>>>> >>>>>> Further info: >>>>>> >>>>>> /rhev/data-center/mnt/glusterSD/10.70.40.112:_slavevol >>>>>> ├── 5e1a37cf-933d-424c-8e3d-eb9e40b690a7 >>>>>> │ ├── dom_md >>>>>> │ │ ├── ids >>>>>> │ │ ├── inbox >>>>>> │ │ ├── leases >>>>>> │ │ ├── metadata >>>>>> │ │ └── outbox >>>>>> │ ├── images >>>>>> │ │ ├── 202efaa6-0d01-40f3-a541-10eee920d221 >>>>>> │ │ │ ├── eb701046-6ee1-4c9d-b097-e51a8fd283e1 >>>>>> │ │ │ ├── eb701046-6ee1-4c9d-b097-e51a8fd283e1.lease >>>>>> │ │ │ └── eb701046-6ee1-4c9d-b097-e51a8fd283e1.meta >>>>>> │ │ ├── c52e4e02-dc6c-4a77-a184-9fcab88106c2 >>>>>> │ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659 >>>>>> │ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659.lease >>>>>> │ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659.meta >>>>>> │ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2 >>>>>> │ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2.lease >>>>>> │ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2.meta >>>>>> │ │ │ ├── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa >>>>>> │ │ │ ├── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa.lease >>>>>> │ │ │ └── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa.meta >>>>>> │ │ ├── c75de5b7-aa88-48d7-ba1b-067181eac6ae >>>>>> │ │ │ ├── ff09e16a-e8a0-452b-b95c-e160e68d09a9 >>>>>> │ │ │ ├── ff09e16a-e8a0-452b-b95c-e160e68d09a9.lease >>>>>> │ │ │ └── ff09e16a-e8a0-452b-b95c-e160e68d09a9.meta >>>>>> │ │ ├── efa94a0d-c08e-4ad9-983b-4d1d76bca865 >>>>>> │ │ │ ├── 64e3913c-da91-447c-8b69-1cff1f34e4b7 >>>>>> │ │ │ ├── 64e3913c-da91-447c-8b69-1cff1f34e4b7.lease >>>>>> │ │ │ ├── 64e3913c-da91-447c-8b69-1cff1f34e4b7.meta >>>>>> │ │ │ ├── 8174e8b4-3605-4db3-86a1-cb62c3a079f4 >>>>>> │ │ │ ├── 8174e8b4-3605-4db3-86a1-cb62c3a079f4.lease >>>>>> │ │ │ ├── 8174e8b4-3605-4db3-86a1-cb62c3a079f4.meta >>>>>> │ │ │ ├── e79a8821-bb4a-436a-902d-3876f107dd99 >>>>>> │ │ │ ├── e79a8821-bb4a-436a-902d-3876f107dd99.lease >>>>>> │ │ │ └── e79a8821-bb4a-436a-902d-3876f107dd99.meta >>>>>> │ │ └── f5eacc6e-4f16-4aa5-99ad-53ac1cda75b7 >>>>>> │ │ ├── 476bbfe9-1805-4c43-bde6-e7de5f7bd75d >>>>>> │ │ ├── 476bbfe9-1805-4c43-bde6-e7de5f7bd75d.lease >>>>>> │ │ └── 476bbfe9-1805-4c43-bde6-e7de5f7bd75d.meta >>>>>> │ └── master >>>>>> │ ├── tasks >>>>>> │ └── vms >>>>>> └── __DIRECT_IO_TEST__ >>>>>> >>>>>> engine.log: >>>>>> 2016-04-26 06:37:57,715 INFO >>>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] >>>>>> (org.ovirt.thread.pool-6-thread-25) [5e6b7a53] FINISH, >>>>>> GetImageInfoVDSCommand, return: org.ov >>>>>> irt.engine.core.common.businessentities.storage.DiskImage@d4b3ac2f, >>>>>> log id: 7b693bad >>>>>> 2016-04-26 06:37:57,724 INFO >>>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.GetVolumesListVDSCommand] >>>>>> (org.ovirt.thread.pool-6-thread-25) [5e6b7a53] START, >>>>>> GetVolumesListVDSCommand( StoragePool >>>>>> DomainAndGroupIdBaseVDSCommandParameters:{runAsync='true', >>>>>> storagePoolId='ed338557-5995-4634-97e2-15454a9d8800', >>>>>> ignoreFailoverLimit='false', >>>>>> storageDomainId='5e1a37cf-933d-424c-8e3d-eb9e40b >>>>>> 690a7', imageGroupId='c52e4e02-dc6c-4a77-a184-9fcab88106c2'}), log id: >>>>>> 741b9214 >>>>>> 2016-04-26 06:37:58,748 INFO >>>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.GetVolumesListVDSCommand] >>>>>> (org.ovirt.thread.pool-6-thread-25) [5e6b7a53] FINISH, >>>>>> GetVolumesListVDSCommand, return: [9 >>>>>> 0f1e26a-00e9-4ea5-9e92-2e448b9b8bfa, >>>>>> 766a15b9-57db-417d-bfa0-beadbbb84ad2, 34e46104-8fad-4510-a5bf-0730b97a6659], >>>>>> log id: 741b9214 >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>>>