Re: [ovirt-users] Import storage domain - disks not listed

On 05/02/2016 09:36 PM, Maor Lipchuk wrote: > > On Mon, May 2, 2016 at 5:45 PM, Sahina Bose <sabose(a)redhat.com&gt; wrote: >> >> >> >> On 05/02/2016 05:57 PM, Maor Lipchuk wrote: >> >> >> >> On Mon, May 2, 2016 at 1:08 PM, Sahina Bose <sabose(a)redhat.com&gt; wrote: >>> >>> >>> >>> On 05/02/2016 03:15 PM, Maor Lipchuk wrote: >>> >>> >>> >>> On Mon, May 2, 2016 at 12:29 PM, Sahina Bose <sabose(a)redhat.com&gt; wrote: >>>> >>>> >>>> >>>> On 05/01/2016 05:33 AM, Maor Lipchuk wrote: >>>> >>>> Hi Sahina, >>>> >>>> The disks with snapshots should be part of the VMs, once you will >>>> register those VMs you should see those disks in the disks sub tab. >>>> >>>> >>>> Maor, >>>> >>>> I was unable to import VM which prompted question - I assumed we had to >>>> register disks first. So maybe I need to troubleshoot why I could not import >>>> VMs from the domain first. >>>> It fails with an error "Image does not exist". Where does it look for >>>> volume IDs to pass to GetImageInfoVDSCommand - the OVF disk? >>>> >>>> >>>> In engine.log >>>> >>>> 2016-05-02 04:15:14,812 ERROR >>>> [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] >>>> (ajp-/127.0.0.1:8702-1) [32f0b27c] Ir >>>> sBroker::getImageInfo::Failed getting image info >>>> imageId='6f4da17a-05a2-4d77-8091-d2fca3bbea1c' does not exist on >>>> domainName='sahinasl >>>> ave', domainId='5e1a37cf-933d-424c-8e3d-eb9e40b690a7', error code: >>>> 'VolumeDoesNotExist', message: Volume does not exist: (u'6f4da17a-0 >>>> 5a2-4d77-8091-d2fca3bbea1c',) >>>> 2016-05-02 04:15:14,814 WARN >>>> [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] >>>> (ajp-/127.0.0.1:8702-1) [32f0b27c] >>>> executeIrsBrokerCommand: getImageInfo on >>>> '6f4da17a-05a2-4d77-8091-d2fca3bbea1c' threw an exception - assuming image >>>> doesn't exist: IRS >>>> GenericException: IRSErrorException: VolumeDoesNotExist >>>> 2016-05-02 04:15:14,814 INFO >>>> [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] >>>> (ajp-/127.0.0.1:8702-1) [32f0b27c] >>>> FINISH, DoesImageExistVDSCommand, return: false, log id: 3366f39b >>>> 2016-05-02 04:15:14,814 WARN >>>> [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] >>>> (ajp-/127.0.0.1:8702-1) [32f0b27c] CanDoAction of action >>>> 'ImportVmFromConfiguration' failed for user admin@internal. Reasons: >>>> VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_IMAGE_DOES_NOT_EXIST >>>> >>>> >>>> >>>> jsonrpc.Executor/2::DEBUG::2016-05-02 >>>> 13:45:13,903::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling >>>> 'Volume.getInfo' in >>>> bridge with {u'imageID': u'c52e4e02-dc6c-4a77-a184-9fcab88106c2', >>>> u'storagepoolID': u'46ac4975-a84e-4e76-8e73-7971d0dadf0b', u'volumeI >>>> D': u'6f4da17a-05a2-4d77-8091-d2fca3bbea1c', u'storagedomainID': >>>> u'5e1a37cf-933d-424c-8e3d-eb9e40b690a7'} >>>> >>>> jsonrpc.Executor/2::DEBUG::2016-05-02 >>>> 13:45:13,910::fileVolume::535::Storage.Volume::(validateVolumePath) validate >>>> path for 6f4da17a-05a2-4d77-8091-d2fca3bbea1c >>>> jsonrpc.Executor/2::ERROR::2016-05-02 >>>> 13:45:13,914::task::866::Storage.TaskManager.Task::(_setError) >>>> Task=`94dba16f-f7eb-439e-95e2-a04b34b92f84`::Unexpected error >>>> Traceback (most recent call last): >>>> File "/usr/share/vdsm/storage/task.py", line 873, in _run >>>> return fn(*args, **kargs) >>>> File "/usr/share/vdsm/logUtils.py", line 49, in wrapper >>>> res = f(*args, **kwargs) >>>> File "/usr/share/vdsm/storage/hsm.py", line 3162, in getVolumeInfo >>>> volUUID=volUUID).getInfo() >>>> File "/usr/share/vdsm/storage/sd.py", line 457, in produceVolume >>>> volUUID) >>>> File "/usr/share/vdsm/storage/glusterVolume.py", line 16, in >>>> __init__ >>>> volUUID) >>>> File "/usr/share/vdsm/storage/fileVolume.py", line 58, in __init__ >>>> volume.Volume.__init__(self, repoPath, sdUUID, imgUUID, volUUID) >>>> File "/usr/share/vdsm/storage/volume.py", line 181, in __init__ >>>> self.validate() >>>> File "/usr/share/vdsm/storage/volume.py", line 194, in validate >>>> self.validateVolumePath() >>>> File "/usr/share/vdsm/storage/fileVolume.py", line 540, in >>>> validateVolumePath >>>> raise se.VolumeDoesNotExist(self.volUUID) >>>> VolumeDoesNotExist: Volume does not exist: >>>> (u'6f4da17a-05a2-4d77-8091-d2fca3bbea1c',) >>>> >>>> When I look at the tree output - there's no >>>> 6f4da17a-05a2-4d77-8091-d2fca3bbea1c file. >>>> >>>> >>>> ├── c52e4e02-dc6c-4a77-a184-9fcab88106c2 >>>> │ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659 >>>> │ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659.lease >>>> │ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659.meta >>>> │ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2 >>>> │ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2.lease >>>> │ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2.meta >>>> │ │ │ ├── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa >>>> │ │ │ ├── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa.lease >>>> │ │ │ └── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa.meta >>> >>> >>> >>> Usually the "image does not exists" message is prompted once the VM's >>> disk is managed in a different storage domain which were not imported yet. >>> >>> Few questions: >>> 1. Were there any other Storage Domain which are not present in the >>> setup? >>> >>> >>> In the original RHEV instance - there were 3 storage domain >>> i) Hosted engine storage domain: engine >>> ii) Master data domain: vmstore >>> iii) An export domain: expVol (no data here) >>> >>> To my backup RHEV server, I only imported vmstore. >> >> >> Just to be sure, can you look into the server which the engine storage >> domain resides on. >> Is there a chance that 6f4da17a-05a2-4d77-8091-d2fca3bbea1c could be >> there? >> >> Also do you have an old engine log, is there a chance to look for image >> 6f4da17a-05a2-4d77-8091-d2fca3bbea1c in there, to see when it was created >> and if there were any problems in the process. >> >> >> >> I checked the engine logs and it seems that image >> 6f4da17a-05a2-4d77-8091-d2fca3bbea1c was removed as part of snapshot merge. >> Could it be that the OVF was not updated? > > > > The OVF_STORE is being update every 60 minutes or when a Storage > Domain is moved to maintenance. > If the DR process was done before the OVF_STORE got updated and the > setup was destroyed then that might be the reason for the missing > image in the OVF. > > The problem is that this image id is already part of the images chain > configured in the VM's OVF. > As a work around maybe you can create a new volume with VdsClient > createVolume on the Host with the same UUID. > or maybe to merge all the disk's snapshots through the vdsClient and > once there will be only one volume you can register this disk as > floating disk. Is there a way via API to update OVF_STORE? This way before the DR process we can ensure that the OVF_STORE is referring to the correct images. > > >> >> engine.log-20160426.gz:2016-04-26 01:54:28,576 INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] >> (pool-5-thread-1) [6f94943f] START, MergeVDSCommand(HostName = rhsdev9, >> MergeVDSCommandParameters:{runAsync='true', >> hostId='b9a662bf-e05e-4e6a-9dfe-ec1be76d48e7', >> vmId='e2b89e45-6f99-465d-aa08-fc4f746f0dd0', >> storagePoolId='00000001-0001-0001-0001-000000000305', >> storageDomainId='5e1a37cf-933d-424c-8e3d-eb9e40b690a7', >> imageGroupId='c52e4e02-dc6c-4a77-a184-9fcab88106c2', >> imageId='6f4da17a-05a2-4d77-8091-d2fca3bbea1c', >> baseImageId='90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa', >> topImageId='6f4da17a-05a2-4d77-8091-d2fca3bbea1c', bandwidth='0'}), log id: >> 5a65a864 >> engine.log-20160426.gz:2016-04-26 01:55:03,629 INFO >> [org.ovirt.engine.core.bll.MergeCommandCallback] >> (DefaultQuartzScheduler_Worker-57) [6f94943f] Merge command has completed >> for images >> '90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa'..'6f4da17a-05a2-4d77-8091-d2fca3bbea1c' >> engine.log-20160426.gz:2016-04-26 01:55:06,707 INFO >> [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-5-thread-2) [3c72e9f8] >> Successfully removed volume(s): [6f4da17a-05a2-4d77-8091-d2fca3bbea1c] >> >> >> >>> >>> 2. Can you look for the image id 6f4da17a-05a2-4d77-8091-d2fca3bbea1c in >>> your storage server (Search on all the rest of the storage domains)? >>> >>> >>> No - there was no file with this id (checked both in the original RHEV >>> instance) >>> [root@dhcp42-105 mnt]# pwd >>> /rhev/data-center/mnt >>> [root@dhcp42-105 mnt]# find . -name >>> 6f4da17a-05a2-4d77-8091-d2fca3bbea1c >>> [root@dhcp42-105 mnt]# >>> >>> >>> Were there any operations being done on the VM before the recovery such >>> as remove disk, move disk, or a new creation of a disk? >>> >>> >>> No. The only operation done was creation of a snapshot - and it >>> completed before the recovery was attempted. >>> >>> >>> Regards, >>> Maor >>> >>>> >>>> Regarding floating disks (without snapshots), you can register them >>>> through REST. >>>> If you are working on the master branch there should be a sub tab >>>> dedicated for those also. >>>> >>>> Regards, >>>> Maor >>>> >>>> On Tue, Apr 26, 2016 at 1:44 PM, Sahina Bose <sabose(a)redhat.com&gt; wrote: >>>>> >>>>> Hi all, >>>>> >>>>> I have a gluster volume used as data storage domain which is >>>>> replicated to a slave gluster volume (say, slavevol) using gluster's >>>>> geo-replication feature. >>>>> >>>>> Now, in a new oVirt instance, I use the import storage domain to >>>>> import the slave gluster volume. The "VM Import" tab correctly lists the VMs >>>>> that were present in my original gluster volume. However the "Disks" tab is >>>>> empty. >>>>> >>>>> GET >>>>> https://new-ovitt/api/storagedomains/5e1a37cf-933d-424c-8e3d-eb9e40b690a7... >>>>> --> >>>>> <disks/> >>>>> >>>>> >>>>> In the code GetUnregisteredDiskQuery - if volumesList.size() != 1 - >>>>> the image is skipped with a comment that we can't deal with snapshots. >>>>> >>>>> How do I recover the disks/images in this case? >>>>> >>>>> >>>>> Further info: >>>>> >>>>> /rhev/data-center/mnt/glusterSD/10.70.40.112:_slavevol >>>>> ├── 5e1a37cf-933d-424c-8e3d-eb9e40b690a7 >>>>> │ ├── dom_md >>>>> │ │ ├── ids >>>>> │ │ ├── inbox >>>>> │ │ ├── leases >>>>> │ │ ├── metadata >>>>> │ │ └── outbox >>>>> │ ├── images >>>>> │ │ ├── 202efaa6-0d01-40f3-a541-10eee920d221 >>>>> │ │ │ ├── eb701046-6ee1-4c9d-b097-e51a8fd283e1 >>>>> │ │ │ ├── eb701046-6ee1-4c9d-b097-e51a8fd283e1.lease >>>>> │ │ │ └── eb701046-6ee1-4c9d-b097-e51a8fd283e1.meta >>>>> │ │ ├── c52e4e02-dc6c-4a77-a184-9fcab88106c2 >>>>> │ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659 >>>>> │ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659.lease >>>>> │ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659.meta >>>>> │ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2 >>>>> │ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2.lease >>>>> │ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2.meta >>>>> │ │ │ ├── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa >>>>> │ │ │ ├── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa.lease >>>>> │ │ │ └── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa.meta >>>>> │ │ ├── c75de5b7-aa88-48d7-ba1b-067181eac6ae >>>>> │ │ │ ├── ff09e16a-e8a0-452b-b95c-e160e68d09a9 >>>>> │ │ │ ├── ff09e16a-e8a0-452b-b95c-e160e68d09a9.lease >>>>> │ │ │ └── ff09e16a-e8a0-452b-b95c-e160e68d09a9.meta >>>>> │ │ ├── efa94a0d-c08e-4ad9-983b-4d1d76bca865 >>>>> │ │ │ ├── 64e3913c-da91-447c-8b69-1cff1f34e4b7 >>>>> │ │ │ ├── 64e3913c-da91-447c-8b69-1cff1f34e4b7.lease >>>>> │ │ │ ├── 64e3913c-da91-447c-8b69-1cff1f34e4b7.meta >>>>> │ │ │ ├── 8174e8b4-3605-4db3-86a1-cb62c3a079f4 >>>>> │ │ │ ├── 8174e8b4-3605-4db3-86a1-cb62c3a079f4.lease >>>>> │ │ │ ├── 8174e8b4-3605-4db3-86a1-cb62c3a079f4.meta >>>>> │ │ │ ├── e79a8821-bb4a-436a-902d-3876f107dd99 >>>>> │ │ │ ├── e79a8821-bb4a-436a-902d-3876f107dd99.lease >>>>> │ │ │ └── e79a8821-bb4a-436a-902d-3876f107dd99.meta >>>>> │ │ └── f5eacc6e-4f16-4aa5-99ad-53ac1cda75b7 >>>>> │ │ ├── 476bbfe9-1805-4c43-bde6-e7de5f7bd75d >>>>> │ │ ├── 476bbfe9-1805-4c43-bde6-e7de5f7bd75d.lease >>>>> │ │ └── 476bbfe9-1805-4c43-bde6-e7de5f7bd75d.meta >>>>> │ └── master >>>>> │ ├── tasks >>>>> │ └── vms >>>>> └── __DIRECT_IO_TEST__ >>>>> >>>>> engine.log: >>>>> 2016-04-26 06:37:57,715 INFO >>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] >>>>> (org.ovirt.thread.pool-6-thread-25) [5e6b7a53] FINISH, >>>>> GetImageInfoVDSCommand, return: org.ov >>>>> irt.engine.core.common.businessentities.storage.DiskImage@d4b3ac2f, >>>>> log id: 7b693bad >>>>> 2016-04-26 06:37:57,724 INFO >>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.GetVolumesListVDSCommand] >>>>> (org.ovirt.thread.pool-6-thread-25) [5e6b7a53] START, >>>>> GetVolumesListVDSCommand( StoragePool >>>>> DomainAndGroupIdBaseVDSCommandParameters:{runAsync='true', >>>>> storagePoolId='ed338557-5995-4634-97e2-15454a9d8800', >>>>> ignoreFailoverLimit='false', >>>>> storageDomainId='5e1a37cf-933d-424c-8e3d-eb9e40b >>>>> 690a7', imageGroupId='c52e4e02-dc6c-4a77-a184-9fcab88106c2'}), log id: >>>>> 741b9214 >>>>> 2016-04-26 06:37:58,748 INFO >>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.GetVolumesListVDSCommand] >>>>> (org.ovirt.thread.pool-6-thread-25) [5e6b7a53] FINISH, >>>>> GetVolumesListVDSCommand, return: [9 >>>>> 0f1e26a-00e9-4ea5-9e92-2e448b9b8bfa, >>>>> 766a15b9-57db-417d-bfa0-beadbbb84ad2, 34e46104-8fad-4510-a5bf-0730b97a6659], >>>>> log id: 741b9214 >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>>> >>>> >>> >>