On Fri, May 6, 2016 at 3:08 PM, Sahina Bose <sabose@redhat.com> wrote:

Hi,

Back to Import Storage domain questions -

To ensure consistency, we snapshot the running VMs prior to replicating the gluster volume to a central site. The VMs have been setup such that OS disks are on a gluster volume called "vmstore" and non-OS disks are on a gluster volume called "data"
For back up , we are only interested in "data" - so only this storage domain is imported at backup recovery site.

Since the "data" storage domain does not contain all disks - VMs cannot be imported - that's expected.

To retrieve the backed up disks - since the disk has snapshot - this does not seem possible either. How do we work around this?

Since the engine can't support floating disks with snapshot you can try to merge those snapshots (by using qemu-img merge and qemu-img rebase.) until there will be only one volume left.

Once there will be only one volume oVirt can import the disk.

Another workaround might be that if you still have the setup which could not import the VM because of missing volume, then, you can try manipulate the OVF in the Data Base.

The VMs' and Template's XML representation are contained in the "unregistered_ovf_of_entities" table.

You can fetch the VM's OVF from that table:

SELECT ovf_data FROM unregistered_ovf_of_entities where entity_name ilike '%(the VM name (You can also fetch by id with entity_id))%';

Update the OVF with the correlated volume ids in the storage server. After updating this, the import of the VM should work

thanks!

On 05/03/2016 03:21 PM, Maor Lipchuk wrote:

There is this bug https://bugzilla.redhat.com/1270562 which forces
OVF_STORE update but it is not yet implemented.

On Tue, May 3, 2016 at 8:52 AM, Sahina Bose <sabose@redhat.com> wrote:

On 05/02/2016 09:36 PM, Maor Lipchuk wrote:

On Mon, May 2, 2016 at 5:45 PM, Sahina Bose <sabose@redhat.com> wrote:

On 05/02/2016 05:57 PM, Maor Lipchuk wrote:

On Mon, May 2, 2016 at 1:08 PM, Sahina Bose <sabose@redhat.com> wrote:

On 05/02/2016 03:15 PM, Maor Lipchuk wrote:

On Mon, May 2, 2016 at 12:29 PM, Sahina Bose <sabose@redhat.com> wrote:

On 05/01/2016 05:33 AM, Maor Lipchuk wrote:

Hi Sahina,

The disks with snapshots should be part of the VMs, once you will
register those VMs you should see those disks in the disks sub tab.

Maor,

I was unable to import VM which prompted question - I assumed we had to
register disks first. So maybe I need to troubleshoot why I could not import
VMs from the domain first.
It fails with an error "Image does not exist". Where does it look for
volume IDs to pass to GetImageInfoVDSCommand - the OVF disk?

In engine.log

2016-05-02 04:15:14,812 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand]
(ajp-/127.0.0.1:8702-1) [32f0b27c] Ir
sBroker::getImageInfo::Failed getting image info
imageId='6f4da17a-05a2-4d77-8091-d2fca3bbea1c' does not exist on
domainName='sahinasl
ave', domainId='5e1a37cf-933d-424c-8e3d-eb9e40b690a7', error code:
'VolumeDoesNotExist', message: Volume does not exist: (u'6f4da17a-0
5a2-4d77-8091-d2fca3bbea1c',)
2016-05-02 04:15:14,814 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand]
(ajp-/127.0.0.1:8702-1) [32f0b27c]
executeIrsBrokerCommand: getImageInfo on
'6f4da17a-05a2-4d77-8091-d2fca3bbea1c' threw an exception - assuming image
doesn't exist: IRS
GenericException: IRSErrorException: VolumeDoesNotExist
2016-05-02 04:15:14,814 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand]
(ajp-/127.0.0.1:8702-1) [32f0b27c]
FINISH, DoesImageExistVDSCommand, return: false, log id: 3366f39b
2016-05-02 04:15:14,814 WARN
[org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand]
(ajp-/127.0.0.1:8702-1) [32f0b27c] CanDoAction of action
'ImportVmFromConfiguration' failed for user admin@internal. Reasons:
VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_IMAGE_DOES_NOT_EXIST

jsonrpc.Executor/2::DEBUG::2016-05-02
13:45:13,903::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling
'Volume.getInfo' in
bridge with {u'imageID': u'c52e4e02-dc6c-4a77-a184-9fcab88106c2',
u'storagepoolID': u'46ac4975-a84e-4e76-8e73-7971d0dadf0b', u'volumeI
D': u'6f4da17a-05a2-4d77-8091-d2fca3bbea1c', u'storagedomainID':
u'5e1a37cf-933d-424c-8e3d-eb9e40b690a7'}

jsonrpc.Executor/2::DEBUG::2016-05-02
13:45:13,910::fileVolume::535::Storage.Volume::(validateVolumePath) validate
path for 6f4da17a-05a2-4d77-8091-d2fca3bbea1c
jsonrpc.Executor/2::ERROR::2016-05-02
13:45:13,914::task::866::Storage.TaskManager.Task::(_setError)
Task=`94dba16f-f7eb-439e-95e2-a04b34b92f84`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 49, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 3162, in getVolumeInfo
volUUID=volUUID).getInfo()
File "/usr/share/vdsm/storage/sd.py", line 457, in produceVolume
volUUID)
File "/usr/share/vdsm/storage/glusterVolume.py", line 16, in
__init__
volUUID)
File "/usr/share/vdsm/storage/fileVolume.py", line 58, in __init__
volume.Volume.__init__(self, repoPath, sdUUID, imgUUID, volUUID)
File "/usr/share/vdsm/storage/volume.py", line 181, in __init__
self.validate()
File "/usr/share/vdsm/storage/volume.py", line 194, in validate
self.validateVolumePath()
File "/usr/share/vdsm/storage/fileVolume.py", line 540, in
validateVolumePath
raise se.VolumeDoesNotExist(self.volUUID)
VolumeDoesNotExist: Volume does not exist:
(u'6f4da17a-05a2-4d77-8091-d2fca3bbea1c',)

When I look at the tree output - there's no
6f4da17a-05a2-4d77-8091-d2fca3bbea1c file.

├── c52e4e02-dc6c-4a77-a184-9fcab88106c2
│ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659
│ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659.lease
│ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659.meta
│ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2
│ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2.lease
│ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2.meta
│ │ │ ├── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa
│ │ │ ├── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa.lease
│ │ │ └── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa.meta

Usually the "image does not exists" message is prompted once the VM's
disk is managed in a different storage domain which were not imported yet.

Few questions:
1. Were there any other Storage Domain which are not present in the
setup?

In the original RHEV instance - there were 3 storage domain
i) Hosted engine storage domain: engine
ii) Master data domain: vmstore
iii) An export domain: expVol (no data here)

To my backup RHEV server, I only imported vmstore.

Just to be sure, can you look into the server which the engine storage
domain resides on.
Is there a chance that 6f4da17a-05a2-4d77-8091-d2fca3bbea1c could be
there?

Also do you have an old engine log, is there a chance to look for image
6f4da17a-05a2-4d77-8091-d2fca3bbea1c in there, to see when it was created
and if there were any problems in the process.

I checked the engine logs and it seems that image
6f4da17a-05a2-4d77-8091-d2fca3bbea1c was removed as part of snapshot merge.
Could it be that the OVF was not updated?

The OVF_STORE is being update every 60 minutes or when a Storage
Domain is moved to maintenance.
If the DR process was done before the OVF_STORE got updated and the
setup was destroyed then that might be the reason for the missing
image in the OVF.

The problem is that this image id is already part of the images chain
configured in the VM's OVF.
As a work around maybe you can create a new volume with VdsClient
createVolume on the Host with the same UUID.
or maybe to merge all the disk's snapshots through the vdsClient and
once there will be only one volume you can register this disk as
floating disk.

Is there a way via API to update OVF_STORE?
This way before the DR process we can ensure that the OVF_STORE is referring
to the correct images.

engine.log-20160426.gz:2016-04-26 01:54:28,576 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand]
(pool-5-thread-1) [6f94943f] START, MergeVDSCommand(HostName = rhsdev9,
MergeVDSCommandParameters:{runAsync='true',
hostId='b9a662bf-e05e-4e6a-9dfe-ec1be76d48e7',
vmId='e2b89e45-6f99-465d-aa08-fc4f746f0dd0',
storagePoolId='00000001-0001-0001-0001-000000000305',
storageDomainId='5e1a37cf-933d-424c-8e3d-eb9e40b690a7',
imageGroupId='c52e4e02-dc6c-4a77-a184-9fcab88106c2',
imageId='6f4da17a-05a2-4d77-8091-d2fca3bbea1c',
baseImageId='90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa',
topImageId='6f4da17a-05a2-4d77-8091-d2fca3bbea1c', bandwidth='0'}), log id:
5a65a864
engine.log-20160426.gz:2016-04-26 01:55:03,629 INFO
[org.ovirt.engine.core.bll.MergeCommandCallback]
(DefaultQuartzScheduler_Worker-57) [6f94943f] Merge command has completed
for images
'90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa'..'6f4da17a-05a2-4d77-8091-d2fca3bbea1c'
engine.log-20160426.gz:2016-04-26 01:55:06,707 INFO
[org.ovirt.engine.core.bll.MergeStatusCommand] (pool-5-thread-2) [3c72e9f8]
Successfully removed volume(s): [6f4da17a-05a2-4d77-8091-d2fca3bbea1c]

2. Can you look for the image id 6f4da17a-05a2-4d77-8091-d2fca3bbea1c in
your storage server (Search on all the rest of the storage domains)?

No - there was no file with this id (checked both in the original RHEV
instance)
[root@dhcp42-105 mnt]# pwd
/rhev/data-center/mnt
[root@dhcp42-105 mnt]# find . -name
6f4da17a-05a2-4d77-8091-d2fca3bbea1c
[root@dhcp42-105 mnt]#

Were there any operations being done on the VM before the recovery such
as remove disk, move disk, or a new creation of a disk?

No. The only operation done was creation of a snapshot - and it
completed before the recovery was attempted.

Regards,
Maor

Regarding floating disks (without snapshots), you can register them
through REST.
If you are working on the master branch there should be a sub tab
dedicated for those also.

Regards,
Maor

On Tue, Apr 26, 2016 at 1:44 PM, Sahina Bose <sabose@redhat.com> wrote:

Hi all,

I have a gluster volume used as data storage domain which is
replicated to a slave gluster volume (say, slavevol) using gluster's
geo-replication feature.

Now, in a new oVirt instance, I use the import storage domain to
import the slave gluster volume. The "VM Import" tab correctly lists the VMs
that were present in my original gluster volume. However the "Disks" tab is
empty.

GET
https://new-ovitt/api/storagedomains/5e1a37cf-933d-424c-8e3d-eb9e40b690a7/disks;unregistered
-->
<disks/>

In the code GetUnregisteredDiskQuery - if volumesList.size() != 1 -
the image is skipped with a comment that we can't deal with snapshots.

How do I recover the disks/images in this case?

Further info:

/rhev/data-center/mnt/glusterSD/10.70.40.112:_slavevol
├── 5e1a37cf-933d-424c-8e3d-eb9e40b690a7
│ ├── dom_md
│ │ ├── ids
│ │ ├── inbox
│ │ ├── leases
│ │ ├── metadata
│ │ └── outbox
│ ├── images
│ │ ├── 202efaa6-0d01-40f3-a541-10eee920d221
│ │ │ ├── eb701046-6ee1-4c9d-b097-e51a8fd283e1
│ │ │ ├── eb701046-6ee1-4c9d-b097-e51a8fd283e1.lease
│ │ │ └── eb701046-6ee1-4c9d-b097-e51a8fd283e1.meta
│ │ ├── c52e4e02-dc6c-4a77-a184-9fcab88106c2
│ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659
│ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659.lease
│ │ │ ├── 34e46104-8fad-4510-a5bf-0730b97a6659.meta
│ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2
│ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2.lease
│ │ │ ├── 766a15b9-57db-417d-bfa0-beadbbb84ad2.meta
│ │ │ ├── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa
│ │ │ ├── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa.lease
│ │ │ └── 90f1e26a-00e9-4ea5-9e92-2e448b9b8bfa.meta
│ │ ├── c75de5b7-aa88-48d7-ba1b-067181eac6ae
│ │ │ ├── ff09e16a-e8a0-452b-b95c-e160e68d09a9
│ │ │ ├── ff09e16a-e8a0-452b-b95c-e160e68d09a9.lease
│ │ │ └── ff09e16a-e8a0-452b-b95c-e160e68d09a9.meta
│ │ ├── efa94a0d-c08e-4ad9-983b-4d1d76bca865
│ │ │ ├── 64e3913c-da91-447c-8b69-1cff1f34e4b7
│ │ │ ├── 64e3913c-da91-447c-8b69-1cff1f34e4b7.lease
│ │ │ ├── 64e3913c-da91-447c-8b69-1cff1f34e4b7.meta
│ │ │ ├── 8174e8b4-3605-4db3-86a1-cb62c3a079f4
│ │ │ ├── 8174e8b4-3605-4db3-86a1-cb62c3a079f4.lease
│ │ │ ├── 8174e8b4-3605-4db3-86a1-cb62c3a079f4.meta
│ │ │ ├── e79a8821-bb4a-436a-902d-3876f107dd99
│ │ │ ├── e79a8821-bb4a-436a-902d-3876f107dd99.lease
│ │ │ └── e79a8821-bb4a-436a-902d-3876f107dd99.meta
│ │ └── f5eacc6e-4f16-4aa5-99ad-53ac1cda75b7
│ │ ├── 476bbfe9-1805-4c43-bde6-e7de5f7bd75d
│ │ ├── 476bbfe9-1805-4c43-bde6-e7de5f7bd75d.lease
│ │ └── 476bbfe9-1805-4c43-bde6-e7de5f7bd75d.meta
│ └── master
│ ├── tasks
│ └── vms
└── __DIRECT_IO_TEST__

engine.log:
2016-04-26 06:37:57,715 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand]
(org.ovirt.thread.pool-6-thread-25) [5e6b7a53] FINISH,
GetImageInfoVDSCommand, return: org.ov
irt.engine.core.common.businessentities.storage.DiskImage@d4b3ac2f,
log id: 7b693bad
2016-04-26 06:37:57,724 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.GetVolumesListVDSCommand]
(org.ovirt.thread.pool-6-thread-25) [5e6b7a53] START,
GetVolumesListVDSCommand( StoragePool
DomainAndGroupIdBaseVDSCommandParameters:{runAsync='true',
storagePoolId='ed338557-5995-4634-97e2-15454a9d8800',
ignoreFailoverLimit='false',
storageDomainId='5e1a37cf-933d-424c-8e3d-eb9e40b
690a7', imageGroupId='c52e4e02-dc6c-4a77-a184-9fcab88106c2'}), log id:
741b9214
2016-04-26 06:37:58,748 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.GetVolumesListVDSCommand]
(org.ovirt.thread.pool-6-thread-25) [5e6b7a53] FINISH,
GetVolumesListVDSCommand, return: [9
0f1e26a-00e9-4ea5-9e92-2e448b9b8bfa,
766a15b9-57db-417d-bfa0-beadbbb84ad2, 34e46104-8fad-4510-a5bf-0730b97a6659],
log id: 741b9214

_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users