Hi Nir,
I identified the reason for the failing OVF updates on the initial VG – metadata was affected by blkdiscard tests in scope of
https://bugzilla.redhat.com/show_bug.cgi?id=1562369
However, the OVF updates are failing on other installations as well (on 2 out of 40 storage domains). Here is the output of your commands:
# lvs -o vg_name,lv_name,tags | grep 3ad1987a-8b7d-426d-9d51-4a78cb0a888f
3ad1987a-8b7d-426d-9d51-4a78cb0a888f 212d644c-0155-4999-9df9-72bacfc7f141 IU_0ebefe5e-9053-4bf1-bdfd-fdb26579c179,MD_4,PU_00000000-0000-0000-0000-000000000000
3ad1987a-8b7d-426d-9d51-4a78cb0a888f 94f519de-bc19-4557-82c4-86bbcfc5dd2f IU_60d9eec7-951f-4594-ae99-7d31318ee3a9,MD_5,PU_00000000-0000-0000-0000-000000000000
3ad1987a-8b7d-426d-9d51-4a78cb0a888f ids
3ad1987a-8b7d-426d-9d51-4a78cb0a888f inbox
3ad1987a-8b7d-426d-9d51-4a78cb0a888f leases
3ad1987a-8b7d-426d-9d51-4a78cb0a888f master
3ad1987a-8b7d-426d-9d51-4a78cb0a888f metadata
3ad1987a-8b7d-426d-9d51-4a78cb0a888f outbox
3ad1987a-8b7d-426d-9d51-4a78cb0a888f xleases
# for i in 4 5; do
dd if=/dev/3ad1987a-8b7d-426d-9d51-4a78cb0a888f/metadata bs=512 count=1 skip=$i of=metadata.$i
done
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00121297 s, 422 kB/s
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.000735026 s, 697 kB/s
# file metadata.*
metadata.4: data
metadata.5: ASCII text
# cat metadata.5
DOMAIN=3ad1987a-8b7d-426d-9d51-4a78cb0a888f
CTIME=1520597691
FORMAT=RAW
DISKTYPE=OVFS
LEGALITY=LEGAL
SIZE=262144
VOLTYPE=LEAF
DESCRIPTION={"Updated":true,"Size":4669440,"Last Updated":"Fri Jun 08 09:51:18 CEST 2018","Storage Domains":[{"uuid":"3ad1987a-8b7d-426d-9d51-4a78cb0a888f"}],"Disk Description":"OVF_STORE"}
IMAGE=60d9eec7-951f-4594-ae99-7d31318ee3a9
PUUID=00000000-0000-0000-0000-000000000000
MTIME=0
POOL_UUID=
TYPE=PREALLOCATED
GEN=0
EOF
# od -c metadata.4
0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
0001000
vdsm.log from manual OVF update test:
2018-06-20 09:28:27,840+0200 INFO (jsonrpc/7) [vdsm.api] START setVolumeDescription(sdUUID=u'3ad1987a-8b7d-426d-9d51-4a78cb0a888f', spUUID=u'5849b030-626e-47cb-ad90-3ce782d831b3',
imgUUID=u'0ebefe5e-9053-4bf1-bdfd-fdb26579c179', volUUID=u'212d644c-0155-4999-9df9-72bacfc7f141', description=u'{"Updated":false,"Last Updated":"Fri Jun 08 09:51:18 CEST 2018","Storage Domains":[{"uuid":"3ad1987a-8b7d-426d-9d51-4a78cb0a888f"}],"Disk Description":"OVF_STORE"}',
options=None) from=::ffff:<IP>,51790, flow_id=7e4edb74, task_id=5f1fda67-a073-419a-bba5-9bf680c0e5d5 (api:46)
2018-06-20 09:28:28,072+0200 WARN (jsonrpc/7) [storage.ResourceManager] Resource factory failed to create resource '01_img_3ad1987a-8b7d-426d-9d51-4a78cb0a888f.0ebefe5e-9053-4bf1-bdfd-fdb26579c179'.
Canceling request. (resourceManager:543)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceManager.py", line 539, in registerResource
obj = namespaceObj.factory.createResource(name, lockType)
File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceFactories.py", line 193, in createResource
lockType)
File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceFactories.py", line 122, in __getResourceCandidatesList
imgUUID=resourceName)
File "/usr/lib/python2.7/site-packages/vdsm/storage/image.py", line 206, in getChain
if len(uuidlist) == 1 and srcVol.isShared():
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 1434, in isShared
return self._manifest.isShared()
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 141, in isShared
return self.getVolType() == sc.type2name(sc.SHARED_VOL)
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 134, in getVolType
self.voltype = self.getMetaParam(sc.VOLTYPE)
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 118, in getMetaParam
meta = self.getMetadata()
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line 112, in getMetadata
md = VolumeMetadata.from_lines(lines)
File "/usr/lib/python2.7/site-packages/vdsm/storage/volumemetadata.py", line 103, in from_lines
"Missing metadata key: %s: found: %s" % (e, md))
MetaDataKeyNotFoundError: Meta Data key not found error: ("Missing metadata key: 'DOMAIN': found: {}",)
2018-06-20 09:28:28,072+0200 WARN (jsonrpc/7) [storage.ResourceManager.Request] (ResName='01_img_3ad1987a-8b7d-426d-9d51-4a78cb0a888f.0ebefe5e-9053-4bf1-bdfd-fdb26579c179',
ReqID='10c95223-f349-4ac3-ab2f-7a5f3d1c7749') Tried to cancel a processed request (resourceManager:187)
2018-06-20 09:28:28,073+0200 INFO (jsonrpc/7) [vdsm.api] FINISH setVolumeDescription error=Could not acquire resource. Probably resource factory threw an exception.:
() from=::ffff:<IP>,51790, flow_id=7e4edb74, task_id=5f1fda67-a073-419a-bba5-9bf680c0e5d5 (api:50)
2018-06-20 09:28:28,073+0200 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='5f1fda67-a073-419a-bba5-9bf680c0e5d5') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run
return fn(*args, **kargs)
File "<string>", line 2, in setVolumeDescription
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1294, in setVolumeDescription
pool.setVolumeDescription(sdUUID, imgUUID, volUUID, description)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 2011, in setVolumeDescription
with rm.acquireResource(img_ns, imgUUID, rm.EXCLUSIVE):
File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceManager.py", line 1025, in acquireResource
return _manager.acquireResource(namespace, name, lockType, timeout=timeout)
File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceManager.py", line 475, in acquireResource
raise se.ResourceAcqusitionFailed()
ResourceAcqusitionFailed: Could not acquire resource. Probably resource factory threw an exception.: ()
2018-06-20 09:28:28,073+0200 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='5f1fda67-a073-419a-bba5-9bf680c0e5d5') aborting: Task is aborted: u'Could not acquire
resource. Probably resource factory threw an exception.: ()' - code 100 (task:1181)
2018-06-20 09:28:28,073+0200 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH setVolumeDescription error=Could not acquire resource. Probably resource factory threw an exception.:
() (dispatcher:82)
I worked around the SPM problem by running engine-setup (neither vds-client Host/StoragePool *Task*, async_task_entries or taskcleaner.sh showed any tasks).
All the best,
Oliver
Von: Nir Soffer <nsoffer@redhat.com>
Gesendet: Dienstag, 19. Juni 2018 23:39
An: Elad Ben Aharon <ebenahar@redhat.com>
Cc: users@ovirt.org; Albl, Oliver <Oliver.Albl@fabasoft.com>
Betreff: [ovirt-users] Re: Failed to update VMs/Templates OVF data, cannot change SPM
On Wed, Jun 20, 2018 at 12:11 AM Elad Ben Aharon <ebenahar@redhat.com> wrote: