5:46 a.m.
Hi,
We have plenty of space, I have checked with both df -h and df -i.
Here are the vdsm logs from one node.
2021-03-08 12:09:15,550+0100 INFO (jsonrpc/7) [vdsm.api] START
deleteImage(sdUUID=u'fb14e013-15f1-49b6-b129-210517aca0da',
spUUID=u'aec485fa-cdbb-4979-8dc9-a2376559b2a4',
imgUUID=u'459872a1-8783-463a-98ae-9e56d8883abb', postZero=u'false',
force=u'false', discard=False) from=::ffff:172.16.122.253,44650, flow_id=39b2eddb,
task_id=969cdd60-9b4c-43d9-960a-2a3b467b227c (api:48)
2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] Unlink volume
symlink
u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images/01af322e-ff54-4981-9842-0f0a608da17c/46bf0806-8d16-40db-92df-8436f43628a4'
(blockSD:722)
2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] Unlink volume
symlink
u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images/01af322e-ff54-4981-9842-0f0a608da17c/27638d78-e935-44f4-b0cc-5ee2459e204a'
(blockSD:722)
2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] removed:
[u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images/01af322e-ff54-4981-9842-0f0a608da17c/27638d78-e935-44f4-b0cc-5ee2459e204a']
(blockSD:729)
2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] Removing
image directory
u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images/01af322e-ff54-4981-9842-0f0a608da17c'
(blockSD:735)
2021-03-08 12:09:15,729+0100 INFO (tasks/9) [storage.ThreadPool.WorkerThread] FINISH task
9a57dcf5-85b2-4c7c-8ffb-f11eecc2313f (threadPool:210)
2021-03-08 12:09:16,671+0100 WARN (jsonrpc/7) [storage.LVM] Removing stale lv:
fb14e013-15f1-49b6-b129-210517aca0da/46bf0806-8d16-40db-92df-8436f43628a4 (lvm:564)
2021-03-08 12:09:17,920+0100 WARN (tasks/7) [storage.LVM] lv:
46bf0806-8d16-40db-92df-8436f43628a4 not found in lvs vg:
fb14e013-15f1-49b6-b129-210517aca0da response (lvm:736)
2021-03-08 12:09:17,921+0100 ERROR (tasks/7) [storage.TaskManager.Task]
(Task='d4aa81a1-9adc-45c9-be46-05bc63862f4e') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in
run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in
wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1966, in
deleteVolume
vol = dom.produceVolume(imgUUID, volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 913, in
produceVolume
volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 814, in
__init__
self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line 62,
in __init__
volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86, in
__init__
self.validate()
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line 151,
in validate
raise se.VolumeDoesNotExist(self.volUUID)
VolumeDoesNotExist: Volume does not exist:
(u'46bf0806-8d16-40db-92df-8436f43628a4',)
2021-03-08 12:09:17,946+0100 INFO (tasks/7) [storage.ThreadPool.WorkerThread] FINISH task
d4aa81a1-9adc-45c9-be46-05bc63862f4e (threadPool:210)
2021-03-08 12:09:18,273+0100 WARN (jsonrpc/7) [storage.LVM] Command
['/usr/sbin/lvm', 'lvchange', '--config', 'devices {
preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/2181ad2edfa7b485c6c9ce90045609718$|^/dev/mapper/2763fdda70ca458906c9ce90045609718$|^/dev/mapper/28ba7d1d12c7099ef6c9ce90045609718$|^/dev/mapper/2e9d013564898d36f6c9ce90045609718$|",
"r|.*|"] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--autobackup',
'n', '-a', 'y', '--deltag',
u'IU_459872a1-8783-463a-98ae-9e56d8883abb', '--addtag',
u'IU__remove_me_459872a1-8783-463a-98ae-9e56d8883abb',
'fb14e013-15f1-49b6-b129-210517aca0da/5d51973d-4198-4cec-994a-727317640ccb']
succeeded with warnings: [' WARNING: Combining activation change with other commands
is not advised.', ' Scan of VG fb14e013-15f1-49b6-b129-210517aca0da from
/dev/mapper/2763fdda70ca
458906c9ce90045609718 found metadata seqno 26448 vs previous 26447.', ' Scan of
VG fb14e013-15f1-49b6-b129-210517aca0da from /dev/mapper/2763fdda70ca458906c9ce90045609718
found metadata seqno 26448 vs previous 26447.'] (lvm:385)
2021-03-08 12:09:18,281+0100 INFO (jsonrpc/7) [vdsm.api] FINISH deleteImage return=None
from=::ffff:172.16.122.253,44650, flow_id=39b2eddb,
task_id=969cdd60-9b4c-43d9-960a-2a3b467b227c (api:54)
2021-03-08 12:09:18,293+0100 INFO (tasks/9) [storage.ThreadPool.WorkerThread] START task
969cdd60-9b4c-43d9-960a-2a3b467b227c (cmd=<bound method Task.commit of
<vdsm.storage.task.Task instance at 0x7f7b606c27e8>>, args=None)
(threadPool:208)
2021-03-08 12:09:18,295+0100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call
Image.delete succeeded in 2.75 seconds (__init__:312)
2021-03-08 12:09:18,299+0100 INFO (jsonrpc/0) [vdsm.api] START
getAllTasksStatuses(spUUID=None, options=None) from=::ffff:172.16.122.253,44628,
task_id=f6411446-8bed-4766-a038-892ebcaab65e (api:48)
We have a SAN storage, with plenty of space, plus some NFS storage. But, again, there is
plenty of space on all of those. From what I could find, the sanlock issue is not
specifically related to space on the storage devices.