On Monday, 8 March 2021 12:46:58 CET kim.kargaard@noroff.no wrote:
Hi,
We have plenty of space, I have checked with both df -h and df -i.
Here are the vdsm logs from one node.
what is the whole flow you are running? This looks like a flow where you try to delete an image, which is probably missing one of its volumes. In the log bellow there isn't anything related to sanlock. Or maybe wrong log?
2021-03-08 12:09:15,550+0100 INFO (jsonrpc/7) [vdsm.api] START deleteImage(sdUUID=u'fb14e013-15f1-49b6-b129-210517aca0da', spUUID=u'aec485fa-cdbb-4979-8dc9-a2376559b2a4', imgUUID=u'459872a1-8783-463a-98ae-9e56d8883abb', postZero=u'false', force=u'false', discard=False) from=::ffff:172.16.122.253,44650, flow_id=39b2eddb, task_id=969cdd60-9b4c-43d9-960a-2a3b467b227c (api:48) 2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] Unlink volume symlink u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images /01af322e-ff54-4981-9842-0f0a608da17c/46bf0806-8d16-40db-92df-8436f43628a4' (blockSD:722) 2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] Unlink volume symlink u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images /01af322e-ff54-4981-9842-0f0a608da17c/27638d78-e935-44f4-b0cc-5ee2459e204a' (blockSD:722) 2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] removed: [u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/image s/01af322e-ff54-4981-9842-0f0a608da17c/27638d78-e935-44f4-b0cc-5ee2459e204a' ] (blockSD:729) 2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] Removing image directory u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images /01af322e-ff54-4981-9842-0f0a608da17c' (blockSD:735) 2021-03-08 12:09:15,729+0100 INFO (tasks/9) [storage.ThreadPool.WorkerThread] FINISH task 9a57dcf5-85b2-4c7c-8ffb-f11eecc2313f (threadPool:210) 2021-03-08 12:09:16,671+0100 WARN (jsonrpc/7) [storage.LVM] Removing stale lv: fb14e013-15f1-49b6-b129-210517aca0da/46bf0806-8d16-40db-92df-8436f43628a4 (lvm:564) 2021-03-08 12:09:17,920+0100 WARN (tasks/7) [storage.LVM] lv: 46bf0806-8d16-40db-92df-8436f43628a4 not found in lvs vg: fb14e013-15f1-49b6-b129-210517aca0da response (lvm:736) 2021-03-08 12:09:17,921+0100 ERROR (tasks/7) [storage.TaskManager.Task] (Task='d4aa81a1-9adc-45c9-be46-05bc63862f4e') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1966, in deleteVolume vol = dom.produceVolume(imgUUID, volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 913, in produceVolume volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 814, in __init__ self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line 62, in __init__ volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86, in __init__ self.validate() File "/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line 151, in validate raise se.VolumeDoesNotExist(self.volUUID) VolumeDoesNotExist: Volume does not exist: (u'46bf0806-8d16-40db-92df-8436f43628a4',) 2021-03-08 12:09:17,946+0100 INFO (tasks/7) [storage.ThreadPool.WorkerThread] FINISH task d4aa81a1-9adc-45c9-be46-05bc63862f4e (threadPool:210) 2021-03-08 12:09:18,273+0100 WARN (jsonrpc/7) [storage.LVM] Command ['/usr/sbin/lvm', 'lvchange', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/2181ad2edfa7b485c6c9ce90045609718$|^/dev/mapper/276 3fdda70ca458906c9ce90045609718$|^/dev/mapper/28ba7d1d12c7099ef6c9ce900456097 18$|^/dev/mapper/2e9d013564898d36f6c9ce90045609718$|", "r|.*|"] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--autobackup', 'n', '-a', 'y', '--deltag', u'IU_459872a1-8783-463a-98ae-9e56d8883abb', '--addtag', u'IU__remove_me_459872a1-8783-463a-98ae-9e56d8883abb', 'fb14e013-15f1-49b6-b129-210517aca0da/5d51973d-4198-4cec-994a-727317640ccb' ] succeeded with warnings: [' WARNING: Combining activation change with other commands is not advised.', ' Scan of VG fb14e013-15f1-49b6-b129-210517aca0da from /dev/mapper/2763fdda70ca 458906c9ce90045609718 found metadata seqno 26448 vs previous 26447.', ' Scan of VG fb14e013-15f1-49b6-b129-210517aca0da from /dev/mapper/2763fdda70ca458906c9ce90045609718 found metadata seqno 26448 vs previous 26447.'] (lvm:385) 2021-03-08 12:09:18,281+0100 INFO (jsonrpc/7) [vdsm.api] FINISH deleteImage return=None from=::ffff:172.16.122.253,44650, flow_id=39b2eddb, task_id=969cdd60-9b4c-43d9-960a-2a3b467b227c (api:54) 2021-03-08 12:09:18,293+0100 INFO (tasks/9) [storage.ThreadPool.WorkerThread] START task 969cdd60-9b4c-43d9-960a-2a3b467b227c (cmd=<bound method Task.commit of <vdsm.storage.task.Task instance at 0x7f7b606c27e8>>, args=None) (threadPool:208) 2021-03-08 12:09:18,295+0100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Image.delete succeeded in 2.75 seconds (__init__:312) 2021-03-08 12:09:18,299+0100 INFO (jsonrpc/0) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:172.16.122.253,44628, task_id=f6411446-8bed-4766-a038-892ebcaab65e (api:48) We have a SAN storage, with plenty of space, plus some NFS storage. But, again, there is plenty of space on all of those. From what I could find, the sanlock issue is not specifically related to space on the storage devices. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IPEV2PSURWQJY ZYLK6TQXPQ2BWYD3TD5/