5:12 a.m.
Hi,
We are running ovirt 4.3 and whenever we create a new VM, we get the error:
HSMGetAllTasksStatusesVDS failed: Error creating a new volume: (u"Volume creation
77be733a-a38a-4125-a3ae-beea960d9e28 failed: (28, 'Sanlock resource write
failure', 'No space left on device')",)
3/8/2111:56:57 AM
I have checked the engine logs and see the following:
2021-03-08 12:09:18,306+01 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(EE-ManagedThreadFactory-engineScheduled-Thread-40) [] BaseAsyncTask::logEndTaskFailure:
Task 'd4aa81a1-9adc-45c9-be46-05bc63862f4e' (Parent Command 'Unknown',
Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
ended with failure:
-- Result: 'cleanSuccess'
-- Message: 'VDSGenericException: VDSErrorException: Failed to
HSMGetAllTasksStatusesVDS, error = Volume does not exist:
(u'46bf0806-8d16-40db-92df-8436f43628a4',), code = 201',
-- Exception: 'VDSGenericException: VDSErrorException: Failed to
HSMGetAllTasksStatusesVDS, error = Volume does not exist:
(u'46bf0806-8d16-40db-92df-8436f43628a4',), code = 201'
Not sure how to solve this issue. I have tried restarting the engine, but that did not
solve it. Any tips?
Thanks.
5:25 a.m.
Hello,
I would recommend to check how much space you have available and which
partition is filling up with the following command:
df -h
Also if you have enough space, make sure to check your Inode usage using df
-i
On Mon, Mar 8, 2021 at 4:43 PM <kim.kargaard(a)noroff.no> wrote:
Hi,
We are running ovirt 4.3 and whenever we create a new VM, we get the error:
HSMGetAllTasksStatusesVDS failed: Error creating a new volume: (u"Volume
creation 77be733a-a38a-4125-a3ae-beea960d9e28 failed: (28, 'Sanlock
resource write failure', 'No space left on device')",)
3/8/2111:56:57 AM
I have checked the engine logs and see the following:
2021-03-08 12:09:18,306+01 ERROR
[org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(EE-ManagedThreadFactory-engineScheduled-Thread-40) []
BaseAsyncTask::logEndTaskFailure: Task
'd4aa81a1-9adc-45c9-be46-05bc63862f4e' (Parent Command 'Unknown',
Parameters Type
'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with
failure:
-- Result: 'cleanSuccess'
-- Message: 'VDSGenericException: VDSErrorException: Failed to
HSMGetAllTasksStatusesVDS, error = Volume does not exist:
(u'46bf0806-8d16-40db-92df-8436f43628a4',), code = 201',
-- Exception: 'VDSGenericException: VDSErrorException: Failed to
HSMGetAllTasksStatusesVDS, error = Volume does not exist:
(u'46bf0806-8d16-40db-92df-8436f43628a4',), code = 201'
Not sure how to solve this issue. I have tried restarting the engine, but
that did not solve it. Any tips?
Thanks.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JC7KB2SGXBB...
5:26 a.m.
Also share vdsm logs from all the hosts at the time it will be usefull
On Mon, Mar 8, 2021 at 4:55 PM Ritesh Chikatwar <rchikatw(a)redhat.com> wrote:
Hello,
I would recommend to check how much space you have available and which
partition is filling up with the following command:
df -h
Also if you have enough space, make sure to check your Inode usage using
df -i
On Mon, Mar 8, 2021 at 4:43 PM <kim.kargaard(a)noroff.no> wrote:
> Hi,
>
> We are running ovirt 4.3 and whenever we create a new VM, we get the
> error:
>
> HSMGetAllTasksStatusesVDS failed: Error creating a new volume: (u"Volume
> creation 77be733a-a38a-4125-a3ae-beea960d9e28 failed: (28, 'Sanlock
> resource write failure', 'No space left on device')",)
> 3/8/2111:56:57 AM
>
> I have checked the engine logs and see the following:
>
> 2021-03-08 12:09:18,306+01 ERROR
> [org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
> (EE-ManagedThreadFactory-engineScheduled-Thread-40) []
> BaseAsyncTask::logEndTaskFailure: Task
> 'd4aa81a1-9adc-45c9-be46-05bc63862f4e' (Parent Command 'Unknown',
> Parameters Type
> 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with
> failure:
> -- Result: 'cleanSuccess'
> -- Message: 'VDSGenericException: VDSErrorException: Failed to
> HSMGetAllTasksStatusesVDS, error = Volume does not exist:
> (u'46bf0806-8d16-40db-92df-8436f43628a4',), code = 201',
> -- Exception: 'VDSGenericException: VDSErrorException: Failed to
> HSMGetAllTasksStatusesVDS, error = Volume does not exist:
> (u'46bf0806-8d16-40db-92df-8436f43628a4',), code = 201'
>
> Not sure how to solve this issue. I have tried restarting the engine, but
> that did not solve it. Any tips?
>
> Thanks.
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JC7KB2SGXBB...
>
5:46 a.m.
Hi,
We have plenty of space, I have checked with both df -h and df -i.
Here are the vdsm logs from one node.
2021-03-08 12:09:15,550+0100 INFO (jsonrpc/7) [vdsm.api] START
deleteImage(sdUUID=u'fb14e013-15f1-49b6-b129-210517aca0da',
spUUID=u'aec485fa-cdbb-4979-8dc9-a2376559b2a4',
imgUUID=u'459872a1-8783-463a-98ae-9e56d8883abb', postZero=u'false',
force=u'false', discard=False) from=::ffff:172.16.122.253,44650, flow_id=39b2eddb,
task_id=969cdd60-9b4c-43d9-960a-2a3b467b227c (api:48)
2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] Unlink volume
symlink
u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images/01af322e-ff54-4981-9842-0f0a608da17c/46bf0806-8d16-40db-92df-8436f43628a4'
(blockSD:722)
2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] Unlink volume
symlink
u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images/01af322e-ff54-4981-9842-0f0a608da17c/27638d78-e935-44f4-b0cc-5ee2459e204a'
(blockSD:722)
2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] removed:
[u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images/01af322e-ff54-4981-9842-0f0a608da17c/27638d78-e935-44f4-b0cc-5ee2459e204a']
(blockSD:729)
2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] Removing
image directory
u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images/01af322e-ff54-4981-9842-0f0a608da17c'
(blockSD:735)
2021-03-08 12:09:15,729+0100 INFO (tasks/9) [storage.ThreadPool.WorkerThread] FINISH task
9a57dcf5-85b2-4c7c-8ffb-f11eecc2313f (threadPool:210)
2021-03-08 12:09:16,671+0100 WARN (jsonrpc/7) [storage.LVM] Removing stale lv:
fb14e013-15f1-49b6-b129-210517aca0da/46bf0806-8d16-40db-92df-8436f43628a4 (lvm:564)
2021-03-08 12:09:17,920+0100 WARN (tasks/7) [storage.LVM] lv:
46bf0806-8d16-40db-92df-8436f43628a4 not found in lvs vg:
fb14e013-15f1-49b6-b129-210517aca0da response (lvm:736)
2021-03-08 12:09:17,921+0100 ERROR (tasks/7) [storage.TaskManager.Task]
(Task='d4aa81a1-9adc-45c9-be46-05bc63862f4e') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in
run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in
wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1966, in
deleteVolume
vol = dom.produceVolume(imgUUID, volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 913, in
produceVolume
volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 814, in
__init__
self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line 62,
in __init__
volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86, in
__init__
self.validate()
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line 151,
in validate
raise se.VolumeDoesNotExist(self.volUUID)
VolumeDoesNotExist: Volume does not exist:
(u'46bf0806-8d16-40db-92df-8436f43628a4',)
2021-03-08 12:09:17,946+0100 INFO (tasks/7) [storage.ThreadPool.WorkerThread] FINISH task
d4aa81a1-9adc-45c9-be46-05bc63862f4e (threadPool:210)
2021-03-08 12:09:18,273+0100 WARN (jsonrpc/7) [storage.LVM] Command
['/usr/sbin/lvm', 'lvchange', '--config', 'devices {
preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/2181ad2edfa7b485c6c9ce90045609718$|^/dev/mapper/2763fdda70ca458906c9ce90045609718$|^/dev/mapper/28ba7d1d12c7099ef6c9ce90045609718$|^/dev/mapper/2e9d013564898d36f6c9ce90045609718$|",
"r|.*|"] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--autobackup',
'n', '-a', 'y', '--deltag',
u'IU_459872a1-8783-463a-98ae-9e56d8883abb', '--addtag',
u'IU__remove_me_459872a1-8783-463a-98ae-9e56d8883abb',
'fb14e013-15f1-49b6-b129-210517aca0da/5d51973d-4198-4cec-994a-727317640ccb']
succeeded with warnings: [' WARNING: Combining activation change with other commands
is not advised.', ' Scan of VG fb14e013-15f1-49b6-b129-210517aca0da from
/dev/mapper/2763fdda70ca
458906c9ce90045609718 found metadata seqno 26448 vs previous 26447.', ' Scan of
VG fb14e013-15f1-49b6-b129-210517aca0da from /dev/mapper/2763fdda70ca458906c9ce90045609718
found metadata seqno 26448 vs previous 26447.'] (lvm:385)
2021-03-08 12:09:18,281+0100 INFO (jsonrpc/7) [vdsm.api] FINISH deleteImage return=None
from=::ffff:172.16.122.253,44650, flow_id=39b2eddb,
task_id=969cdd60-9b4c-43d9-960a-2a3b467b227c (api:54)
2021-03-08 12:09:18,293+0100 INFO (tasks/9) [storage.ThreadPool.WorkerThread] START task
969cdd60-9b4c-43d9-960a-2a3b467b227c (cmd=<bound method Task.commit of
<vdsm.storage.task.Task instance at 0x7f7b606c27e8>>, args=None)
(threadPool:208)
2021-03-08 12:09:18,295+0100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call
Image.delete succeeded in 2.75 seconds (__init__:312)
2021-03-08 12:09:18,299+0100 INFO (jsonrpc/0) [vdsm.api] START
getAllTasksStatuses(spUUID=None, options=None) from=::ffff:172.16.122.253,44628,
task_id=f6411446-8bed-4766-a038-892ebcaab65e (api:48)
We have a SAN storage, with plenty of space, plus some NFS storage. But, again, there is
plenty of space on all of those. From what I could find, the sanlock issue is not
specifically related to space on the storage devices.
6:34 a.m.
On Monday, 8 March 2021 12:46:58 CET kim.kargaard(a)noroff.no wrote:
Hi,
We have plenty of space, I have checked with both df -h and df -i.
Here are the vdsm logs from one node.
what is the whole flow you are running? This looks like a flow where you try
to delete an image, which is probably missing one of its volumes.
In the log bellow there isn't anything related to sanlock. Or maybe wrong log?
2021-03-08 12:09:15,550+0100 INFO (jsonrpc/7) [vdsm.api] START
deleteImage(sdUUID=u'fb14e013-15f1-49b6-b129-210517aca0da',
spUUID=u'aec485fa-cdbb-4979-8dc9-a2376559b2a4',
imgUUID=u'459872a1-8783-463a-98ae-9e56d8883abb', postZero=u'false',
force=u'false', discard=False) from=::ffff:172.16.122.253,44650,
flow_id=39b2eddb, task_id=969cdd60-9b4c-43d9-960a-2a3b467b227c (api:48)
2021-03-08 12:09:15,719+0100 INFO (tasks/9)
[storage.StorageDomainManifest] Unlink volume symlink
u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images
/01af322e-ff54-4981-9842-0f0a608da17c/46bf0806-8d16-40db-92df-8436f43628a4'
(blockSD:722) 2021-03-08 12:09:15,719+0100 INFO (tasks/9)
[storage.StorageDomainManifest] Unlink volume symlink
u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images
/01af322e-ff54-4981-9842-0f0a608da17c/27638d78-e935-44f4-b0cc-5ee2459e204a'
(blockSD:722) 2021-03-08 12:09:15,719+0100 INFO (tasks/9)
[storage.StorageDomainManifest] removed:
[u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/image
s/01af322e-ff54-4981-9842-0f0a608da17c/27638d78-e935-44f4-b0cc-5ee2459e204a'
] (blockSD:729) 2021-03-08 12:09:15,719+0100 INFO (tasks/9)
[storage.StorageDomainManifest] Removing image directory
u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images
/01af322e-ff54-4981-9842-0f0a608da17c' (blockSD:735) 2021-03-08
12:09:15,729+0100 INFO (tasks/9) [storage.ThreadPool.WorkerThread] FINISH
task 9a57dcf5-85b2-4c7c-8ffb-f11eecc2313f (threadPool:210) 2021-03-08
12:09:16,671+0100 WARN (jsonrpc/7) [storage.LVM] Removing stale lv:
fb14e013-15f1-49b6-b129-210517aca0da/46bf0806-8d16-40db-92df-8436f43628a4
(lvm:564) 2021-03-08 12:09:17,920+0100 WARN (tasks/7) [storage.LVM] lv:
46bf0806-8d16-40db-92df-8436f43628a4 not found in lvs vg:
fb14e013-15f1-49b6-b129-210517aca0da response (lvm:736) 2021-03-08
12:09:17,921+0100 ERROR (tasks/7) [storage.TaskManager.Task]
(Task='d4aa81a1-9adc-45c9-be46-05bc63862f4e') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in
run
return self.cmd(*self.argslist, **self.argsdict)
File
"/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line
79, in wrapper
return method(self, *args, **kwargs)
File
"/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1966, in
deleteVolume
vol = dom.produceVolume(imgUUID, volUUID)
File
"/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 913, in
produceVolume
volUUID)
File
"/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 814,
in __init__
self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID,
volUUID) File
"/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line 62, in
__init__ volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86,
in __init__
self.validate()
File
"/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line
151, in validate
raise se.VolumeDoesNotExist(self.volUUID)
VolumeDoesNotExist: Volume does not exist:
(u'46bf0806-8d16-40db-92df-8436f43628a4',)
2021-03-08 12:09:17,946+0100
INFO (tasks/7) [storage.ThreadPool.WorkerThread] FINISH task
d4aa81a1-9adc-45c9-be46-05bc63862f4e (threadPool:210) 2021-03-08
12:09:18,273+0100 WARN (jsonrpc/7) [storage.LVM] Command ['/usr/sbin/lvm',
'lvchange', '--config', 'devices {
preferred_names=["^/dev/mapper/"]
ignore_suspended_devices=1 write_cache_state=0
disable_after_error_count=3
filter=["a|^/dev/mapper/2181ad2edfa7b485c6c9ce90045609718$|^/dev/mapper/276
3fdda70ca458906c9ce90045609718$|^/dev/mapper/28ba7d1d12c7099ef6c9ce900456097
18$|^/dev/mapper/2e9d013564898d36f6c9ce90045609718$|", "r|.*|"] } global {
locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 }
backup { retain_min=50 retain_days=0 }', '--autobackup', 'n',
'-a', 'y',
'--deltag', u'IU_459872a1-8783-463a-98ae-9e56d8883abb',
'--addtag',
u'IU__remove_me_459872a1-8783-463a-98ae-9e56d8883abb',
'fb14e013-15f1-49b6-b129-210517aca0da/5d51973d-4198-4cec-994a-727317640ccb'
] succeeded with warnings: [' WARNING: Combining activation change with
other commands is not advised.', ' Scan of VG
fb14e013-15f1-49b6-b129-210517aca0da from /dev/mapper/2763fdda70ca
458906c9ce90045609718 found metadata seqno 26448 vs previous 26447.', '
Scan of VG fb14e013-15f1-49b6-b129-210517aca0da from
/dev/mapper/2763fdda70ca458906c9ce90045609718 found metadata seqno 26448 vs
previous 26447.'] (lvm:385) 2021-03-08 12:09:18,281+0100 INFO (jsonrpc/7)
[vdsm.api] FINISH deleteImage return=None from=::ffff:172.16.122.253,44650,
flow_id=39b2eddb, task_id=969cdd60-9b4c-43d9-960a-2a3b467b227c (api:54)
2021-03-08 12:09:18,293+0100 INFO (tasks/9)
[storage.ThreadPool.WorkerThread] START task
969cdd60-9b4c-43d9-960a-2a3b467b227c (cmd=<bound method Task.commit of
<vdsm.storage.task.Task instance at 0x7f7b606c27e8>>, args=None)
(threadPool:208) 2021-03-08 12:09:18,295+0100 INFO (jsonrpc/7)
[jsonrpc.JsonRpcServer] RPC call Image.delete succeeded in 2.75 seconds
(__init__:312) 2021-03-08 12:09:18,299+0100 INFO (jsonrpc/0) [vdsm.api]
START getAllTasksStatuses(spUUID=None, options=None)
from=::ffff:172.16.122.253,44628,
task_id=f6411446-8bed-4766-a038-892ebcaab65e (api:48)
We have a SAN storage, with plenty of space, plus some NFS storage. But,
again, there is plenty of space on all of those. From what I could find,
the sanlock issue is not specifically related to space on the storage
devices. _______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/IPEV2PSURWQJY
ZYLK6TQXPQ2BWYD3TD5/
1148
days inactive
1148
days old
4 comments
3 participants
participants (3)
-
kim.kargaard＠noroff.no -
Ritesh Chikatwar -
Vojtech Juranek