Cannot acquire Lock .... snapshot error
Dear All, my ovirt 4.2.1.7-1.el7.centos has three hypervisors, lvm storage and virtiual machine with ovirt-engine. All works fine but with one vm when I try to remove its snapshot I have this error: 2018-06-22 07:35:48,155+0200 INFO (jsonrpc/5) [vdsm.api] START prepareMerge(spUUID=u'18d57688-6ed4-43b8-bd7c-0665b55950b7', subchainInfo={u'img_id': u'c5611862-6504-445e-a6c8-f1e1a95b5df7', u'sd_id': u'47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5', u'top_id': u'0e6f7512-871d-4645-b9c6-320ba7e3bee7', u'base_id': u'e156ac2e-09ac-4e1e-a139-17fa374a96d4'}) from=::ffff:10.0.0.46,53304, flow_id=07011450-2296-4a13-a9ed-5d5d2b91be98, task_id=87f95d85-cc3d-4f29-9883-a4dbb3808f88 (api:46) 2018-06-22 07:35:48,406+0200 INFO (tasks/3) [storage.merge] Preparing subchain <SubchainInfo sd_id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, img_id=c5611862-6504-445e-a6c8-f1e1a95b5df7, top_id=0e6f7512-871d-4645-b9c6-320ba7e3bee7, base_id=e156ac2e-09ac-4e1e-a139-17fa374a96d4 base_generation=None at 0x7fcf84ae2510> for merge (merge:177) 2018-06-22 07:35:48,614+0200 INFO (tasks/3) [storage.SANLock] Acquiring Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360) for host id 1 (clusterlock:377) 2018-06-22 07:35:48,634+0200 ERROR (tasks/3) [storage.guarded] Error acquiring lock <VolumeLease ns=04_lease_47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, name=e156ac2e-09ac-4e1e-a139-17fa374a96d4, mode=exclusive at 0x7fcfe09ddf90> (guarded:96) AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')" 2018-06-22 07:35:56,881+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'87f95d85-cc3d-4f29-9883-a4dbb3808f88': {'code': 651, 'message': 'Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name=\'e156ac2e-09ac-4e1e-a139-17fa374a96d4\', path=\'/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases\', offset=115343360), err=(-227, \'Sanlock resource not acquired\', \'Lease resource name is incorrect\')"', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '87f95d85-cc3d-4f29-9883-a4dbb3808f88'}}} from=::ffff:10.0.0.46,53136, task_id=d0e2f4e3-90cb-43c6-aa08-98d1f7efb1bd (api:52) Any ideas ? Many thanks to everybody. Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
On Fri, Jun 22, 2018 at 3:13 PM Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Dear All, my ovirt 4.2.1.7-1.el7.centos has three hypervisors, lvm storage and virtiual machine with ovirt-engine. All works fine but with one vm when I try to remove its snapshot I have this error:
2018-06-22 07:35:48,155+0200 INFO (jsonrpc/5) [vdsm.api] START prepareMerge(spUUID=u'18d57688-6ed4-43b8-bd7c-0665b55950b7', subchainInfo={u'img_id': u'c5611862-6504-445e-a6c8-f1e1a95b5df7', u'sd_id': u'47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5', u'top_id': u'0e6f7512-871d-4645-b9c6-320ba7e3bee7', u'base_id': u'e156ac2e-09ac-4e1e-a139-17fa374a96d4'}) from=::ffff:10.0.0.46,53304, flow_id=07011450-2296-4a13-a9ed-5d5d2b91be98, task_id=87f95d85-cc3d-4f29-9883-a4dbb3808f88 (api:46) 2018-06-22 07:35:48,406+0200 INFO (tasks/3) [storage.merge] Preparing subchain <SubchainInfo sd_id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, img_id=c5611862-6504-445e-a6c8-f1e1a95b5df7, top_id=0e6f7512-871d-4645-b9c6-320ba7e3bee7, base_id=e156ac2e-09ac-4e1e-a139-17fa374a96d4 base_generation=None at 0x7fcf84ae2510> for merge (merge:177) 2018-06-22 07:35:48,614+0200 INFO (tasks/3) [storage.SANLock] Acquiring Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360) for host id 1 (clusterlock:377) 2018-06-22 07:35:48,634+0200 ERROR (tasks/3) [storage.guarded] Error acquiring lock <VolumeLease ns=04_lease_47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, name=e156ac2e-09ac-4e1e-a139-17fa374a96d4, mode=exclusive at 0x7fcfe09ddf90> (guarded:96) AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')" 2018-06-22 07:35:56,881+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'87f95d85-cc3d-4f29-9883-a4dbb3808f88': {'code': 651, 'message': 'Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name=\'e156ac2e-09ac-4e1e-a139-17fa374a96d4\', path=\'/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases\', offset=115343360), err=(-227, \'Sanlock resource not acquired\', \'Lease resource name is incorrect\')"', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '87f95d85-cc3d-4f29-9883-a4dbb3808f88'}}} from=::ffff:10.0.0.46,53136, task_id=d0e2f4e3-90cb-43c6-aa08-98d1f7efb1bd (api:52)
The issue is corrupted lease for this volume: AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')" This root cause is faulty merge code in ovirt < 4.1, creating volume leases with incorrect name. These corrupted leases were not detected until you upgrade to ovirt >= 4.1, because we started to use volume leases for storage operations. The fix is to run vdsm-tool check-volume-leases This will check and repair corrupted leases. Adding Ala to add more info if needed. Nir
Dear Friends , to fix my problem I've try vdsm-tool command but it's seem an error: [root@infn-vm05 vdsm]# vdsm-tool check-volume-lease Usage: /usr/bin/vdsm-tool [options] <action> [arguments] Valid options: ...... as you can see there isn't check-volumes-option and my ovirt engine is already at 4.2. Any other ideas ? Thanks a lot ! Best Regards Enrico Il 22/06/2018 17:46, Nir Soffer ha scritto:
On Fri, Jun 22, 2018 at 3:13 PM Enrico Becchetti <enrico.becchetti@pg.infn.it <mailto:enrico.becchetti@pg.infn.it>> wrote:
Dear All, my ovirt 4.2.1.7-1.el7.centos has three hypervisors, lvm storage and virtiual machine with ovirt-engine. All works fine but with one vm when I try to remove its snapshot I have this error:
2018-06-22 07:35:48,155+0200 INFO (jsonrpc/5) [vdsm.api] START prepareMerge(spUUID=u'18d57688-6ed4-43b8-bd7c-0665b55950b7', subchainInfo={u'img_id': u'c5611862-6504-445e-a6c8-f1e1a95b5df7', u'sd_id': u'47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5', u'top_id': u'0e6f7512-871d-4645-b9c6-320ba7e3bee7', u'base_id': u'e156ac2e-09ac-4e1e-a139-17fa374a96d4'}) from=::ffff:10.0.0.46,53304, flow_id=07011450-2296-4a13-a9ed-5d5d2b91be98, task_id=87f95d85-cc3d-4f29-9883-a4dbb3808f88 (api:46) 2018-06-22 07:35:48,406+0200 INFO (tasks/3) [storage.merge] Preparing subchain <SubchainInfo sd_id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, img_id=c5611862-6504-445e-a6c8-f1e1a95b5df7, top_id=0e6f7512-871d-4645-b9c6-320ba7e3bee7, base_id=e156ac2e-09ac-4e1e-a139-17fa374a96d4 base_generation=None at 0x7fcf84ae2510> for merge (merge:177) 2018-06-22 07:35:48,614+0200 INFO (tasks/3) [storage.SANLock] Acquiring Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360) for host id 1 (clusterlock:377) 2018-06-22 07:35:48,634+0200 ERROR (tasks/3) [storage.guarded] Error acquiring lock <VolumeLease ns=04_lease_47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, name=e156ac2e-09ac-4e1e-a139-17fa374a96d4, mode=exclusive at 0x7fcfe09ddf90> (guarded:96) AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')" 2018-06-22 07:35:56,881+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'87f95d85-cc3d-4f29-9883-a4dbb3808f88': {'code': 651, 'message': 'Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name=\'e156ac2e-09ac-4e1e-a139-17fa374a96d4\', path=\'/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases\', offset=115343360), err=(-227, \'Sanlock resource not acquired\', \'Lease resource name is incorrect\')"', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '87f95d85-cc3d-4f29-9883-a4dbb3808f88'}}} from=::ffff:10.0.0.46,53136, task_id=d0e2f4e3-90cb-43c6-aa08-98d1f7efb1bd (api:52)
The issue is corrupted lease for this volume:
AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')"
This root cause is faulty merge code in ovirt < 4.1, creating volume leases with incorrect name. These corrupted leases were not detected until you upgrade to ovirt >= 4.1, because we started to use volume leases for storage operations.
The fix is to run
vdsm-tool check-volume-leases
This will check and repair corrupted leases.
Adding Ala to add more info if needed.
Nir
-- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
Dear Ala, if you have a few minutes for me I'd like to ask you to read my issue. It's a strange problem because my vm works fine but I can't delete its snapshoot. Thanks a lot Best Regards Enrico -------- Messaggio Inoltrato -------- Oggetto: [ovirt-users] Re: Cannot acquire Lock .... snapshot error Data: Mon, 25 Jun 2018 14:20:21 +0200 Mittente: Enrico Becchetti <enrico.becchetti@pg.infn.it> A: Nir Soffer <nsoffer@redhat.com> CC: users <users@ovirt.org> Dear Friends , to fix my problem I've try vdsm-tool command but it's seem an error: [root@infn-vm05 vdsm]# vdsm-tool check-volume-lease Usage: /usr/bin/vdsm-tool [options] <action> [arguments] Valid options: ...... as you can see there isn't check-volumes-option and my ovirt engine is already at 4.2. Any other ideas ? Thanks a lot ! Best Regards Enrico Il 22/06/2018 17:46, Nir Soffer ha scritto:
On Fri, Jun 22, 2018 at 3:13 PM Enrico Becchetti <enrico.becchetti@pg.infn.it <mailto:enrico.becchetti@pg.infn.it>> wrote:
Dear All, my ovirt 4.2.1.7-1.el7.centos has three hypervisors, lvm storage and virtiual machine with ovirt-engine. All works fine but with one vm when I try to remove its snapshot I have this error:
2018-06-22 07:35:48,155+0200 INFO (jsonrpc/5) [vdsm.api] START prepareMerge(spUUID=u'18d57688-6ed4-43b8-bd7c-0665b55950b7', subchainInfo={u'img_id': u'c5611862-6504-445e-a6c8-f1e1a95b5df7', u'sd_id': u'47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5', u'top_id': u'0e6f7512-871d-4645-b9c6-320ba7e3bee7', u'base_id': u'e156ac2e-09ac-4e1e-a139-17fa374a96d4'}) from=::ffff:10.0.0.46,53304, flow_id=07011450-2296-4a13-a9ed-5d5d2b91be98, task_id=87f95d85-cc3d-4f29-9883-a4dbb3808f88 (api:46) 2018-06-22 07:35:48,406+0200 INFO (tasks/3) [storage.merge] Preparing subchain <SubchainInfo sd_id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, img_id=c5611862-6504-445e-a6c8-f1e1a95b5df7, top_id=0e6f7512-871d-4645-b9c6-320ba7e3bee7, base_id=e156ac2e-09ac-4e1e-a139-17fa374a96d4 base_generation=None at 0x7fcf84ae2510> for merge (merge:177) 2018-06-22 07:35:48,614+0200 INFO (tasks/3) [storage.SANLock] Acquiring Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360) for host id 1 (clusterlock:377) 2018-06-22 07:35:48,634+0200 ERROR (tasks/3) [storage.guarded] Error acquiring lock <VolumeLease ns=04_lease_47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, name=e156ac2e-09ac-4e1e-a139-17fa374a96d4, mode=exclusive at 0x7fcfe09ddf90> (guarded:96) AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')" 2018-06-22 07:35:56,881+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'87f95d85-cc3d-4f29-9883-a4dbb3808f88': {'code': 651, 'message': 'Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name=\'e156ac2e-09ac-4e1e-a139-17fa374a96d4\', path=\'/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases\', offset=115343360), err=(-227, \'Sanlock resource not acquired\', \'Lease resource name is incorrect\')"', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '87f95d85-cc3d-4f29-9883-a4dbb3808f88'}}} from=::ffff:10.0.0.46,53136, task_id=d0e2f4e3-90cb-43c6-aa08-98d1f7efb1bd (api:52)
The issue is corrupted lease for this volume:
AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')"
This root cause is faulty merge code in ovirt < 4.1, creating volume leases with incorrect name. These corrupted leases were not detected until you upgrade to ovirt >= 4.1, because we started to use volume leases for storage operations.
The fix is to run
vdsm-tool check-volume-leases
This will check and repair corrupted leases.
Adding Ala to add more info if needed.
Nir
-- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
Hi Enrico, What's the vdsm version that you are using? The tool introduced in vdsm 4.20.24. On Tue, Jun 26, 2018 at 3:51 PM, Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Dear Ala, if you have a few minutes for me I'd like to ask you to read my issue. It's a strange problem because my vm works fine but I can't delete its snapshoot. Thanks a lot Best Regards Enrico
-------- Messaggio Inoltrato -------- Oggetto: [ovirt-users] Re: Cannot acquire Lock .... snapshot error Data: Mon, 25 Jun 2018 14:20:21 +0200 Mittente: Enrico Becchetti <enrico.becchetti@pg.infn.it> <enrico.becchetti@pg.infn.it> A: Nir Soffer <nsoffer@redhat.com> <nsoffer@redhat.com> CC: users <users@ovirt.org> <users@ovirt.org>
Dear Friends , to fix my problem I've try vdsm-tool command but it's seem an error:
[root@infn-vm05 vdsm]# vdsm-tool check-volume-lease Usage: /usr/bin/vdsm-tool [options] <action> [arguments] Valid options: ......
as you can see there isn't check-volumes-option and my ovirt engine is already at 4.2. Any other ideas ? Thanks a lot ! Best Regards Enrico
Il 22/06/2018 17:46, Nir Soffer ha scritto:
On Fri, Jun 22, 2018 at 3:13 PM Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Dear All, my ovirt 4.2.1.7-1.el7.centos has three hypervisors, lvm storage and virtiual machine with ovirt-engine. All works fine but with one vm when I try to remove its snapshot I have this error:
2018-06-22 07:35:48,155+0200 INFO (jsonrpc/5) [vdsm.api] START prepareMerge(spUUID=u'18d57688-6ed4-43b8-bd7c-0665b55950b7', subchainInfo={u'img_id': u'c5611862-6504-445e-a6c8-f1e1a95b5df7', u'sd_id': u'47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5', u'top_id': u'0e6f7512-871d-4645-b9c6-320ba7e3bee7', u'base_id': u'e156ac2e-09ac-4e1e-a139-17fa374a96d4'}) from=::ffff:10.0.0.46,53304, flow_id=07011450-2296-4a13-a9ed-5d5d2b91be98, task_id=87f95d85-cc3d-4f29-9883-a4dbb3808f88 (api:46) 2018-06-22 07:35:48,406+0200 INFO (tasks/3) [storage.merge] Preparing subchain <SubchainInfo sd_id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, img_id=c5611862-6504-445e-a6c8-f1e1a95b5df7, top_id=0e6f7512-871d-4645-b9c6-320ba7e3bee7, base_id=e156ac2e-09ac-4e1e-a139-17fa374a96d4 base_generation=None at 0x7fcf84ae2510> for merge (merge:177) 2018-06-22 07:35:48,614+0200 INFO (tasks/3) [storage.SANLock] Acquiring Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360) for host id 1 (clusterlock:377) 2018-06-22 07:35:48,634+0200 ERROR (tasks/3) [storage.guarded] Error acquiring lock <VolumeLease ns=04_lease_47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, name=e156ac2e-09ac-4e1e-a139-17fa374a96d4, mode=exclusive at 0x7fcfe09ddf90> (guarded:96) AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')" 2018-06-22 07:35:56,881+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'87f95d85-cc3d-4f29-9883-a4dbb3808f88': {'code': 651, 'message': 'Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name=\'e156ac2e-09ac-4e1e-a139-17fa374a96d4\', path=\'/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases\', offset=115343360), err=(-227, \'Sanlock resource not acquired\', \'Lease resource name is incorrect\')"', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '87f95d85-cc3d-4f29-9883-a4dbb3808f88'}}} from=::ffff:10.0.0.46,53136, task_id=d0e2f4e3-90cb-43c6-aa08-98d1f7efb1bd (api:52)
The issue is corrupted lease for this volume:
AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')"
This root cause is faulty merge code in ovirt < 4.1, creating volume leases with incorrect name. These corrupted leases were not detected until you upgrade to ovirt >= 4.1, because we started to use volume leases for storage operations.
The fix is to run
vdsm-tool check-volume-leases
This will check and repair corrupted leases.
Adding Ala to add more info if needed.
Nir
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
Hi, I run this command from my SPM , Centos 7.4.1708: [root@infn-vm05 vdsm]# rpm -qa | grep -i vdsm vdsm-hook-vmfex-dev-4.20.17-1.el7.centos.noarch vdsm-python-4.20.17-1.el7.centos.noarch vdsm-hook-fcoe-4.20.17-1.el7.centos.noarch vdsm-common-4.20.17-1.el7.centos.noarch vdsm-jsonrpc-4.20.17-1.el7.centos.noarch vdsm-hook-ethtool-options-4.20.17-1.el7.centos.noarch vdsm-hook-openstacknet-4.20.17-1.el7.centos.noarch vdsm-http-4.20.17-1.el7.centos.noarch vdsm-client-4.20.17-1.el7.centos.noarch vdsm-gluster-4.20.17-1.el7.centos.noarch vdsm-hook-vfio-mdev-4.20.17-1.el7.centos.noarch vdsm-api-4.20.17-1.el7.centos.noarch vdsm-network-4.20.17-1.el7.centos.x86_64 vdsm-yajsonrpc-4.20.17-1.el7.centos.noarch vdsm-4.20.17-1.el7.centos.x86_64 vdsm-hook-vhostmd-4.20.17-1.el7.centos.noarch Thanks !!! Enrico Il 26/06/2018 15:21, Ala Hino ha scritto:
Hi Enrico,
What's the vdsm version that you are using?
The tool introduced in vdsm 4.20.24.
On Tue, Jun 26, 2018 at 3:51 PM, Enrico Becchetti <enrico.becchetti@pg.infn.it <mailto:enrico.becchetti@pg.infn.it>> wrote:
Dear Ala, if you have a few minutes for me I'd like to ask you to read my issue. It's a strange problem because my vm works fine but I can't delete its snapshoot. Thanks a lot Best Regards Enrico
-------- Messaggio Inoltrato -------- Oggetto: [ovirt-users] Re: Cannot acquire Lock .... snapshot error Data: Mon, 25 Jun 2018 14:20:21 +0200 Mittente: Enrico Becchetti <enrico.becchetti@pg.infn.it> <mailto:enrico.becchetti@pg.infn.it> A: Nir Soffer <nsoffer@redhat.com> <mailto:nsoffer@redhat.com> CC: users <users@ovirt.org> <mailto:users@ovirt.org>
Dear Friends , to fix my problem I've try vdsm-tool command but it's seem an error:
[root@infn-vm05 vdsm]# vdsm-tool check-volume-lease Usage: /usr/bin/vdsm-tool [options] <action> [arguments] Valid options: ......
as you can see there isn't check-volumes-option and my ovirt engine is already at 4.2. Any other ideas ? Thanks a lot ! Best Regards Enrico
Il 22/06/2018 17:46, Nir Soffer ha scritto:
On Fri, Jun 22, 2018 at 3:13 PM Enrico Becchetti <enrico.becchetti@pg.infn.it <mailto:enrico.becchetti@pg.infn.it>> wrote:
Dear All, my ovirt 4.2.1.7-1.el7.centos has three hypervisors, lvm storage and virtiual machine with ovirt-engine. All works fine but with one vm when I try to remove its snapshot I have this error:
2018-06-22 07:35:48,155+0200 INFO (jsonrpc/5) [vdsm.api] START prepareMerge(spUUID=u'18d57688-6ed4-43b8-bd7c-0665b55950b7', subchainInfo={u'img_id': u'c5611862-6504-445e-a6c8-f1e1a95b5df7', u'sd_id': u'47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5', u'top_id': u'0e6f7512-871d-4645-b9c6-320ba7e3bee7', u'base_id': u'e156ac2e-09ac-4e1e-a139-17fa374a96d4'}) from=::ffff:10.0.0.46,53304, flow_id=07011450-2296-4a13-a9ed-5d5d2b91be98, task_id=87f95d85-cc3d-4f29-9883-a4dbb3808f88 (api:46) 2018-06-22 07:35:48,406+0200 INFO (tasks/3) [storage.merge] Preparing subchain <SubchainInfo sd_id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, img_id=c5611862-6504-445e-a6c8-f1e1a95b5df7, top_id=0e6f7512-871d-4645-b9c6-320ba7e3bee7, base_id=e156ac2e-09ac-4e1e-a139-17fa374a96d4 base_generation=None at 0x7fcf84ae2510> for merge (merge:177) 2018-06-22 07:35:48,614+0200 INFO (tasks/3) [storage.SANLock] Acquiring Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360) for host id 1 (clusterlock:377) 2018-06-22 07:35:48,634+0200 ERROR (tasks/3) [storage.guarded] Error acquiring lock <VolumeLease ns=04_lease_47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, name=e156ac2e-09ac-4e1e-a139-17fa374a96d4, mode=exclusive at 0x7fcfe09ddf90> (guarded:96) AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')" 2018-06-22 07:35:56,881+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'87f95d85-cc3d-4f29-9883-a4dbb3808f88': {'code': 651, 'message': 'Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name=\'e156ac2e-09ac-4e1e-a139-17fa374a96d4\', path=\'/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases\', offset=115343360), err=(-227, \'Sanlock resource not acquired\', \'Lease resource name is incorrect\')"', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '87f95d85-cc3d-4f29-9883-a4dbb3808f88'}}} from=::ffff:10.0.0.46,53136, task_id=d0e2f4e3-90cb-43c6-aa08-98d1f7efb1bd (api:52)
The issue is corrupted lease for this volume:
AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')"
This root cause is faulty merge code in ovirt < 4.1, creating volume leases with incorrect name. These corrupted leases were not detected until you upgrade to ovirt >= 4.1, because we started to use volume leases for storage operations.
The fix is to run
vdsm-tool check-volume-leases
This will check and repair corrupted leases.
Adding Ala to add more info if needed.
Nir
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it <http://pg.infn.it> ______________________________________________________________________
-- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
You are running vdsm-4.20.17, and the tool introduced in vdsm-4.20.24. You will have to upgrade vdsm to be able to use the tool. On Tue, Jun 26, 2018 at 4:29 PM, Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Hi, I run this command from my SPM , Centos 7.4.1708:
[root@infn-vm05 vdsm]# rpm -qa | grep -i vdsm vdsm-hook-vmfex-dev-4.20.17-1.el7.centos.noarch vdsm-python-4.20.17-1.el7.centos.noarch vdsm-hook-fcoe-4.20.17-1.el7.centos.noarch vdsm-common-4.20.17-1.el7.centos.noarch vdsm-jsonrpc-4.20.17-1.el7.centos.noarch vdsm-hook-ethtool-options-4.20.17-1.el7.centos.noarch vdsm-hook-openstacknet-4.20.17-1.el7.centos.noarch vdsm-http-4.20.17-1.el7.centos.noarch vdsm-client-4.20.17-1.el7.centos.noarch vdsm-gluster-4.20.17-1.el7.centos.noarch vdsm-hook-vfio-mdev-4.20.17-1.el7.centos.noarch vdsm-api-4.20.17-1.el7.centos.noarch vdsm-network-4.20.17-1.el7.centos.x86_64 vdsm-yajsonrpc-4.20.17-1.el7.centos.noarch vdsm-4.20.17-1.el7.centos.x86_64 vdsm-hook-vhostmd-4.20.17-1.el7.centos.noarch
Thanks !!! Enrico
Il 26/06/2018 15:21, Ala Hino ha scritto:
Hi Enrico,
What's the vdsm version that you are using?
The tool introduced in vdsm 4.20.24.
On Tue, Jun 26, 2018 at 3:51 PM, Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Dear Ala, if you have a few minutes for me I'd like to ask you to read my issue. It's a strange problem because my vm works fine but I can't delete its snapshoot. Thanks a lot Best Regards Enrico
-------- Messaggio Inoltrato -------- Oggetto: [ovirt-users] Re: Cannot acquire Lock .... snapshot error Data: Mon, 25 Jun 2018 14:20:21 +0200 Mittente: Enrico Becchetti <enrico.becchetti@pg.infn.it> <enrico.becchetti@pg.infn.it> A: Nir Soffer <nsoffer@redhat.com> <nsoffer@redhat.com> CC: users <users@ovirt.org> <users@ovirt.org>
Dear Friends , to fix my problem I've try vdsm-tool command but it's seem an error:
[root@infn-vm05 vdsm]# vdsm-tool check-volume-lease Usage: /usr/bin/vdsm-tool [options] <action> [arguments] Valid options: ......
as you can see there isn't check-volumes-option and my ovirt engine is already at 4.2. Any other ideas ? Thanks a lot ! Best Regards Enrico
Il 22/06/2018 17:46, Nir Soffer ha scritto:
On Fri, Jun 22, 2018 at 3:13 PM Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Dear All, my ovirt 4.2.1.7-1.el7.centos has three hypervisors, lvm storage and virtiual machine with ovirt-engine. All works fine but with one vm when I try to remove its snapshot I have this error:
2018-06-22 07:35:48,155+0200 INFO (jsonrpc/5) [vdsm.api] START prepareMerge(spUUID=u'18d57688-6ed4-43b8-bd7c-0665b55950b7', subchainInfo={u'img_id': u'c5611862-6504-445e-a6c8-f1e1a95b5df7', u'sd_id': u'47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5', u'top_id': u'0e6f7512-871d-4645-b9c6-320ba7e3bee7', u'base_id': u'e156ac2e-09ac-4e1e-a139-17fa374a96d4'}) from=::ffff:10.0.0.46,53304, flow_id=07011450-2296-4a13-a9ed-5d5d2b91be98, task_id=87f95d85-cc3d-4f29-9883-a4dbb3808f88 (api:46) 2018-06-22 07:35:48,406+0200 INFO (tasks/3) [storage.merge] Preparing subchain <SubchainInfo sd_id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, img_id=c5611862-6504-445e-a6c8-f1e1a95b5df7, top_id=0e6f7512-871d-4645-b9c6-320ba7e3bee7, base_id=e156ac2e-09ac-4e1e-a139-17fa374a96d4 base_generation=None at 0x7fcf84ae2510> for merge (merge:177) 2018-06-22 07:35:48,614+0200 INFO (tasks/3) [storage.SANLock] Acquiring Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360) for host id 1 (clusterlock:377) 2018-06-22 07:35:48,634+0200 ERROR (tasks/3) [storage.guarded] Error acquiring lock <VolumeLease ns=04_lease_47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, name=e156ac2e-09ac-4e1e-a139-17fa374a96d4, mode=exclusive at 0x7fcfe09ddf90> (guarded:96) AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')" 2018-06-22 07:35:56,881+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'87f95d85-cc3d-4f29-9883-a4dbb3808f88': {'code': 651, 'message': 'Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name=\'e156ac2e-09ac-4e1e-a139-17fa374a96d4\', path=\'/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases\', offset=115343360), err=(-227, \'Sanlock resource not acquired\', \'Lease resource name is incorrect\')"', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '87f95d85-cc3d-4f29-9883-a4dbb3808f88'}}} from=::ffff:10.0.0.46,53136, task_id=d0e2f4e3-90cb-43c6-aa08-98d1f7efb1bd (api:52)
The issue is corrupted lease for this volume:
AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')"
This root cause is faulty merge code in ovirt < 4.1, creating volume leases with incorrect name. These corrupted leases were not detected until you upgrade to ovirt >= 4.1, because we started to use volume leases for storage operations.
The fix is to run
vdsm-tool check-volume-leases
This will check and repair corrupted leases.
Adding Ala to add more info if needed.
Nir
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
Hi all, after update vdsm I run this command: /[root@infm-vm04 ~]# vdsm-tool -v check-volume-leases// //WARNING: Make sure there are no running storage operations.// // //Do you want to check volume leases? [yes,NO] yes// // //Checking active storage domains. This can take several minutes, please wait.// / After that I saw some volumes with issue: /The following volume leases need repair:// // //- domain: 47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5// // // - image: 4b2a6552-847c-43f4-a180-b037d0b93a30// // - volume: 6eb8caf0-b120-40e5-86b5-405f15d1245a// // - image: 871196b2-9d8b-422f-9a3e-be54e100dc5c// // - volume: 861ff7dd-a01c-47f7-8a01-95bc766c2607// // - image: 267b8b8c-da09-44ef-81b3-065dfa2e7085// // - volume: d5f3158a-87ac-4c02-84ba-fcb86b8688a0// // - image: c5611862-6504-445e-a6c8-f1e1a95b5df7// // - volume: e156ac2e-09ac-4e1e-a139-17fa374a96d4// // - image: e765c9c4-2ef9-4c8f-a573-8cd2b0a0a3a2// // - volume: 47d81cbe-598a-402a-9795-1d046c45b7b1// // - image: ab88a08c-910d-44dd-bdf8-8242001ba527// // - volume: 86c72239-525f-4b0b-9aa6-763fc71340bc// // - image: 5e8c4620-b6a5-4fc6-a5bb-f209173d186c// // - volume: 0f52831b-ec35-4140-9a8c-fa24c2647f17// // - image: a67809fc-b830-4ea3-af66-b5b9285b4924// // - volume: 26c6bdd7-1382-4e8e-addc-dcd3898b317f// // //Do you want to repair the leases? [yes,NO] / what happen If I try to repair them ? Is there any impact to my running vm ? Thanks a lot !!! Best Regards Enrico Il 26/06/2018 15:32, Ala Hino ha scritto:
You are running vdsm-4.20.17, and the tool introduced in vdsm-4.20.24. You will have to upgrade vdsm to be able to use the tool.
On Tue, Jun 26, 2018 at 4:29 PM, Enrico Becchetti <enrico.becchetti@pg.infn.it <mailto:enrico.becchetti@pg.infn.it>> wrote:
Hi, I run this command from my SPM , Centos 7.4.1708:
[root@infn-vm05 vdsm]# rpm -qa | grep -i vdsm vdsm-hook-vmfex-dev-4.20.17-1.el7.centos.noarch vdsm-python-4.20.17-1.el7.centos.noarch vdsm-hook-fcoe-4.20.17-1.el7.centos.noarch vdsm-common-4.20.17-1.el7.centos.noarch vdsm-jsonrpc-4.20.17-1.el7.centos.noarch vdsm-hook-ethtool-options-4.20.17-1.el7.centos.noarch vdsm-hook-openstacknet-4.20.17-1.el7.centos.noarch vdsm-http-4.20.17-1.el7.centos.noarch vdsm-client-4.20.17-1.el7.centos.noarch vdsm-gluster-4.20.17-1.el7.centos.noarch vdsm-hook-vfio-mdev-4.20.17-1.el7.centos.noarch vdsm-api-4.20.17-1.el7.centos.noarch vdsm-network-4.20.17-1.el7.centos.x86_64 vdsm-yajsonrpc-4.20.17-1.el7.centos.noarch vdsm-4.20.17-1.el7.centos.x86_64 vdsm-hook-vhostmd-4.20.17-1.el7.centos.noarch
Thanks !!! Enrico
Il 26/06/2018 15:21, Ala Hino ha scritto:
Hi Enrico,
What's the vdsm version that you are using?
The tool introduced in vdsm 4.20.24.
On Tue, Jun 26, 2018 at 3:51 PM, Enrico Becchetti <enrico.becchetti@pg.infn.it <mailto:enrico.becchetti@pg.infn.it>> wrote:
Dear Ala, if you have a few minutes for me I'd like to ask you to read my issue. It's a strange problem because my vm works fine but I can't delete its snapshoot. Thanks a lot Best Regards Enrico
-------- Messaggio Inoltrato -------- Oggetto: [ovirt-users] Re: Cannot acquire Lock .... snapshot error Data: Mon, 25 Jun 2018 14:20:21 +0200 Mittente: Enrico Becchetti <enrico.becchetti@pg.infn.it> <mailto:enrico.becchetti@pg.infn.it> A: Nir Soffer <nsoffer@redhat.com> <mailto:nsoffer@redhat.com> CC: users <users@ovirt.org> <mailto:users@ovirt.org>
Dear Friends , to fix my problem I've try vdsm-tool command but it's seem an error:
[root@infn-vm05 vdsm]# vdsm-tool check-volume-lease Usage: /usr/bin/vdsm-tool [options] <action> [arguments] Valid options: ......
as you can see there isn't check-volumes-option and my ovirt engine is already at 4.2. Any other ideas ? Thanks a lot ! Best Regards Enrico
Il 22/06/2018 17:46, Nir Soffer ha scritto:
On Fri, Jun 22, 2018 at 3:13 PM Enrico Becchetti <enrico.becchetti@pg.infn.it <mailto:enrico.becchetti@pg.infn.it>> wrote:
Dear All, my ovirt 4.2.1.7-1.el7.centos has three hypervisors, lvm storage and virtiual machine with ovirt-engine. All works fine but with one vm when I try to remove its snapshot I have this error:
2018-06-22 07:35:48,155+0200 INFO (jsonrpc/5) [vdsm.api] START prepareMerge(spUUID=u'18d57688-6ed4-43b8-bd7c-0665b55950b7', subchainInfo={u'img_id': u'c5611862-6504-445e-a6c8-f1e1a95b5df7', u'sd_id': u'47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5', u'top_id': u'0e6f7512-871d-4645-b9c6-320ba7e3bee7', u'base_id': u'e156ac2e-09ac-4e1e-a139-17fa374a96d4'}) from=::ffff:10.0.0.46,53304, flow_id=07011450-2296-4a13-a9ed-5d5d2b91be98, task_id=87f95d85-cc3d-4f29-9883-a4dbb3808f88 (api:46) 2018-06-22 07:35:48,406+0200 INFO (tasks/3) [storage.merge] Preparing subchain <SubchainInfo sd_id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, img_id=c5611862-6504-445e-a6c8-f1e1a95b5df7, top_id=0e6f7512-871d-4645-b9c6-320ba7e3bee7, base_id=e156ac2e-09ac-4e1e-a139-17fa374a96d4 base_generation=None at 0x7fcf84ae2510> for merge (merge:177) 2018-06-22 07:35:48,614+0200 INFO (tasks/3) [storage.SANLock] Acquiring Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360) for host id 1 (clusterlock:377) 2018-06-22 07:35:48,634+0200 ERROR (tasks/3) [storage.guarded] Error acquiring lock <VolumeLease ns=04_lease_47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, name=e156ac2e-09ac-4e1e-a139-17fa374a96d4, mode=exclusive at 0x7fcfe09ddf90> (guarded:96) AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')" 2018-06-22 07:35:56,881+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'87f95d85-cc3d-4f29-9883-a4dbb3808f88': {'code': 651, 'message': 'Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name=\'e156ac2e-09ac-4e1e-a139-17fa374a96d4\', path=\'/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases\', offset=115343360), err=(-227, \'Sanlock resource not acquired\', \'Lease resource name is incorrect\')"', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '87f95d85-cc3d-4f29-9883-a4dbb3808f88'}}} from=::ffff:10.0.0.46,53136, task_id=d0e2f4e3-90cb-43c6-aa08-98d1f7efb1bd (api:52)
The issue is corrupted lease for this volume:
AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')"
This root cause is faulty merge code in ovirt < 4.1, creating volume leases with incorrect name. These corrupted leases were not detected until you upgrade to ovirt >= 4.1, because we started to use volume leases for storage operations.
The fix is to run
vdsm-tool check-volume-leases
This will check and repair corrupted leases.
Adding Ala to add more info if needed.
Nir
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it <http://pg.infn.it> ______________________________________________________________________
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it <http://pg.infn.it> ______________________________________________________________________
-- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
The listed leases are broken, there should be no affect on the vm On Wed, Jun 27, 2018, 4:19 PM Enrico Becchetti <enrico.becchetti@pg.infn.it> wrote:
Hi all, after update vdsm I run this command:
*[root@infm-vm04 ~]# vdsm-tool -v check-volume-leases* *WARNING: Make sure there are no running storage operations.*
*Do you want to check volume leases? [yes,NO] yes*
*Checking active storage domains. This can take several minutes, please wait.*
After that I saw some volumes with issue:
*The following volume leases need repair:*
*- domain: 47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5*
* - image: 4b2a6552-847c-43f4-a180-b037d0b93a30* * - volume: 6eb8caf0-b120-40e5-86b5-405f15d1245a* * - image: 871196b2-9d8b-422f-9a3e-be54e100dc5c* * - volume: 861ff7dd-a01c-47f7-8a01-95bc766c2607* * - image: 267b8b8c-da09-44ef-81b3-065dfa2e7085* * - volume: d5f3158a-87ac-4c02-84ba-fcb86b8688a0* * - image: c5611862-6504-445e-a6c8-f1e1a95b5df7* * - volume: e156ac2e-09ac-4e1e-a139-17fa374a96d4* * - image: e765c9c4-2ef9-4c8f-a573-8cd2b0a0a3a2* * - volume: 47d81cbe-598a-402a-9795-1d046c45b7b1* * - image: ab88a08c-910d-44dd-bdf8-8242001ba527* * - volume: 86c72239-525f-4b0b-9aa6-763fc71340bc* * - image: 5e8c4620-b6a5-4fc6-a5bb-f209173d186c* * - volume: 0f52831b-ec35-4140-9a8c-fa24c2647f17* * - image: a67809fc-b830-4ea3-af66-b5b9285b4924* * - volume: 26c6bdd7-1382-4e8e-addc-dcd3898b317f*
*Do you want to repair the leases? [yes,NO] *
what happen If I try to repair them ? Is there any impact to my running vm ?
Thanks a lot !!! Best Regards Enrico
Il 26/06/2018 15:32, Ala Hino ha scritto:
You are running vdsm-4.20.17, and the tool introduced in vdsm-4.20.24. You will have to upgrade vdsm to be able to use the tool.
On Tue, Jun 26, 2018 at 4:29 PM, Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Hi, I run this command from my SPM , Centos 7.4.1708:
[root@infn-vm05 vdsm]# rpm -qa | grep -i vdsm vdsm-hook-vmfex-dev-4.20.17-1.el7.centos.noarch vdsm-python-4.20.17-1.el7.centos.noarch vdsm-hook-fcoe-4.20.17-1.el7.centos.noarch vdsm-common-4.20.17-1.el7.centos.noarch vdsm-jsonrpc-4.20.17-1.el7.centos.noarch vdsm-hook-ethtool-options-4.20.17-1.el7.centos.noarch vdsm-hook-openstacknet-4.20.17-1.el7.centos.noarch vdsm-http-4.20.17-1.el7.centos.noarch vdsm-client-4.20.17-1.el7.centos.noarch vdsm-gluster-4.20.17-1.el7.centos.noarch vdsm-hook-vfio-mdev-4.20.17-1.el7.centos.noarch vdsm-api-4.20.17-1.el7.centos.noarch vdsm-network-4.20.17-1.el7.centos.x86_64 vdsm-yajsonrpc-4.20.17-1.el7.centos.noarch vdsm-4.20.17-1.el7.centos.x86_64 vdsm-hook-vhostmd-4.20.17-1.el7.centos.noarch
Thanks !!! Enrico
Il 26/06/2018 15:21, Ala Hino ha scritto:
Hi Enrico,
What's the vdsm version that you are using?
The tool introduced in vdsm 4.20.24.
On Tue, Jun 26, 2018 at 3:51 PM, Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Dear Ala, if you have a few minutes for me I'd like to ask you to read my issue. It's a strange problem because my vm works fine but I can't delete its snapshoot. Thanks a lot Best Regards Enrico
-------- Messaggio Inoltrato -------- Oggetto: [ovirt-users] Re: Cannot acquire Lock .... snapshot error Data: Mon, 25 Jun 2018 14:20:21 +0200 Mittente: Enrico Becchetti <enrico.becchetti@pg.infn.it> <enrico.becchetti@pg.infn.it> A: Nir Soffer <nsoffer@redhat.com> <nsoffer@redhat.com> CC: users <users@ovirt.org> <users@ovirt.org>
Dear Friends , to fix my problem I've try vdsm-tool command but it's seem an error:
[root@infn-vm05 vdsm]# vdsm-tool check-volume-lease Usage: /usr/bin/vdsm-tool [options] <action> [arguments] Valid options: ......
as you can see there isn't check-volumes-option and my ovirt engine is already at 4.2. Any other ideas ? Thanks a lot ! Best Regards Enrico
Il 22/06/2018 17:46, Nir Soffer ha scritto:
On Fri, Jun 22, 2018 at 3:13 PM Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Dear All, my ovirt 4.2.1.7-1.el7.centos has three hypervisors, lvm storage and virtiual machine with ovirt-engine. All works fine but with one vm when I try to remove its snapshot I have this error:
2018-06-22 07:35:48,155+0200 INFO (jsonrpc/5) [vdsm.api] START prepareMerge(spUUID=u'18d57688-6ed4-43b8-bd7c-0665b55950b7', subchainInfo={u'img_id': u'c5611862-6504-445e-a6c8-f1e1a95b5df7', u'sd_id': u'47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5', u'top_id': u'0e6f7512-871d-4645-b9c6-320ba7e3bee7', u'base_id': u'e156ac2e-09ac-4e1e-a139-17fa374a96d4'}) from=::ffff:10.0.0.46,53304, flow_id=07011450-2296-4a13-a9ed-5d5d2b91be98, task_id=87f95d85-cc3d-4f29-9883-a4dbb3808f88 (api:46) 2018-06-22 07:35:48,406+0200 INFO (tasks/3) [storage.merge] Preparing subchain <SubchainInfo sd_id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, img_id=c5611862-6504-445e-a6c8-f1e1a95b5df7, top_id=0e6f7512-871d-4645-b9c6-320ba7e3bee7, base_id=e156ac2e-09ac-4e1e-a139-17fa374a96d4 base_generation=None at 0x7fcf84ae2510> for merge (merge:177) 2018-06-22 07:35:48,614+0200 INFO (tasks/3) [storage.SANLock] Acquiring Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360) for host id 1 (clusterlock:377) 2018-06-22 07:35:48,634+0200 ERROR (tasks/3) [storage.guarded] Error acquiring lock <VolumeLease ns=04_lease_47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, name=e156ac2e-09ac-4e1e-a139-17fa374a96d4, mode=exclusive at 0x7fcfe09ddf90> (guarded:96) AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')" 2018-06-22 07:35:56,881+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'87f95d85-cc3d-4f29-9883-a4dbb3808f88': {'code': 651, 'message': 'Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name=\'e156ac2e-09ac-4e1e-a139-17fa374a96d4\', path=\'/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases\', offset=115343360), err=(-227, \'Sanlock resource not acquired\', \'Lease resource name is incorrect\')"', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '87f95d85-cc3d-4f29-9883-a4dbb3808f88'}}} from=::ffff:10.0.0.46,53136, task_id=d0e2f4e3-90cb-43c6-aa08-98d1f7efb1bd (api:52)
The issue is corrupted lease for this volume:
AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')"
This root cause is faulty merge code in ovirt < 4.1, creating volume leases with incorrect name. These corrupted leases were not detected until you upgrade to ovirt >= 4.1, because we started to use volume leases for storage operations.
The fix is to run
vdsm-tool check-volume-leases
This will check and repair corrupted leases.
Adding Ala to add more info if needed.
Nir
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
On Wed, Jun 27, 2018 at 4:21 PM Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Hi all, after update vdsm I run this command:
*[root@infm-vm04 ~]# vdsm-tool -v check-volume-leases* *WARNING: Make sure there are no running storage operations.*
*Do you want to check volume leases? [yes,NO] yes*
*Checking active storage domains. This can take several minutes, please wait.*
After that I saw some volumes with issue:
*The following volume leases need repair:*
*- domain: 47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5*
* - image: 4b2a6552-847c-43f4-a180-b037d0b93a30* * - volume: 6eb8caf0-b120-40e5-86b5-405f15d1245a* * - image: 871196b2-9d8b-422f-9a3e-be54e100dc5c* * - volume: 861ff7dd-a01c-47f7-8a01-95bc766c2607* * - image: 267b8b8c-da09-44ef-81b3-065dfa2e7085* * - volume: d5f3158a-87ac-4c02-84ba-fcb86b8688a0* * - image: c5611862-6504-445e-a6c8-f1e1a95b5df7* * - volume: e156ac2e-09ac-4e1e-a139-17fa374a96d4* * - image: e765c9c4-2ef9-4c8f-a573-8cd2b0a0a3a2* * - volume: 47d81cbe-598a-402a-9795-1d046c45b7b1* * - image: ab88a08c-910d-44dd-bdf8-8242001ba527* * - volume: 86c72239-525f-4b0b-9aa6-763fc71340bc* * - image: 5e8c4620-b6a5-4fc6-a5bb-f209173d186c* * - volume: 0f52831b-ec35-4140-9a8c-fa24c2647f17* * - image: a67809fc-b830-4ea3-af66-b5b9285b4924* * - volume: 26c6bdd7-1382-4e8e-addc-dcd3898b317f*
*Do you want to repair the leases? [yes,NO] *
what happen If I try to repair them ? Is there any impact to my running vm ?
This is safe. Each of these volumes has certain area on storage, reserved for the sanlock lease for this volume. The name of the volume e.g. "*6eb8caf0-b120-40e5-86b5-405f15d1245a"* *must appear in that storage area. Sanlock will not allow acquiring* *the lease if the name on storage does not match the name asked by* *the caller.* Repairing the leases will correct the name of the lease, and future operations that aquire the volume lease (e.g. merge snapshots) will succeed. Nir
Thanks a lot !!! Best Regards Enrico
Il 26/06/2018 15:32, Ala Hino ha scritto:
You are running vdsm-4.20.17, and the tool introduced in vdsm-4.20.24. You will have to upgrade vdsm to be able to use the tool.
On Tue, Jun 26, 2018 at 4:29 PM, Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Hi, I run this command from my SPM , Centos 7.4.1708:
[root@infn-vm05 vdsm]# rpm -qa | grep -i vdsm vdsm-hook-vmfex-dev-4.20.17-1.el7.centos.noarch vdsm-python-4.20.17-1.el7.centos.noarch vdsm-hook-fcoe-4.20.17-1.el7.centos.noarch vdsm-common-4.20.17-1.el7.centos.noarch vdsm-jsonrpc-4.20.17-1.el7.centos.noarch vdsm-hook-ethtool-options-4.20.17-1.el7.centos.noarch vdsm-hook-openstacknet-4.20.17-1.el7.centos.noarch vdsm-http-4.20.17-1.el7.centos.noarch vdsm-client-4.20.17-1.el7.centos.noarch vdsm-gluster-4.20.17-1.el7.centos.noarch vdsm-hook-vfio-mdev-4.20.17-1.el7.centos.noarch vdsm-api-4.20.17-1.el7.centos.noarch vdsm-network-4.20.17-1.el7.centos.x86_64 vdsm-yajsonrpc-4.20.17-1.el7.centos.noarch vdsm-4.20.17-1.el7.centos.x86_64 vdsm-hook-vhostmd-4.20.17-1.el7.centos.noarch
Thanks !!! Enrico
Il 26/06/2018 15:21, Ala Hino ha scritto:
Hi Enrico,
What's the vdsm version that you are using?
The tool introduced in vdsm 4.20.24.
On Tue, Jun 26, 2018 at 3:51 PM, Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Dear Ala, if you have a few minutes for me I'd like to ask you to read my issue. It's a strange problem because my vm works fine but I can't delete its snapshoot. Thanks a lot Best Regards Enrico
-------- Messaggio Inoltrato -------- Oggetto: [ovirt-users] Re: Cannot acquire Lock .... snapshot error Data: Mon, 25 Jun 2018 14:20:21 +0200 Mittente: Enrico Becchetti <enrico.becchetti@pg.infn.it> <enrico.becchetti@pg.infn.it> A: Nir Soffer <nsoffer@redhat.com> <nsoffer@redhat.com> CC: users <users@ovirt.org> <users@ovirt.org>
Dear Friends , to fix my problem I've try vdsm-tool command but it's seem an error:
[root@infn-vm05 vdsm]# vdsm-tool check-volume-lease Usage: /usr/bin/vdsm-tool [options] <action> [arguments] Valid options: ......
as you can see there isn't check-volumes-option and my ovirt engine is already at 4.2. Any other ideas ? Thanks a lot ! Best Regards Enrico
Il 22/06/2018 17:46, Nir Soffer ha scritto:
On Fri, Jun 22, 2018 at 3:13 PM Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Dear All, my ovirt 4.2.1.7-1.el7.centos has three hypervisors, lvm storage and virtiual machine with ovirt-engine. All works fine but with one vm when I try to remove its snapshot I have this error:
2018-06-22 07:35:48,155+0200 INFO (jsonrpc/5) [vdsm.api] START prepareMerge(spUUID=u'18d57688-6ed4-43b8-bd7c-0665b55950b7', subchainInfo={u'img_id': u'c5611862-6504-445e-a6c8-f1e1a95b5df7', u'sd_id': u'47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5', u'top_id': u'0e6f7512-871d-4645-b9c6-320ba7e3bee7', u'base_id': u'e156ac2e-09ac-4e1e-a139-17fa374a96d4'}) from=::ffff:10.0.0.46,53304, flow_id=07011450-2296-4a13-a9ed-5d5d2b91be98, task_id=87f95d85-cc3d-4f29-9883-a4dbb3808f88 (api:46) 2018-06-22 07:35:48,406+0200 INFO (tasks/3) [storage.merge] Preparing subchain <SubchainInfo sd_id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, img_id=c5611862-6504-445e-a6c8-f1e1a95b5df7, top_id=0e6f7512-871d-4645-b9c6-320ba7e3bee7, base_id=e156ac2e-09ac-4e1e-a139-17fa374a96d4 base_generation=None at 0x7fcf84ae2510> for merge (merge:177) 2018-06-22 07:35:48,614+0200 INFO (tasks/3) [storage.SANLock] Acquiring Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360) for host id 1 (clusterlock:377) 2018-06-22 07:35:48,634+0200 ERROR (tasks/3) [storage.guarded] Error acquiring lock <VolumeLease ns=04_lease_47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, name=e156ac2e-09ac-4e1e-a139-17fa374a96d4, mode=exclusive at 0x7fcfe09ddf90> (guarded:96) AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')" 2018-06-22 07:35:56,881+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'87f95d85-cc3d-4f29-9883-a4dbb3808f88': {'code': 651, 'message': 'Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name=\'e156ac2e-09ac-4e1e-a139-17fa374a96d4\', path=\'/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases\', offset=115343360), err=(-227, \'Sanlock resource not acquired\', \'Lease resource name is incorrect\')"', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '87f95d85-cc3d-4f29-9883-a4dbb3808f88'}}} from=::ffff:10.0.0.46,53136, task_id=d0e2f4e3-90cb-43c6-aa08-98d1f7efb1bd (api:52)
The issue is corrupted lease for this volume:
AcquireLockFailure: Cannot obtain lock: "id=47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5, rc=-227, out=Cannot acquire Lease(name='e156ac2e-09ac-4e1e-a139-17fa374a96d4', path='/dev/47b7c9aa-ef53-48bc-bb55-4a1a0ba5c8d5/leases', offset=115343360), err=(-227, 'Sanlock resource not acquired', 'Lease resource name is incorrect')"
This root cause is faulty merge code in ovirt < 4.1, creating volume leases with incorrect name. These corrupted leases were not detected until you upgrade to ovirt >= 4.1, because we started to use volume leases for storage operations.
The fix is to run
vdsm-tool check-volume-leases
This will check and repair corrupted leases.
Adding Ala to add more info if needed.
Nir
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 <+39%20075%20585%202777> Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 <+39%20075%20585%202777> Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 <+39%20075%20585%202777> Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/MOLONDHLH7B6WA...
participants (3)
-
Ala Hino -
Enrico Becchetti -
Nir Soffer