4:55 p.m.
Hello!
Been looking to get setup with oVirt for a few weeks and had a chance the past week or so
to attempt getting it all setup. Ended up doing a bit of troubleshooting but finally got
to the point where the Cockpit setup prompts for me setting up the storage piece, for
which I've opted for NFS (path of least resistance for now). However, I constantly run
into this issue (pulled from the administration portal events page):
VDSM asgard.shackfam.net command CreateStorageDomainVDS failed: Could not initialize
cluster lock: ()
This specifically shows up in the following task for the Ansible piece:
[ovirt.hosted_engine_setup : Add NFS storage domain]
When I browse the NFS share, I do see a file structure being created, which kind of
confuses me based on the error provide in the Administration portal.
Any suggestions on what would be best steps forward at this point?
For reference, here's my setup:
1x oVirt Node
1x NFS server (storage01: NFS shares mainly managed by ZFS)
9:17 a.m.
On Mon, Aug 3, 2020 at 4:06 PM Arden Shackelford <zuntaruk(a)librem.one>
wrote:
Hello!
Been looking to get setup with oVirt for a few weeks and had a chance the
past week or so to attempt getting it all setup. Ended up doing a bit of
troubleshooting but finally got to the point where the Cockpit setup
prompts for me setting up the storage piece, for which I've opted for NFS
(path of least resistance for now). However, I constantly run into this
issue (pulled from the administration portal events page):
Are your NFS exports permissions set correctly (probably yes if you can see
something created on your share)?
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/...
VDSM asgard.shackfam.net command CreateStorageDomainVDS failed: Could not
initialize cluster lock: ()
Can you share the full error trace from vdsm.log ?
This specifically shows up in the following task for the Ansible piece:
[ovirt.hosted_engine_setup : Add NFS storage domain]
When I browse the NFS share, I do see a file structure being created,
which kind of confuses me based on the error provide in the Administration
portal.
Can you list your share contents with ls -lhZ ?
Any suggestions on what would be best steps forward at this point?
For reference, here's my setup:
1x oVirt Node
1x NFS server (storage01: NFS shares mainly managed by ZFS)
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BNVXUH5B26F...
2:59 p.m.
Hey Amit,
Thanks for the response. Here's what I've got:
Are your NFS exports permissions set correctly (probably yes if you
can see
something created on your share)?
Here's the perms on the folder (/mnt/ovirt on NFS server):
File: ovirt
Size: 2 Blocks: 1 IO Block: 512 directory
Device: 33h/51d Inode: 34 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 36/ vdsm) Gid: ( 36/kvmovirt)
Access: 2020-08-04 17:54:06.971018988 +0000
Modify: 2020-08-04 17:54:09.410982092 +0000
Change: 2020-08-04 17:54:09.410982092 +0000
Birth: -
Can you list your share contents with ls -lhZ?
Root of share:
drwxrwxr-x 4 vdsm kvmovirt ? 4 Aug 4 18:52 c6244268-aaeb-4b67-b12b-8a0e81d7c205
Inside of that:
drwxrwxr-x 2 vdsm kvmovirt ? 8 Aug 4 18:52 dom_md
drwxrwxr-x 2 vdsm kvmovirt ? 2 Aug 4 18:52 images
Can you share the full error trace from vdsm.log?
Here's what I see:
2020-08-04 13:52:58,695-0500 INFO (jsonrpc/1) [storage.xlease] Formatting index for
lockspace 'c6244268-aaeb-4b67-b12b-8a0e81d7c205' (version=1) (xlease:701)
2020-08-04 13:52:58,767-0500 INFO (jsonrpc/1) [storage.StorageDomain] Creating domain
images directory
'/rhev/data-center/mnt/storage01:_mnt_ovirt/c6244268-aaeb-4b67-b12b-8a0e81d7c205/images'
(nfsSD:110)
2020-08-04 13:52:58,817-0500 INFO (jsonrpc/1) [storage.StorageDomain] Removing remnants
of deleted images [] (fileSD:735)
2020-08-04 13:52:58,819-0500 ERROR (jsonrpc/1) [storage.initSANLock] Cannot initialize
SANLock for domain c6244268-aaeb-4b67-b12b-8a0e81d7c205 (clusterlock:259)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250,
in initSANLock
lockspace_name, idsPath, align=alignment, sector=block_size)
sanlock.SanlockException: (19, 'Sanlock lockspace write failure', 'No such
device')
2020-08-04 13:52:58,819-0500 INFO (jsonrpc/1) [vdsm.api] FINISH createStorageDomain
error=Could not initialize cluster lock: () from=::ffff:192.168.222.200,37322,
flow_id=228288c7, task_id=6bb381fd-9620-46bd-8c9a-0c6206e2cea4 (api:52)
2020-08-04 13:52:58,819-0500 ERROR (jsonrpc/1) [storage.TaskManager.Task]
(Task='6bb381fd-9620-46bd-8c9a-0c6206e2cea4') Unexpected error (task:880)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250,
in initSANLock
lockspace_name, idsPath, align=alignment, sector=block_size)
sanlock.SanlockException: (19, 'Sanlock lockspace write failure', 'No such
device')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in _run
return fn(*args, **kargs)
File "<decorator-gen-121>", line 2, in createStorageDomain
File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2644, in
createStorageDomain
max_hosts=max_hosts)
File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120, in
create
fsd.initSPMlease()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 999, in
initSPMlease
return self._manifest.initDomainLock()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 627, in
initDomainLock
self._domainLock.initLock(self.getDomainLease())
File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 308,
in initLock
block_size=self._block_size)
File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 260,
in initSANLock
raise se.ClusterLockInitError()
vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster lock: ()
2020-08-04 13:52:58,819-0500 INFO (jsonrpc/1) [storage.TaskManager.Task]
(Task='6bb381fd-9620-46bd-8c9a-0c6206e2cea4') aborting: Task is aborted:
'value=Could not initialize cluster lock: () abortedcode=701' (task:1190)
2020-08-04 13:52:58,819-0500 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH
createStorageDomain error=Could not initialize cluster lock: () (dispatcher:83)
2020-08-04 13:52:58,819-0500 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call
StorageDomain.create failed (error 701) in 0.79 seconds (__init__:312)
2020-08-04 13:52:58,918-0500 INFO (jsonrpc/7) [vdsm.api] START
disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000',
conList=[{'password': '********', 'protocol_version':
'auto', 'port': '', 'iqn': '',
'connection': 'storage01:/mnt/ovirt',
'ipv6_enabled':'false', 'id':
'49d6da66-aecb-442e-9852-4c91f6440182', 'user': '',
'tpgt': '1'}], options=None) from=::ffff:192.168.222.200,37322,
flow_id=55c0cda3-5d44-4cd8-8020-47893c09ad5f, task_id=c2ecd861-39d7-4dc0-aca2-938794adefcb
(api:48)
2020-08-04 13:52:58,918-0500 INFO (jsonrpc/7) [storage.Mount] unmounting
/rhev/data-center/mnt/storage01:_mnt_ovirt (mount:215)
2020-08-04 13:52:58,961-0500 INFO (jsonrpc/7) [storage.StorageDomainCache] Refreshing
storage domain cache (resize=False) (sdc:80)
2020-08-04 13:52:58,961-0500 INFO (jsonrpc/7) [storage.ISCSI] Scanning iSCSI devices
(iscsi:442)
2020-08-04 13:52:58,986-0500 INFO (jsonrpc/7) [storage.ISCSI] Scanning iSCSI devices:
0.02 seconds (utils:390)
2020-08-04 13:52:58,986-0500 INFO (jsonrpc/7) [storage.HBA] Scanning FC devices (hba:60)
2020-08-04 13:52:58,988-0500 INFO (jsonrpc/4) [api.host] START getAllVmStats()
from=::ffff:192.168.222.200,37322 (api:48)
2020-08-04 13:52:58,989-0500 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats
return={'status': {'code': 0, 'message': 'Done'},
'statsList': (suppressed)} from=::ffff:192.168.222.200,37322 (api:54)
2020-08-04 13:52:58,989-0500 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
Host.getAllVmStats succeeded in 0.00 seconds (__init__:312)
2020-08-04 13:52:59,038-0500 INFO (jsonrpc/7) [storage.HBA] Scanning FC devices: 0.05
seconds (utils:390)
2020-08-04 13:52:59,049-0500 INFO (jsonrpc/7) [storage.StorageDomainCache] Refreshing
storage domain cache: 0.09 seconds (utils:390)
2020-08-04 13:52:59,049-0500 INFO (jsonrpc/7) [vdsm.api] FINISH disconnectStorageServer
return={'statuslist': [{'id':
'49d6da66-aecb-442e-9852-4c91f6440182', 'status': 0}]}
from=::ffff:192.168.222.200,37322, flow_id=55c0cda3-5d44-4cd8-8020-47893c09ad5f,
task_id=c2ecd861-39d7-4dc0-aca2-938794adefcb (api:54)
5:26 p.m.
On Tue, Aug 4, 2020 at 10:02 PM Arden Shackelford <zuntaruk(a)librem.one> wrote:
Hey Amit,
Thanks for the response. Here's what I've got:
>Are your NFS exports permissions set correctly (probably yes if you can see
something created on your share)?
Here's the perms on the folder (/mnt/ovirt on NFS server):
File: ovirt
Size: 2 Blocks: 1 IO Block: 512 directory
Device: 33h/51d Inode: 34 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 36/ vdsm) Gid: ( 36/kvmovirt)
Access: 2020-08-04 17:54:06.971018988 +0000
Modify: 2020-08-04 17:54:09.410982092 +0000
Change: 2020-08-04 17:54:09.410982092 +0000
Birth: -
>Can you list your share contents with ls -lhZ?
Root of share:
drwxrwxr-x 4 vdsm kvmovirt ? 4 Aug 4 18:52 c6244268-aaeb-4b67-b12b-8a0e81d7c205
I think selinux is not supported on your file system. Here is a working setup:
On the host:
# ls -lhZ /rhev/data-center/mnt/nfs1\:_export_1/
total 0
drwxr-xr-x. 5 vdsm kvm system_u:object_r:nfs_t:s0 48 Jun 20 03:56
56ecc03c-4bb5-4792-8971-3c51ea924d2e
On the server:
# ls -lhZ /export/1
total 0
drwxr-xr-x. 5 vdsm kvm system_u:object_r:unlabeled_t:s0 48 Jun 20
03:56 56ecc03c-4bb5-4792-8971-3c51ea924d2e
Inside of that:
drwxrwxr-x 2 vdsm kvmovirt ? 8 Aug 4 18:52 dom_md
drwxrwxr-x 2 vdsm kvmovirt ? 2 Aug 4 18:52 images
> Can you share the full error trace from vdsm.log?
Here's what I see:
...
2020-08-04 13:52:58,819-0500 ERROR (jsonrpc/1) [storage.initSANLock]
Cannot initialize SANLock for domain c6244268-aaeb-4b67-b12b-8a0e81d7c205
(clusterlock:259)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250,
in initSANLock
lockspace_name, idsPath, align=alignment, sector=block_size)
sanlock.SanlockException: (19, 'Sanlock lockspace write failure', 'No such
device')
I think this is this bug:
https://bugzilla.redhat.com/1778485
Sanlok gets EACCESS - probably because selinux is disabled on the server,
or not supported with ZFS, and then it returns ENODEV.
Unfortunately this was not fixed yet in sanlock.
To fix this, try to get selinux working on your server, or use another
filesystem
instead of ZFS. ZFS is not tested with oVirt, so while it may have
great features,
you may have even more trouble later even if you can get it working.
Nir
5:44 p.m.
Dang! Missed the part of ZFS not being supported :(
Yeah, NFS server is running Ubuntu, so not there by default. I'll see what I can do :)
11:41 p.m.
Just for reference in case anyone else comes across this, I opted to take the path of
using GlusterFS in place of the NFS storage server that I've got to avoid having to
make changes to the NFS server.
The glusterfs was setup on the same node, which is actually my preferred way to go for
this.
Anyway, with the switch to gluster I'm up and running! Very excited to start playing
with oVirt!
Thanks for the responses here to help me get a better understanding of how this stuff
works!
1601
days inactive
1606
days old
5 comments
3 participants
participants (3)
-
Amit Bawer -
Arden Shackelford -
Nir Soffer