Setting Up oVirt + NFS Storage Issues
Hello! Been looking to get setup with oVirt for a few weeks and had a chance the past week or so to attempt getting it all setup. Ended up doing a bit of troubleshooting but finally got to the point where the Cockpit setup prompts for me setting up the storage piece, for which I've opted for NFS (path of least resistance for now). However, I constantly run into this issue (pulled from the administration portal events page): VDSM asgard.shackfam.net command CreateStorageDomainVDS failed: Could not initialize cluster lock: () This specifically shows up in the following task for the Ansible piece: [ovirt.hosted_engine_setup : Add NFS storage domain] When I browse the NFS share, I do see a file structure being created, which kind of confuses me based on the error provide in the Administration portal. Any suggestions on what would be best steps forward at this point? For reference, here's my setup: 1x oVirt Node 1x NFS server (storage01: NFS shares mainly managed by ZFS)
On Mon, Aug 3, 2020 at 4:06 PM Arden Shackelford <zuntaruk@librem.one> wrote:
Hello!
Been looking to get setup with oVirt for a few weeks and had a chance the past week or so to attempt getting it all setup. Ended up doing a bit of troubleshooting but finally got to the point where the Cockpit setup prompts for me setting up the storage piece, for which I've opted for NFS (path of least resistance for now). However, I constantly run into this issue (pulled from the administration portal events page):
Are your NFS exports permissions set correctly (probably yes if you can see something created on your share)? https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm...
VDSM asgard.shackfam.net command CreateStorageDomainVDS failed: Could not initialize cluster lock: ()
Can you share the full error trace from vdsm.log ?
This specifically shows up in the following task for the Ansible piece:
[ovirt.hosted_engine_setup : Add NFS storage domain]
When I browse the NFS share, I do see a file structure being created, which kind of confuses me based on the error provide in the Administration portal.
Can you list your share contents with ls -lhZ ?
Any suggestions on what would be best steps forward at this point?
For reference, here's my setup:
1x oVirt Node 1x NFS server (storage01: NFS shares mainly managed by ZFS) _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/BNVXUH5B26FBFC...
Hey Amit, Thanks for the response. Here's what I've got:
Are your NFS exports permissions set correctly (probably yes if you can see something created on your share)?
Here's the perms on the folder (/mnt/ovirt on NFS server): File: ovirt Size: 2 Blocks: 1 IO Block: 512 directory Device: 33h/51d Inode: 34 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 36/ vdsm) Gid: ( 36/kvmovirt) Access: 2020-08-04 17:54:06.971018988 +0000 Modify: 2020-08-04 17:54:09.410982092 +0000 Change: 2020-08-04 17:54:09.410982092 +0000 Birth: -
Can you list your share contents with ls -lhZ?
Root of share: drwxrwxr-x 4 vdsm kvmovirt ? 4 Aug 4 18:52 c6244268-aaeb-4b67-b12b-8a0e81d7c205 Inside of that: drwxrwxr-x 2 vdsm kvmovirt ? 8 Aug 4 18:52 dom_md drwxrwxr-x 2 vdsm kvmovirt ? 2 Aug 4 18:52 images
Can you share the full error trace from vdsm.log?
Here's what I see: 2020-08-04 13:52:58,695-0500 INFO (jsonrpc/1) [storage.xlease] Formatting index for lockspace 'c6244268-aaeb-4b67-b12b-8a0e81d7c205' (version=1) (xlease:701) 2020-08-04 13:52:58,767-0500 INFO (jsonrpc/1) [storage.StorageDomain] Creating domain images directory '/rhev/data-center/mnt/storage01:_mnt_ovirt/c6244268-aaeb-4b67-b12b-8a0e81d7c205/images' (nfsSD:110) 2020-08-04 13:52:58,817-0500 INFO (jsonrpc/1) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:735) 2020-08-04 13:52:58,819-0500 ERROR (jsonrpc/1) [storage.initSANLock] Cannot initialize SANLock for domain c6244268-aaeb-4b67-b12b-8a0e81d7c205 (clusterlock:259) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (19, 'Sanlock lockspace write failure', 'No such device') 2020-08-04 13:52:58,819-0500 INFO (jsonrpc/1) [vdsm.api] FINISH createStorageDomain error=Could not initialize cluster lock: () from=::ffff:192.168.222.200,37322, flow_id=228288c7, task_id=6bb381fd-9620-46bd-8c9a-0c6206e2cea4 (api:52) 2020-08-04 13:52:58,819-0500 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='6bb381fd-9620-46bd-8c9a-0c6206e2cea4') Unexpected error (task:880) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (19, 'Sanlock lockspace write failure', 'No such device') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in _run return fn(*args, **kargs) File "<decorator-gen-121>", line 2, in createStorageDomain File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2644, in createStorageDomain max_hosts=max_hosts) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120, in create fsd.initSPMlease() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 999, in initSPMlease return self._manifest.initDomainLock() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 627, in initDomainLock self._domainLock.initLock(self.getDomainLease()) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 308, in initLock block_size=self._block_size) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 260, in initSANLock raise se.ClusterLockInitError() vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster lock: () 2020-08-04 13:52:58,819-0500 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='6bb381fd-9620-46bd-8c9a-0c6206e2cea4') aborting: Task is aborted: 'value=Could not initialize cluster lock: () abortedcode=701' (task:1190) 2020-08-04 13:52:58,819-0500 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH createStorageDomain error=Could not initialize cluster lock: () (dispatcher:83) 2020-08-04 13:52:58,819-0500 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StorageDomain.create failed (error 701) in 0.79 seconds (__init__:312) 2020-08-04 13:52:58,918-0500 INFO (jsonrpc/7) [vdsm.api] START disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password': '********', 'protocol_version': 'auto', 'port': '', 'iqn': '', 'connection': 'storage01:/mnt/ovirt', 'ipv6_enabled':'false', 'id': '49d6da66-aecb-442e-9852-4c91f6440182', 'user': '', 'tpgt': '1'}], options=None) from=::ffff:192.168.222.200,37322, flow_id=55c0cda3-5d44-4cd8-8020-47893c09ad5f, task_id=c2ecd861-39d7-4dc0-aca2-938794adefcb (api:48) 2020-08-04 13:52:58,918-0500 INFO (jsonrpc/7) [storage.Mount] unmounting /rhev/data-center/mnt/storage01:_mnt_ovirt (mount:215) 2020-08-04 13:52:58,961-0500 INFO (jsonrpc/7) [storage.StorageDomainCache] Refreshing storage domain cache (resize=False) (sdc:80) 2020-08-04 13:52:58,961-0500 INFO (jsonrpc/7) [storage.ISCSI] Scanning iSCSI devices (iscsi:442) 2020-08-04 13:52:58,986-0500 INFO (jsonrpc/7) [storage.ISCSI] Scanning iSCSI devices: 0.02 seconds (utils:390) 2020-08-04 13:52:58,986-0500 INFO (jsonrpc/7) [storage.HBA] Scanning FC devices (hba:60) 2020-08-04 13:52:58,988-0500 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.222.200,37322 (api:48) 2020-08-04 13:52:58,989-0500 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.222.200,37322 (api:54) 2020-08-04 13:52:58,989-0500 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2020-08-04 13:52:59,038-0500 INFO (jsonrpc/7) [storage.HBA] Scanning FC devices: 0.05 seconds (utils:390) 2020-08-04 13:52:59,049-0500 INFO (jsonrpc/7) [storage.StorageDomainCache] Refreshing storage domain cache: 0.09 seconds (utils:390) 2020-08-04 13:52:59,049-0500 INFO (jsonrpc/7) [vdsm.api] FINISH disconnectStorageServer return={'statuslist': [{'id': '49d6da66-aecb-442e-9852-4c91f6440182', 'status': 0}]} from=::ffff:192.168.222.200,37322, flow_id=55c0cda3-5d44-4cd8-8020-47893c09ad5f, task_id=c2ecd861-39d7-4dc0-aca2-938794adefcb (api:54)
On Tue, Aug 4, 2020 at 10:02 PM Arden Shackelford <zuntaruk@librem.one> wrote:
Hey Amit,
Thanks for the response. Here's what I've got:
Are your NFS exports permissions set correctly (probably yes if you can see something created on your share)?
Here's the perms on the folder (/mnt/ovirt on NFS server):
File: ovirt Size: 2 Blocks: 1 IO Block: 512 directory Device: 33h/51d Inode: 34 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 36/ vdsm) Gid: ( 36/kvmovirt) Access: 2020-08-04 17:54:06.971018988 +0000 Modify: 2020-08-04 17:54:09.410982092 +0000 Change: 2020-08-04 17:54:09.410982092 +0000 Birth: -
Can you list your share contents with ls -lhZ?
Root of share:
drwxrwxr-x 4 vdsm kvmovirt ? 4 Aug 4 18:52 c6244268-aaeb-4b67-b12b-8a0e81d7c205
I think selinux is not supported on your file system. Here is a working setup: On the host: # ls -lhZ /rhev/data-center/mnt/nfs1\:_export_1/ total 0 drwxr-xr-x. 5 vdsm kvm system_u:object_r:nfs_t:s0 48 Jun 20 03:56 56ecc03c-4bb5-4792-8971-3c51ea924d2e On the server: # ls -lhZ /export/1 total 0 drwxr-xr-x. 5 vdsm kvm system_u:object_r:unlabeled_t:s0 48 Jun 20 03:56 56ecc03c-4bb5-4792-8971-3c51ea924d2e
Inside of that:
drwxrwxr-x 2 vdsm kvmovirt ? 8 Aug 4 18:52 dom_md drwxrwxr-x 2 vdsm kvmovirt ? 2 Aug 4 18:52 images
Can you share the full error trace from vdsm.log?
Here's what I see: ... 2020-08-04 13:52:58,819-0500 ERROR (jsonrpc/1) [storage.initSANLock] Cannot initialize SANLock for domain c6244268-aaeb-4b67-b12b-8a0e81d7c205 (clusterlock:259) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (19, 'Sanlock lockspace write failure', 'No such device')
I think this is this bug: https://bugzilla.redhat.com/1778485 Sanlok gets EACCESS - probably because selinux is disabled on the server, or not supported with ZFS, and then it returns ENODEV. Unfortunately this was not fixed yet in sanlock. To fix this, try to get selinux working on your server, or use another filesystem instead of ZFS. ZFS is not tested with oVirt, so while it may have great features, you may have even more trouble later even if you can get it working. Nir
Dang! Missed the part of ZFS not being supported :( Yeah, NFS server is running Ubuntu, so not there by default. I'll see what I can do :)
Just for reference in case anyone else comes across this, I opted to take the path of using GlusterFS in place of the NFS storage server that I've got to avoid having to make changes to the NFS server. The glusterfs was setup on the same node, which is actually my preferred way to go for this. Anyway, with the switch to gluster I'm up and running! Very excited to start playing with oVirt! Thanks for the responses here to help me get a better understanding of how this stuff works!
participants (3)
-
Amit Bawer -
Arden Shackelford -
Nir Soffer