[ovirt-devel] Re: Unexpected exception when trying to add a storage domain

On Thu, Jan 2, 2020 at 6:47 PM Dana Elfassy <delfassy(a)redhat.com&gt; wrote: > > Thanks, > I can mount it manually, but when trying to read the file, I'm getting the error: /usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py: No such file or directory > The directory's user & group are on my user (I'm unable to change ownership for some reason), but I have set earlier the directory's (which I tried to add as a storage domain) permissions to 777 This does sound like a healthy storage domain configuration. The way to create a storage domain is: In /etc/exports add entry like: $ cat /etc/exports /export/1 *(rw,async,anonuid=36,anongid=36) /export/2 *(rw,async,anonuid=36,anongid=36) Note: async is not safe for production, I'm using it to simulate a fast nfs server. You probably want to use sync which is the default. Reload nfs server configuration: # exportfs -r If you check the export you will see: # exportfs -v /export/1 <world>(async,wdelay,hide,no_subtree_check,anonuid=36,anongid=36,sec=sys,rw,secure,root_squash,no_all_squash) /export/2 <world>(async,wdelay,hide,no_subtree_check,anonuid=36,anongid=36,sec=sys,rw,secure,root_squash,no_all_squash) Then change the owner of the directory to vdsm:kvm (you many need to add a vdsm user on the nfs server) # chown -R vdsm:kvm /path/to/export On the host, the mount should look like: # mount | grep nfs1 nfs1:/export/2 on /rhev/data-center/mnt/nfs1:_export_2 type nfs4 (rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.122.20,local_lock=none,addr=192.168.122.30) (I'm using NFS 4.2 which is much faster, you should use it too) # tree -pug /rhev/data-center/mnt/nfs1\:_export_2/ /rhev/data-center/mnt/nfs1:_export_2/ └── [drwxr-xr-x vdsm kvm ] 55255570-983a-4d82-907a-19b964abf7ed ├── [drwxr-xr-x vdsm kvm ] dom_md │ ├── [-rw-rw---- vdsm kvm ] ids │ ├── [-rw-rw---- vdsm kvm ] inbox │ ├── [-rw-rw---- vdsm kvm ] leases │ ├── [-rw-r--r-- vdsm kvm ] metadata │ ├── [-rw-rw---- vdsm kvm ] outbox │ └── [-rw-rw---- vdsm kvm ] xleases ├── [drwxr-xr-x vdsm kvm ] images │ ├── [drwxr-xr-x vdsm kvm ] 17b81130-f4a4-4764-ab5a-258a803c7706 │ │ ├── [-rw-rw---- vdsm kvm ] 66738f70-bb6c-4ce0-a98e-c7109d3b4275 │ │ ├── [-rw-rw---- vdsm kvm ] 66738f70-bb6c-4ce0-a98e-c7109d3b4275.lease │ │ └── [-rw-r--r-- vdsm kvm ] 66738f70-bb6c-4ce0-a98e-c7109d3b4275.meta │ ├── [drwxr-xr-x vdsm kvm ] 6fd5eafc-572b-4335-b59b-42282ba10464 │ │ ├── [-rw-rw---- vdsm kvm ] 46e25ec2-04ae-4829-b32d-972c1e15da09 │ │ ├── [-rw-rw---- vdsm kvm ] 46e25ec2-04ae-4829-b32d-972c1e15da09.lease │ │ └── [-rw-r--r-- vdsm kvm ] 46e25ec2-04ae-4829-b32d-972c1e15da09.meta │ ├── [drwxr-xr-x vdsm kvm ] b2cdc744-9992-4a97-8d80-37c40c5b8f84 │ │ ├── [-rw-rw---- vdsm kvm ] 12b58c9b-ccd7-4375-a0df-fc956075f313 │ │ ├── [-rw-rw---- vdsm kvm ] 12b58c9b-ccd7-4375-a0df-fc956075f313.lease │ │ └── [-rw-r--r-- vdsm kvm ] 12b58c9b-ccd7-4375-a0df-fc956075f313.meta │ └── [drwxr-xr-x vdsm kvm ] e95c50b2-d066-499b-9c58-6a479b50e515 │ ├── [-rw-rw---- vdsm kvm ] 69435001-9c25-41fd-85d3-7f7321a503e6 │ ├── [-rw-rw---- vdsm kvm ] 69435001-9c25-41fd-85d3-7f7321a503e6.lease │ └── [-rw-r--r-- vdsm kvm ] 69435001-9c25-41fd-85d3-7f7321a503e6.meta └── [drwxr-xr-x vdsm kvm ] master ├── [drwxr-xr-x vdsm kvm ] tasks └── [drwxr-xr-x vdsm kvm ] vms > On Thu, Jan 2, 2020 at 5:24 PM Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> >> On Thu, Jan 2, 2020 at 4:44 PM Dana Elfassy <delfassy(a)redhat.com&gt; wrote: >> > >> > Hi, >> > When trying to add a storage domain to a 4.4 host I'm getting this error message: >> > Error while executing action New NFS Storage Domain: Unexpected exception >> > >> > The errors from vdsm.log: >> > 2020-01-02 09:38:33,578-0500 ERROR (jsonrpc/0) [storage.initSANLock] Cannot initialize SANLock for domain 6ca1e203-5595-47e5-94b8-82a7e69d99a9 (clusterlock:259) >> > Traceback (most recent call last): >> > File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock >> > lockspace_name, idsPath, align=alignment, sector=block_size) >> > sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout') >> >> This means sanlock operation failed with a timeout - the storage was too slow >> or there some issue on the storage side. >> >> Can you mount this manually and access file as user vdsm? >> >> > 2020-01-02 09:38:33,579-0500 INFO (jsonrpc/0) [vdsm.api] FINISH createStorageDomain error=Could not initialize cluster lock: () from=::ffff:192.168.100.1,36452, flow_id=17dc614 >> > e, task_id=05c2107a-4d59-48d0-a2f7-0938f051c9ab (api:52) >> > 2020-01-02 09:38:33,582-0500 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='05c2107a-4d59-48d0-a2f7-0938f051c9ab') Unexpected error (task:874) >> > Traceback (most recent call last): >> > File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock >> > lockspace_name, idsPath, align=alignment, sector=block_size) >> > sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout') >> >> Looks like same error raised again, a common issue in legacy code. >> >> > During handling of the above exception, another exception occurred: >> > >> > Traceback (most recent call last): >> > File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 881, in _run >> > return fn(*args, **kargs) >> > File "<decorator-gen-121>", line 2, in createStorageDomain >> > File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method >> > ret = func(*args, **kwargs) >> > File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2622, in createStorageDomain >> > max_hosts=max_hosts) >> > File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120, in create >> > fsd.initSPMlease() >> > File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 974, in initSPMlease >> > return self._manifest.initDomainLock() >> > File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 620, in initDomainLock >> > self._domainLock.initLock(self.getDomainLease()) >> > File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 308, in initLock >> > block_size=self._block_size) >> > File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 260, in initSANLock >> > raise se.ClusterLockInitError() >> > vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster lock: () >> >> There is no information about this error, looks like the public >> useless error we raise for the >> underlying error seen before. >>