Unexpected exception when trying to add a storage domain
Hi, When trying to add a storage domain to a 4.4 host I'm getting this error message: Error while executing action New NFS Storage Domain: Unexpected exception The errors from vdsm.log: 2020-01-02 09:38:33,578-0500 ERROR (jsonrpc/0) [storage.initSANLock] Cannot initialize SANLock for domain 6ca1e203-5595-47e5-94b8-82a7e69d99a9 (clusterlock:259) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout') 2020-01-02 09:38:33,579-0500 INFO (jsonrpc/0) [vdsm.api] FINISH createStorageDomain error=Could not initialize cluster lock: () from=::ffff:192.168.100.1,36452, flow_id=17dc614 e, task_id=05c2107a-4d59-48d0-a2f7-0938f051c9ab (api:52) 2020-01-02 09:38:33,582-0500 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='05c2107a-4d59-48d0-a2f7-0938f051c9ab') Unexpected error (task:874) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 881, in _run return fn(*args, **kargs) File "<decorator-gen-121>", line 2, in createStorageDomain File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2622, in createStorageDomain max_hosts=max_hosts) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120, in create fsd.initSPMlease() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 974, in initSPMlease return self._manifest.initDomainLock() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 620, in initDomainLock self._domainLock.initLock(self.getDomainLease()) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 308, in initLock block_size=self._block_size) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 260, in initSANLock raise se.ClusterLockInitError() vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster lock: () 2020-01-02 09:38:33,583-0500 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='05c2107a-4d59-48d0-a2f7-0938f051c9ab') aborting: Task is aborted: 'value=Could not initialize cluster lock: () abortedcode=701' (task:1184) 2020-01-02 09:38:33,584-0500 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH createStorageDomain error=Could not initialize cluster lock: () (dispatcher:83) 2020-01-02 09:38:33,584-0500 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StorageDomain.create failed (error 701) in 18.70 seconds (__init__:312) 2020-01-02 09:38:33,730-0500 INFO (jsonrpc/2) [vdsm.api] START disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password': '********', 'protocol_version': 'auto', 'port': '', 'iqn': '', 'connection': 'vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/nfs1', 'ipv6_enabled': 'false', 'id': 'bf075967-060b-4e60-9b0b-bb170fe073f9', 'user': '', 'tpgt': '1'}], options=None) from=::ffff:192.168.100.1,36452, flow_id=f4f79738-b361-4688-95c5-3454f52b505d, task_id=dd7266ed-69ec-4e5b-9216-850d9db8ea3b (api:48) Can someone help me with it? Thanks, Dana
On Thu, Jan 2, 2020 at 4:44 PM Dana Elfassy <delfassy@redhat.com> wrote:
Hi, When trying to add a storage domain to a 4.4 host I'm getting this error message: Error while executing action New NFS Storage Domain: Unexpected exception
The errors from vdsm.log: 2020-01-02 09:38:33,578-0500 ERROR (jsonrpc/0) [storage.initSANLock] Cannot initialize SANLock for domain 6ca1e203-5595-47e5-94b8-82a7e69d99a9 (clusterlock:259) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout')
This means sanlock operation failed with a timeout - the storage was too slow or there some issue on the storage side. Can you mount this manually and access file as user vdsm?
2020-01-02 09:38:33,579-0500 INFO (jsonrpc/0) [vdsm.api] FINISH createStorageDomain error=Could not initialize cluster lock: () from=::ffff:192.168.100.1,36452, flow_id=17dc614 e, task_id=05c2107a-4d59-48d0-a2f7-0938f051c9ab (api:52) 2020-01-02 09:38:33,582-0500 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='05c2107a-4d59-48d0-a2f7-0938f051c9ab') Unexpected error (task:874) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout')
Looks like same error raised again, a common issue in legacy code.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 881, in _run return fn(*args, **kargs) File "<decorator-gen-121>", line 2, in createStorageDomain File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2622, in createStorageDomain max_hosts=max_hosts) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120, in create fsd.initSPMlease() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 974, in initSPMlease return self._manifest.initDomainLock() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 620, in initDomainLock self._domainLock.initLock(self.getDomainLease()) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 308, in initLock block_size=self._block_size) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 260, in initSANLock raise se.ClusterLockInitError() vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster lock: ()
There is no information about this error, looks like the public useless error we raise for the underlying error seen before.
Thanks, I can mount it manually, but when trying to read the file, I'm getting the error: /usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py: No such file or directory The directory's user & group are on my user (I'm unable to change ownership for some reason), but I have set earlier the directory's (which I tried to add as a storage domain) permissions to 777 On Thu, Jan 2, 2020 at 5:24 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, Jan 2, 2020 at 4:44 PM Dana Elfassy <delfassy@redhat.com> wrote:
Hi, When trying to add a storage domain to a 4.4 host I'm getting this error
Error while executing action New NFS Storage Domain: Unexpected exception
The errors from vdsm.log: 2020-01-02 09:38:33,578-0500 ERROR (jsonrpc/0) [storage.initSANLock] Cannot initialize SANLock for domain 6ca1e203-5595-47e5-94b8-82a7e69d99a9 (clusterlock:259) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py",
message: line 250, in initSANLock
lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO
timeout')
This means sanlock operation failed with a timeout - the storage was too slow or there some issue on the storage side.
Can you mount this manually and access file as user vdsm?
2020-01-02 09:38:33,579-0500 INFO (jsonrpc/0) [vdsm.api] FINISH createStorageDomain error=Could not initialize cluster lock: () from=::ffff:192.168.100.1,36452, flow_id=17dc614 e, task_id=05c2107a-4d59-48d0-a2f7-0938f051c9ab (api:52) 2020-01-02 09:38:33,582-0500 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='05c2107a-4d59-48d0-a2f7-0938f051c9ab') Unexpected error (task:874) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout')
Looks like same error raised again, a common issue in legacy code.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 881, in _run return fn(*args, **kargs) File "<decorator-gen-121>", line 2, in createStorageDomain File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2622, in createStorageDomain max_hosts=max_hosts) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120, in create fsd.initSPMlease() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 974, in initSPMlease return self._manifest.initDomainLock() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 620, in initDomainLock self._domainLock.initLock(self.getDomainLease()) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 308, in initLock block_size=self._block_size) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 260, in initSANLock raise se.ClusterLockInitError() vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster lock: ()
There is no information about this error, looks like the public useless error we raise for the underlying error seen before.
On Thu, Jan 2, 2020 at 6:47 PM Dana Elfassy <delfassy@redhat.com> wrote:
Thanks, I can mount it manually, but when trying to read the file, I'm getting the error: /usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py: No such file or directory The directory's user & group are on my user (I'm unable to change ownership for some reason), but I have set earlier the directory's (which I tried to add as a storage domain) permissions to 777
This does sound like a healthy storage domain configuration. The way to create a storage domain is: In /etc/exports add entry like: $ cat /etc/exports /export/1 *(rw,async,anonuid=36,anongid=36) /export/2 *(rw,async,anonuid=36,anongid=36) Note: async is not safe for production, I'm using it to simulate a fast nfs server. You probably want to use sync which is the default. Reload nfs server configuration: # exportfs -r If you check the export you will see: # exportfs -v /export/1 <world>(async,wdelay,hide,no_subtree_check,anonuid=36,anongid=36,sec=sys,rw,secure,root_squash,no_all_squash) /export/2 <world>(async,wdelay,hide,no_subtree_check,anonuid=36,anongid=36,sec=sys,rw,secure,root_squash,no_all_squash) Then change the owner of the directory to vdsm:kvm (you many need to add a vdsm user on the nfs server) # chown -R vdsm:kvm /path/to/export On the host, the mount should look like: # mount | grep nfs1 nfs1:/export/2 on /rhev/data-center/mnt/nfs1:_export_2 type nfs4 (rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.122.20,local_lock=none,addr=192.168.122.30) (I'm using NFS 4.2 which is much faster, you should use it too) # tree -pug /rhev/data-center/mnt/nfs1\:_export_2/ /rhev/data-center/mnt/nfs1:_export_2/ └── [drwxr-xr-x vdsm kvm ] 55255570-983a-4d82-907a-19b964abf7ed ├── [drwxr-xr-x vdsm kvm ] dom_md │ ├── [-rw-rw---- vdsm kvm ] ids │ ├── [-rw-rw---- vdsm kvm ] inbox │ ├── [-rw-rw---- vdsm kvm ] leases │ ├── [-rw-r--r-- vdsm kvm ] metadata │ ├── [-rw-rw---- vdsm kvm ] outbox │ └── [-rw-rw---- vdsm kvm ] xleases ├── [drwxr-xr-x vdsm kvm ] images │ ├── [drwxr-xr-x vdsm kvm ] 17b81130-f4a4-4764-ab5a-258a803c7706 │ │ ├── [-rw-rw---- vdsm kvm ] 66738f70-bb6c-4ce0-a98e-c7109d3b4275 │ │ ├── [-rw-rw---- vdsm kvm ] 66738f70-bb6c-4ce0-a98e-c7109d3b4275.lease │ │ └── [-rw-r--r-- vdsm kvm ] 66738f70-bb6c-4ce0-a98e-c7109d3b4275.meta │ ├── [drwxr-xr-x vdsm kvm ] 6fd5eafc-572b-4335-b59b-42282ba10464 │ │ ├── [-rw-rw---- vdsm kvm ] 46e25ec2-04ae-4829-b32d-972c1e15da09 │ │ ├── [-rw-rw---- vdsm kvm ] 46e25ec2-04ae-4829-b32d-972c1e15da09.lease │ │ └── [-rw-r--r-- vdsm kvm ] 46e25ec2-04ae-4829-b32d-972c1e15da09.meta │ ├── [drwxr-xr-x vdsm kvm ] b2cdc744-9992-4a97-8d80-37c40c5b8f84 │ │ ├── [-rw-rw---- vdsm kvm ] 12b58c9b-ccd7-4375-a0df-fc956075f313 │ │ ├── [-rw-rw---- vdsm kvm ] 12b58c9b-ccd7-4375-a0df-fc956075f313.lease │ │ └── [-rw-r--r-- vdsm kvm ] 12b58c9b-ccd7-4375-a0df-fc956075f313.meta │ └── [drwxr-xr-x vdsm kvm ] e95c50b2-d066-499b-9c58-6a479b50e515 │ ├── [-rw-rw---- vdsm kvm ] 69435001-9c25-41fd-85d3-7f7321a503e6 │ ├── [-rw-rw---- vdsm kvm ] 69435001-9c25-41fd-85d3-7f7321a503e6.lease │ └── [-rw-r--r-- vdsm kvm ] 69435001-9c25-41fd-85d3-7f7321a503e6.meta └── [drwxr-xr-x vdsm kvm ] master ├── [drwxr-xr-x vdsm kvm ] tasks └── [drwxr-xr-x vdsm kvm ] vms
On Thu, Jan 2, 2020 at 5:24 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, Jan 2, 2020 at 4:44 PM Dana Elfassy <delfassy@redhat.com> wrote:
Hi, When trying to add a storage domain to a 4.4 host I'm getting this error message: Error while executing action New NFS Storage Domain: Unexpected exception
The errors from vdsm.log: 2020-01-02 09:38:33,578-0500 ERROR (jsonrpc/0) [storage.initSANLock] Cannot initialize SANLock for domain 6ca1e203-5595-47e5-94b8-82a7e69d99a9 (clusterlock:259) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout')
This means sanlock operation failed with a timeout - the storage was too slow or there some issue on the storage side.
Can you mount this manually and access file as user vdsm?
2020-01-02 09:38:33,579-0500 INFO (jsonrpc/0) [vdsm.api] FINISH createStorageDomain error=Could not initialize cluster lock: () from=::ffff:192.168.100.1,36452, flow_id=17dc614 e, task_id=05c2107a-4d59-48d0-a2f7-0938f051c9ab (api:52) 2020-01-02 09:38:33,582-0500 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='05c2107a-4d59-48d0-a2f7-0938f051c9ab') Unexpected error (task:874) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout')
Looks like same error raised again, a common issue in legacy code.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 881, in _run return fn(*args, **kargs) File "<decorator-gen-121>", line 2, in createStorageDomain File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2622, in createStorageDomain max_hosts=max_hosts) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120, in create fsd.initSPMlease() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 974, in initSPMlease return self._manifest.initDomainLock() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 620, in initDomainLock self._domainLock.initLock(self.getDomainLease()) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 308, in initLock block_size=self._block_size) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 260, in initSANLock raise se.ClusterLockInitError() vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster lock: ()
There is no information about this error, looks like the public useless error we raise for the underlying error seen before.
Hi, I had permissions issue on the storage domain, it was fixed today & after setting the right ownership I was able to add it successfully Thanks :) On Thu, Jan 2, 2020 at 7:02 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, Jan 2, 2020 at 6:47 PM Dana Elfassy <delfassy@redhat.com> wrote:
Thanks, I can mount it manually, but when trying to read the file, I'm getting
The directory's user & group are on my user (I'm unable to change ownership for some reason), but I have set earlier the directory's (which I
the error: /usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py: No such file or directory tried to add as a storage domain) permissions to 777
This does sound like a healthy storage domain configuration.
The way to create a storage domain is:
In /etc/exports add entry like:
$ cat /etc/exports /export/1 *(rw,async,anonuid=36,anongid=36) /export/2 *(rw,async,anonuid=36,anongid=36)
Note: async is not safe for production, I'm using it to simulate a fast nfs server. You probably want to use sync which is the default.
Reload nfs server configuration:
# exportfs -r
If you check the export you will see:
# exportfs -v /export/1 <world>(async,wdelay,hide,no_subtree_check,anonuid=36,anongid=36,sec=sys,rw,secure,root_squash,no_all_squash) /export/2 <world>(async,wdelay,hide,no_subtree_check,anonuid=36,anongid=36,sec=sys,rw,secure,root_squash,no_all_squash)
Then change the owner of the directory to vdsm:kvm (you many need to add a vdsm user on the nfs server)
# chown -R vdsm:kvm /path/to/export
On the host, the mount should look like:
# mount | grep nfs1 nfs1:/export/2 on /rhev/data-center/mnt/nfs1:_export_2 type nfs4
(rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.122.20,local_lock=none,addr=192.168.122.30)
(I'm using NFS 4.2 which is much faster, you should use it too)
# tree -pug /rhev/data-center/mnt/nfs1\:_export_2/ /rhev/data-center/mnt/nfs1:_export_2/ └── [drwxr-xr-x vdsm kvm ] 55255570-983a-4d82-907a-19b964abf7ed ├── [drwxr-xr-x vdsm kvm ] dom_md │ ├── [-rw-rw---- vdsm kvm ] ids │ ├── [-rw-rw---- vdsm kvm ] inbox │ ├── [-rw-rw---- vdsm kvm ] leases │ ├── [-rw-r--r-- vdsm kvm ] metadata │ ├── [-rw-rw---- vdsm kvm ] outbox │ └── [-rw-rw---- vdsm kvm ] xleases ├── [drwxr-xr-x vdsm kvm ] images │ ├── [drwxr-xr-x vdsm kvm ] 17b81130-f4a4-4764-ab5a-258a803c7706 │ │ ├── [-rw-rw---- vdsm kvm ] 66738f70-bb6c-4ce0-a98e-c7109d3b4275 │ │ ├── [-rw-rw---- vdsm kvm ] 66738f70-bb6c-4ce0-a98e-c7109d3b4275.lease │ │ └── [-rw-r--r-- vdsm kvm ] 66738f70-bb6c-4ce0-a98e-c7109d3b4275.meta │ ├── [drwxr-xr-x vdsm kvm ] 6fd5eafc-572b-4335-b59b-42282ba10464 │ │ ├── [-rw-rw---- vdsm kvm ] 46e25ec2-04ae-4829-b32d-972c1e15da09 │ │ ├── [-rw-rw---- vdsm kvm ] 46e25ec2-04ae-4829-b32d-972c1e15da09.lease │ │ └── [-rw-r--r-- vdsm kvm ] 46e25ec2-04ae-4829-b32d-972c1e15da09.meta │ ├── [drwxr-xr-x vdsm kvm ] b2cdc744-9992-4a97-8d80-37c40c5b8f84 │ │ ├── [-rw-rw---- vdsm kvm ] 12b58c9b-ccd7-4375-a0df-fc956075f313 │ │ ├── [-rw-rw---- vdsm kvm ] 12b58c9b-ccd7-4375-a0df-fc956075f313.lease │ │ └── [-rw-r--r-- vdsm kvm ] 12b58c9b-ccd7-4375-a0df-fc956075f313.meta │ └── [drwxr-xr-x vdsm kvm ] e95c50b2-d066-499b-9c58-6a479b50e515 │ ├── [-rw-rw---- vdsm kvm ] 69435001-9c25-41fd-85d3-7f7321a503e6 │ ├── [-rw-rw---- vdsm kvm ] 69435001-9c25-41fd-85d3-7f7321a503e6.lease │ └── [-rw-r--r-- vdsm kvm ] 69435001-9c25-41fd-85d3-7f7321a503e6.meta └── [drwxr-xr-x vdsm kvm ] master ├── [drwxr-xr-x vdsm kvm ] tasks └── [drwxr-xr-x vdsm kvm ] vms
On Thu, Jan 2, 2020 at 5:24 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, Jan 2, 2020 at 4:44 PM Dana Elfassy <delfassy@redhat.com>
wrote:
Hi, When trying to add a storage domain to a 4.4 host I'm getting this
error message:
Error while executing action New NFS Storage Domain: Unexpected exception
The errors from vdsm.log: 2020-01-02 09:38:33,578-0500 ERROR (jsonrpc/0) [storage.initSANLock] Cannot initialize SANLock for domain 6ca1e203-5595-47e5-94b8-82a7e69d99a9 (clusterlock:259) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout')
This means sanlock operation failed with a timeout - the storage was too slow or there some issue on the storage side.
Can you mount this manually and access file as user vdsm?
2020-01-02 09:38:33,579-0500 INFO (jsonrpc/0) [vdsm.api] FINISH createStorageDomain error=Could not initialize cluster lock: () from=::ffff:192.168.100.1,36452, flow_id=17dc614 e, task_id=05c2107a-4d59-48d0-a2f7-0938f051c9ab (api:52) 2020-01-02 09:38:33,582-0500 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='05c2107a-4d59-48d0-a2f7-0938f051c9ab') Unexpected error (task:874) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout')
Looks like same error raised again, a common issue in legacy code.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 881, in _run return fn(*args, **kargs) File "<decorator-gen-121>", line 2, in createStorageDomain File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2622, in createStorageDomain max_hosts=max_hosts) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120, in create fsd.initSPMlease() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 974, in initSPMlease return self._manifest.initDomainLock() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 620, in initDomainLock self._domainLock.initLock(self.getDomainLease()) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 308, in initLock block_size=self._block_size) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 260, in initSANLock raise se.ClusterLockInitError() vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster lock: ()
There is no information about this error, looks like the public useless error we raise for the underlying error seen before.
participants (2)
-
Dana Elfassy -
Nir Soffer