Error while executing action New NFS Storage Domain: Unexpected exception
Hi, When I'm trying to add a storage domain via the webadmin I'm getting the error message that's on the subject. in the vdsm.log I see this error: 2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata (monitor:525) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 523, in _pathChecked delay = result.delay() File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391, in delay raise exception.MiscFileReadException(self.path, self.rc, self.err) vdsm.storage.exception.MiscFileReadException: Internal file read failure: ('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata', 1, bytearray(b"/usr/bin/dd: failed to open \'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\': No such file or directory\n")) when looking at the directory I do see that metadata was created under this directory, with the following content: [delfassy@delfassy mnt]$ cd new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/ [delfassy@delfassy dom_md]$ less metadata CLASS=Data DESCRIPTION=spider IOOPTIMEOUTSEC=10 LEASERETRIES=3 LEASETIMESEC=60 LOCKPOLICY= LOCKRENEWALINTERVALSEC=5 POOL_UUID= REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs ROLE=Regular SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d TYPE=NFS VERSION=4 _SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7 The permissions on the directory are: drwxrwxrwx. 3 vdsm kvm 4096 Jul 29 11:49 new_mnt Are there any other configurations I'm missing? Any idea how this issue can be resolved? Thanks, Dana
On Wed, Jul 29, 2020, 12:09 Dana Elfassy <delfassy@redhat.com> wrote:
Hi, When I'm trying to add a storage domain via the webadmin I'm getting the error message that's on the subject. in the vdsm.log I see this error: 2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata (monitor:525) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 523, in _pathChecked delay = result.delay() File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391, in delay raise exception.MiscFileReadException(self.path, self.rc, self.err) vdsm.storage.exception.MiscFileReadException: Internal file read failure: ('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata', 1, bytearray(b"/usr/bin/dd: failed to open \'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\': No such file or directory\n"))
This means the metadata file was not accessible when the monitor tried to read. Does it happen now? Anyway this is not the unexpected error that failed the operation. Can you share vdsm log showing the original error?
when looking at the directory I do see that metadata was created under this directory, with the following content: [delfassy@delfassy mnt]$ cd new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/ [delfassy@delfassy dom_md]$ less metadata CLASS=Data DESCRIPTION=spider IOOPTIMEOUTSEC=10 LEASERETRIES=3 LEASETIMESEC=60 LOCKPOLICY= LOCKRENEWALINTERVALSEC=5 POOL_UUID= REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs ROLE=Regular SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d TYPE=NFS VERSION=4 _SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7
Looks correct
The permissions on the directory are: drwxrwxrwx. 3 vdsm kvm 4096 Jul 29 11:49 new_mnt
You don't need 0777, 0750 should work. ls -lhZ?
Are there any other configurations I'm missing? Any idea how this issue can be resolved? Thanks, Dana _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/L6MUN6YP74WYEM...
Thanks Nir, I just checked again, it still happens This is the original error: 2020-07-30 03:53:26,071-0400 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='2904a936-d6ca-464e-9c86-973c9fbb9b78') Unexpected error (task:880) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in _run return fn(*args, **kargs) File "<decorator-gen-121>", line 2, in createStorageDomain File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2644, in createStorageDomain max_hosts=max_hosts) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120, in create fsd.initSPMlease() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 999, in initSPMlease return self._manifest.initDomainLock() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 627, in initDomainLock self._domainLock.initLock(self.getDomainLease()) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 308, in initLock block_size=self._block_size) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 260, in initSANLock raise se.ClusterLockInitError() vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster lock: () 2020-07-30 03:53:26,071-0400 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='2904a936-d6ca-464e-9c86-973c9fbb9b78') aborting: Task is aborted: 'value=Could not initialize cluster lock: () abortedcode=701' (task:1190) 2020-07-30 03:53:26,071-0400 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH createStorageDomain error=Could not initialize cluster lock: () (dispatcher:83) [delfassy@delfassy mnt]$ ls -lhZ drwxrwxrwx. 3 vdsm kvm system_u:object_r:nfs_t:s0 4.0K Jul 30 10:53 new_mnt On Wed, Jul 29, 2020 at 12:30 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Jul 29, 2020, 12:09 Dana Elfassy <delfassy@redhat.com> wrote:
Hi, When I'm trying to add a storage domain via the webadmin I'm getting the error message that's on the subject. in the vdsm.log I see this error: 2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata (monitor:525) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 523, in _pathChecked delay = result.delay() File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391, in delay raise exception.MiscFileReadException(self.path, self.rc, self.err) vdsm.storage.exception.MiscFileReadException: Internal file read failure: ('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata', 1, bytearray(b"/usr/bin/dd: failed to open \'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\': No such file or directory\n"))
This means the metadata file was not accessible when the monitor tried to read.
Does it happen now?
Anyway this is not the unexpected error that failed the operation.
Can you share vdsm log showing the original error?
when looking at the directory I do see that metadata was created under this directory, with the following content: [delfassy@delfassy mnt]$ cd new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/ [delfassy@delfassy dom_md]$ less metadata CLASS=Data DESCRIPTION=spider IOOPTIMEOUTSEC=10 LEASERETRIES=3 LEASETIMESEC=60 LOCKPOLICY= LOCKRENEWALINTERVALSEC=5 POOL_UUID= REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs ROLE=Regular SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d TYPE=NFS VERSION=4 _SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7
Looks correct
The permissions on the directory are: drwxrwxrwx. 3 vdsm kvm 4096 Jul 29 11:49 new_mnt
You don't need 0777, 0750 should work.
ls -lhZ?
Are there any other configurations I'm missing? Any idea how this issue can be resolved? Thanks, Dana _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/L6MUN6YP74WYEM...
On Thu, Jul 30, 2020, 10:57 Dana Elfassy <delfassy@redhat.com> wrote:
Thanks Nir, I just checked again, it still happens This is the original error:
2020-07-30 03:53:26,071-0400 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='2904a936-d6ca-464e-9c86-973c9fbb9b78') Unexpected error (task:880) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout')
So your storage was too slow, and sanlock could not write 1MiB to storage in 10 seconds. You can delete the partly created domain and try again. In 8.3 sanlock will use 60 seconds timeout for this kind of operations, so such failure will be less likely. But this storage failure means your storage is not healthy and you will have more trouble.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in _run return fn(*args, **kargs) File "<decorator-gen-121>", line 2, in createStorageDomain File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2644, in createStorageDomain max_hosts=max_hosts) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120, in create fsd.initSPMlease() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 999, in initSPMlease return self._manifest.initDomainLock() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 627, in initDomainLock self._domainLock.initLock(self.getDomainLease()) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 308, in initLock block_size=self._block_size) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 260, in initSANLock raise se.ClusterLockInitError() vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster lock: () 2020-07-30 03:53:26,071-0400 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='2904a936-d6ca-464e-9c86-973c9fbb9b78') aborting: Task is aborted: 'value=Could not initialize cluster lock: () abortedcode=701' (task:1190) 2020-07-30 03:53:26,071-0400 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH createStorageDomain error=Could not initialize cluster lock: () (dispatcher:83)
[delfassy@delfassy mnt]$ ls -lhZ drwxrwxrwx. 3 vdsm kvm system_u:object_r:nfs_t:s0 4.0K Jul 30 10:53 new_mnt
On Wed, Jul 29, 2020 at 12:30 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Jul 29, 2020, 12:09 Dana Elfassy <delfassy@redhat.com> wrote:
Hi, When I'm trying to add a storage domain via the webadmin I'm getting the error message that's on the subject. in the vdsm.log I see this error: 2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata (monitor:525) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 523, in _pathChecked delay = result.delay() File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391, in delay raise exception.MiscFileReadException(self.path, self.rc, self.err) vdsm.storage.exception.MiscFileReadException: Internal file read failure: ('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata', 1, bytearray(b"/usr/bin/dd: failed to open \'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\': No such file or directory\n"))
This means the metadata file was not accessible when the monitor tried to read.
Does it happen now?
Anyway this is not the unexpected error that failed the operation.
Can you share vdsm log showing the original error?
when looking at the directory I do see that metadata was created under this directory, with the following content: [delfassy@delfassy mnt]$ cd new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/ [delfassy@delfassy dom_md]$ less metadata CLASS=Data DESCRIPTION=spider IOOPTIMEOUTSEC=10 LEASERETRIES=3 LEASETIMESEC=60 LOCKPOLICY= LOCKRENEWALINTERVALSEC=5 POOL_UUID= REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs ROLE=Regular SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d TYPE=NFS VERSION=4 _SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7
Looks correct
The permissions on the directory are: drwxrwxrwx. 3 vdsm kvm 4096 Jul 29 11:49 new_mnt
You don't need 0777, 0750 should work.
ls -lhZ?
Are there any other configurations I'm missing? Any idea how this issue can be resolved? Thanks, Dana _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/L6MUN6YP74WYEM...
participants (2)
-
Dana Elfassy -
Nir Soffer