4:03 a.m.
Hi,
When I'm trying to add a storage domain via the webadmin I'm getting the
error message that's on the subject.
in the vdsm.log I see this error:
2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error
checking path
/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata
(monitor:525)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line
523, in _pathChecked
delay = result.delay()
File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391,
in delay
raise exception.MiscFileReadException(self.path, self.rc, self.err)
vdsm.storage.exception.MiscFileReadException: Internal file read failure:
('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata',
1, bytearray(b"/usr/bin/dd: failed to open
\'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\':
No such file or directory\n"))
when looking at the directory I do see that metadata was created under this
directory, with the following content:
[delfassy@delfassy mnt]$ cd
new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/
[delfassy@delfassy dom_md]$ less metadata
CLASS=Data
DESCRIPTION=spider
IOOPTIMEOUTSEC=10
LEASERETRIES=3
LEASETIMESEC=60
LOCKPOLICY=
LOCKRENEWALINTERVALSEC=5
POOL_UUID=
REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs
ROLE=Regular
SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d
TYPE=NFS
VERSION=4
_SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7
The permissions on the directory are:
drwxrwxrwx. 3 vdsm kvm 4096 Jul 29 11:49 new_mnt
Are there any other configurations I'm missing? Any idea how this issue can
be resolved?
Thanks,
Dana
4:29 a.m.
New subject: Error while executing action New NFS Storage Domain: Unexpected exception
On Wed, Jul 29, 2020, 12:09 Dana Elfassy <delfassy(a)redhat.com> wrote:
Hi,
When I'm trying to add a storage domain via the webadmin I'm getting the
error message that's on the subject.
in the vdsm.log I see this error:
2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error
checking path
/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata
(monitor:525)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line
523, in _pathChecked
delay = result.delay()
File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391,
in delay
raise exception.MiscFileReadException(self.path, self.rc, self.err)
vdsm.storage.exception.MiscFileReadException: Internal file read failure:
('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata',
1, bytearray(b"/usr/bin/dd: failed to open
\'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\':
No such file or directory\n"))
This means the metadata file was not accessible when the monitor tried to
read.
Does it happen now?
Anyway this is not the unexpected error that failed the operation.
Can you share vdsm log showing the original error?
when looking at the directory I do see that metadata was created
under
this directory, with the following content:
[delfassy@delfassy mnt]$ cd
new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/
[delfassy@delfassy dom_md]$ less metadata
CLASS=Data
DESCRIPTION=spider
IOOPTIMEOUTSEC=10
LEASERETRIES=3
LEASETIMESEC=60
LOCKPOLICY=
LOCKRENEWALINTERVALSEC=5
POOL_UUID=
REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs
ROLE=Regular
SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d
TYPE=NFS
VERSION=4
_SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7
Looks correct
The permissions on the directory are:
drwxrwxrwx. 3 vdsm kvm 4096 Jul 29 11:49 new_mnt
You don't need 0777, 0750 should work.
ls -lhZ?
Are there any other configurations I'm missing? Any idea how this
issue
can be resolved?
Thanks,
Dana
_______________________________________________
Devel mailing list -- devel(a)ovirt.org
To unsubscribe send an email to devel-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/L6MUN6YP74W...
2:54 a.m.
New subject: Error while executing action New NFS Storage Domain: Unexpected exception
Thanks Nir,
I just checked again, it still happens
This is the original error:
2020-07-30 03:53:26,071-0400 ERROR (jsonrpc/4) [storage.TaskManager.Task]
(Task='2904a936-d6ca-464e-9c86-973c9fbb9b78') Unexpected error (task:880)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line
250, in initSANLock
lockspace_name, idsPath, align=alignment, sector=block_size)
sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO
timeout')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887,
in _run
return fn(*args, **kargs)
File "<decorator-gen-121>", line 2, in createStorageDomain
File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2644,
in createStorageDomain
max_hosts=max_hosts)
File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120,
in create
fsd.initSPMlease()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 999, in
initSPMlease
return self._manifest.initDomainLock()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 627, in
initDomainLock
self._domainLock.initLock(self.getDomainLease())
File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line
308, in initLock
block_size=self._block_size)
File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line
260, in initSANLock
raise se.ClusterLockInitError()
vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster
lock: ()
2020-07-30 03:53:26,071-0400 INFO (jsonrpc/4) [storage.TaskManager.Task]
(Task='2904a936-d6ca-464e-9c86-973c9fbb9b78') aborting: Task is aborted:
'value=Could not initialize cluster lock: () abortedcode=701' (task:1190)
2020-07-30 03:53:26,071-0400 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH
createStorageDomain error=Could not initialize cluster lock: ()
(dispatcher:83)
[delfassy@delfassy mnt]$ ls -lhZ
drwxrwxrwx. 3 vdsm kvm system_u:object_r:nfs_t:s0 4.0K Jul 30
10:53 new_mnt
On Wed, Jul 29, 2020 at 12:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Wed, Jul 29, 2020, 12:09 Dana Elfassy <delfassy(a)redhat.com> wrote:
> Hi,
> When I'm trying to add a storage domain via the webadmin I'm getting the
> error message that's on the subject.
> in the vdsm.log I see this error:
> 2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error
> checking path
/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata
> (monitor:525)
> Traceback (most recent call last):
> File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line
> 523, in _pathChecked
> delay = result.delay()
> File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line
> 391, in delay
> raise exception.MiscFileReadException(self.path, self.rc, self.err)
> vdsm.storage.exception.MiscFileReadException: Internal file read failure:
>
('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata',
> 1, bytearray(b"/usr/bin/dd: failed to open
>
\'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\':
> No such file or directory\n"))
>
This means the metadata file was not accessible when the monitor tried to
read.
Does it happen now?
Anyway this is not the unexpected error that failed the operation.
Can you share vdsm log showing the original error?
> when looking at the directory I do see that metadata was created under
> this directory, with the following content:
> [delfassy@delfassy mnt]$ cd
> new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/
> [delfassy@delfassy dom_md]$ less metadata
> CLASS=Data
> DESCRIPTION=spider
> IOOPTIMEOUTSEC=10
> LEASERETRIES=3
> LEASETIMESEC=60
> LOCKPOLICY=
> LOCKRENEWALINTERVALSEC=5
> POOL_UUID=
> REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs
> ROLE=Regular
> SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d
> TYPE=NFS
> VERSION=4
> _SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7
>
Looks correct
> The permissions on the directory are:
> drwxrwxrwx. 3 vdsm kvm 4096 Jul 29 11:49 new_mnt
>
You don't need 0777, 0750 should work.
ls -lhZ?
> Are there any other configurations I'm missing? Any idea how this issue
> can be resolved?
> Thanks,
> Dana
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org
> To unsubscribe send an email to devel-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/L6MUN6YP74W...
>
4:03 a.m.
New subject: Error while executing action New NFS Storage Domain: Unexpected exception
On Thu, Jul 30, 2020, 10:57 Dana Elfassy <delfassy(a)redhat.com> wrote:
Thanks Nir,
I just checked again, it still happens
This is the original error:
2020-07-30 03:53:26,071-0400 ERROR (jsonrpc/4) [storage.TaskManager.Task]
(Task='2904a936-d6ca-464e-9c86-973c9fbb9b78') Unexpected error (task:880)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py",
line 250, in initSANLock
lockspace_name, idsPath, align=alignment, sector=block_size)
sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO
timeout')
So your storage was too slow, and sanlock could not write 1MiB to storage
in 10 seconds.
You can delete the partly created domain and try again.
In 8.3 sanlock will use 60 seconds timeout for this kind of operations, so
such failure will be less likely.
But this storage failure means your storage is not healthy and you will
have more trouble.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887,
in _run
return fn(*args, **kargs)
File "<decorator-gen-121>", line 2, in createStorageDomain
File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2644,
in createStorageDomain
max_hosts=max_hosts)
File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120,
in create
fsd.initSPMlease()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 999, in
initSPMlease
return self._manifest.initDomainLock()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 627, in
initDomainLock
self._domainLock.initLock(self.getDomainLease())
File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py",
line 308, in initLock
block_size=self._block_size)
File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py",
line 260, in initSANLock
raise se.ClusterLockInitError()
vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster
lock: ()
2020-07-30 03:53:26,071-0400 INFO (jsonrpc/4) [storage.TaskManager.Task]
(Task='2904a936-d6ca-464e-9c86-973c9fbb9b78') aborting: Task is aborted:
'value=Could not initialize cluster lock: () abortedcode=701' (task:1190)
2020-07-30 03:53:26,071-0400 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH
createStorageDomain error=Could not initialize cluster lock: ()
(dispatcher:83)
[delfassy@delfassy mnt]$ ls -lhZ
drwxrwxrwx. 3 vdsm kvm system_u:object_r:nfs_t:s0 4.0K Jul 30
10:53 new_mnt
On Wed, Jul 29, 2020 at 12:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
>
>
> On Wed, Jul 29, 2020, 12:09 Dana Elfassy <delfassy(a)redhat.com> wrote:
>
>> Hi,
>> When I'm trying to add a storage domain via the webadmin I'm getting the
>> error message that's on the subject.
>> in the vdsm.log I see this error:
>> 2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error
>> checking path
/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata
>> (monitor:525)
>> Traceback (most recent call last):
>> File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py",
line
>> 523, in _pathChecked
>> delay = result.delay()
>> File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line
>> 391, in delay
>> raise exception.MiscFileReadException(self.path, self.rc, self.err)
>> vdsm.storage.exception.MiscFileReadException: Internal file read
>> failure:
('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata',
>> 1, bytearray(b"/usr/bin/dd: failed to open
>>
\'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\':
>> No such file or directory\n"))
>>
>
> This means the metadata file was not accessible when the monitor tried to
> read.
>
> Does it happen now?
>
> Anyway this is not the unexpected error that failed the operation.
>
> Can you share vdsm log showing the original error?
>
>
>> when looking at the directory I do see that metadata was created under
>> this directory, with the following content:
>> [delfassy@delfassy mnt]$ cd
>> new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/
>> [delfassy@delfassy dom_md]$ less metadata
>> CLASS=Data
>> DESCRIPTION=spider
>> IOOPTIMEOUTSEC=10
>> LEASERETRIES=3
>> LEASETIMESEC=60
>> LOCKPOLICY=
>> LOCKRENEWALINTERVALSEC=5
>> POOL_UUID=
>> REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs
>> ROLE=Regular
>> SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d
>> TYPE=NFS
>> VERSION=4
>> _SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7
>>
>
> Looks correct
>
>
>> The permissions on the directory are:
>> drwxrwxrwx. 3 vdsm kvm 4096 Jul 29 11:49 new_mnt
>>
>
> You don't need 0777, 0750 should work.
>
> ls -lhZ?
>
>
>> Are there any other configurations I'm missing? Any idea how this issue
>> can be resolved?
>> Thanks,
>> Dana
>> _______________________________________________
>> Devel mailing list -- devel(a)ovirt.org
>> To unsubscribe send an email to devel-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/L6MUN6YP74W...
>>
>
1586
days inactive
1587
days old
3 comments
2 participants
participants (2)
-
Dana Elfassy -
Nir Soffer