Can't connect vdsm storage: Command StorageDomain.getInfo with args failed: (code=350, message=Error in storage domain action
11:56 a.m.
Hi! I trying to upgrade my hosts and have problem with it. After uprgading one host i see
that this one NonOperational. All was fine with vdsm-4.30.24-1.el7 but after upgrading
with new version vdsm-4.30.40-1.el7.x86_64 and some others i have errors.
Firtst of all i see in ovirt Events: Host srv02 cannot access the Storage Domain(s)
<UNKNOWN> attached to the Data Center Default. Setting Host state to
Non-Operational. My Default storage domain with HE VM data on NFS storage.
In messages log of host:
srv02 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback
(most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/a
gent.py", line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line
55, in action_proper#012 return he.start_monitoring
()#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012 File
"/usr/lib/python2.7/site-packages/
ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012
m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 8
9, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink -
failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory,
[monitor: 'network', options:
{'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None,
'addr': '192.168.2.248'}]
Feb 1 15:41:42 srv02 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent
ERROR Trying to restart agent
In broker log:
MainThread::WARNING::2020-02-01
15:43:35,167::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__)
Can't connect vdsm storage: Command StorageDomain.getInfo with ar
gs {'storagedomainID': 'bbdddea7-9cd6-41e7-ace5-fb9a6795caa8'} failed:
(code=350, message=Error in storage domain action:
(u'sdUUID=bbdddea7-9cd6-41e7-ace5-fb9a6795caa8',))
In vdsm.lod
2020-02-01 15:44:19,930+0600 INFO (jsonrpc/0) [vdsm.api] FINISH getStorageDomainInfo
error=[Errno 1] Operation not permitted from=::1,57528,
task_id=40683f67-d7b0-4105-aab8-6338deb54b00 (api:52)
2020-02-01 15:44:19,930+0600 ERROR (jsonrpc/0) [storage.TaskManager.Task]
(Task='40683f67-d7b0-4105-aab8-6338deb54b00') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in getStorageDomainInfo
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2753, in
getStorageDomainInfo
dom = self.validateSdUUID(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 305, in
validateSdUUID
sdDom = sdCache.produce(sdUUID=sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 145, in
findDomain
return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 378, in
__init__
manifest.sdUUID, manifest.mountpoint)
File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 853, in
_detect_block_size
block_size = iop.probe_block_size(mountpoint)
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line
384, in probe_block_size
return self._ioproc.probe_block_size(dir_path)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 602, in
probe_block_size
"probe_block_size", {"dir": dir_path}, self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in
_sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 1] Operation not permitted
2020-02-01 15:44:19,930+0600 INFO (jsonrpc/0) [storage.TaskManager.Task]
(Task='40683f67-d7b0-4105-aab8-6338deb54b00') aborting: Task is aborted:
u'[Errno 1] Operation not permitted' - code 100 (task:1
181)
2020-02-01 15:44:19,930+0600 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH
getStorageDomainInfo error=[Errno 1] Operation not permitted (dispatcher:87)
But i see that this domain is mounted (by mount command):
storage:/volume3/ovirt-hosted on /rhev/data-center/mnt/storage:_volume3_ovirt-hosted type
nfs4
(rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.2.251,local_lock=none,addr=192.168.2.248)
I didnt see storage directory in /var/run/vdsm? I see many differences with another hosts.
Here is listing of var/run/vdsm:
bonding-defaults.json
dhclientmon
nets_restored
payload
svdsm.sock
v2v
vhostuser
bonding-name2numeric.json
mom-vdsm.sock
ovirt-imageio-daemon.sock
supervdsmd.lock
trackedInterfaces
vdsmd.lock
What whe problem? Please help.
12:29 p.m.
New subject: Can't connect vdsm storage: Command StorageDomain.getInfo with args failed: (code=350, message=Error in storage domain action
Htis issue can be resolve by downgrading of the packages:
Installing : vdsm-api-4.30.32-1.el7.noarch
1/26
Installing : vdsm-common-4.30.32-1.el7.noarch
2/26
Installing : vdsm-yajsonrpc-4.30.32-1.el7.noarch
3/26
Installing : vdsm-network-4.30.32-1.el7.x86_64
4/26
Installing : vdsm-python-4.30.32-1.el7.noarch
5/26
Installing : vdsm-jsonrpc-4.30.32-1.el7.noarch
6/26
Installing : vdsm-http-4.30.32-1.el7.noarch
7/26
Installing : vdsm-hook-vmfex-dev-4.30.32-1.el7.noarch
8/26
Installing : vdsm-4.30.32-1.el7.x86_64
9/26
Installing : vdsm-gluster-4.30.32-1.el7.x86_64
10/26
Installing : vdsm-hook-ethtool-options-4.30.32-1.el7.noarch
11/26
Installing : vdsm-hook-fcoe-4.30.32-1.el7.noarch
12/26
Installing : vdsm-client-4.30.32-1.el7.noarch
13/26
Cleanup : vdsm-client-4.30.33-1.el7.noarch
14/26
Cleanup : vdsm-hook-ethtool-options-4.30.33-1.el7.noarch
15/26
Cleanup : vdsm-gluster-4.30.33-1.el7.x86_64
16/26
Cleanup : vdsm-hook-fcoe-4.30.33-1.el7.noarch
17/26
Cleanup : vdsm-hook-vmfex-dev-4.30.33-1.el7.noarch
18/26
Cleanup : vdsm-4.30.33-1.el7.x86_64
19/26
Cleanup : vdsm-jsonrpc-4.30.33-1.el7.noarch
20/26
Cleanup : vdsm-http-4.30.33-1.el7.noarch
21/26
Cleanup : vdsm-python-4.30.33-1.el7.noarch
22/26
Cleanup : vdsm-network-4.30.33-1.el7.x86_64
23/26
Cleanup : vdsm-common-4.30.33-1.el7.noarch
24/26
Cleanup : vdsm-api-4.30.33-1.el7.noarch
25/26
Cleanup : vdsm-yajsonrpc-4.30.33-1.el7.noarch
26/26
In other words, something happened after the version 4.30.32-1. With 4.30.33-1 all
working fine, after installing 4.30.33-1 - i see this issue.
12:38 p.m.
New subject: Can't connect vdsm storage: Command StorageDomain.getInfo with args failed: (code=350, message=Error in storage domain action
Working fine with 4.30.32-1 of course, sorry.
4:17 p.m.
New subject: Can't connect vdsm storage: Command StorageDomain.getInfo with args failed: (code=350, message=Error in storage domain action
On Saturday, February 1, 2020, <asm(a)pioner.kz> wrote:
Hi! I trying to upgrade my hosts and have problem with it. After
uprgading
one host i see that this one NonOperational. All was fine with
vdsm-4.30.24-1.el7 but after upgrading with new version
vdsm-4.30.40-1.el7.x86_64 and some others i have errors.
Firtst of all i see in ovirt Events: Host srv02 cannot access the Storage
Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state
to Non-Operational. My Default storage domain with HE VM data on NFS
storage.
In messages log of host:
srv02 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent
ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-
packages/ovirt_hosted_engine_ha/agent/a
gent.py", line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring
()#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012 File
"/usr/lib/python2.7/site-packages/
ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in
_initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 8
9, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network',
options:
{'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None,
'addr':
'192.168.2.248'}]
Feb 1 15:41:42 srv02 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent
ERROR Trying to restart agent
In broker log:
MainThread::WARNING::2020-02-01 15:43:35,167::storage_broker::
97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__)
Can't connect vdsm storage: Command StorageDomain.getInfo with ar
gs {'storagedomainID': 'bbdddea7-9cd6-41e7-ace5-fb9a6795caa8'} failed:
(code=350, message=Error in storage domain action:
(u'sdUUID=bbdddea7-9cd6-41e7-ace5-fb9a6795caa8',))
In vdsm.lod
2020-02-01 15:44:19,930+0600 INFO (jsonrpc/0) [vdsm.api] FINISH
getStorageDomainInfo error=[Errno 1] Operation not permitted
from=::1,57528, task_id=40683f67-d7b0-4105-aab8-6338deb54b00 (api:52)
2020-02-01 15:44:19,930+0600 ERROR (jsonrpc/0) [storage.TaskManager.Task]
(Task='40683f67-d7b0-4105-aab8-6338deb54b00') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882,
in _run
return fn(*args, **kargs)
File "<string>", line 2, in getStorageDomainInfo
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2753,
in getStorageDomainInfo
dom = self.validateSdUUID(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 305,
in validateSdUUID
sdDom = sdCache.produce(sdUUID=sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110,
in produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51,
in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134,
in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151,
in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line
145, in findDomain
return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line
378, in __init__
manifest.sdUUID, manifest.mountpoint)
File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line
853, in _detect_block_size
block_size = iop.probe_block_size(mountpoint)
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
line 384, in probe_block_size
return self._ioproc.probe_block_size(dir_path)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line
602, in probe_block_size
"probe_block_size", {"dir": dir_path}, self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line
448, in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 1] Operation not permitted
2020-02-01 15:44:19,930+0600 INFO (jsonrpc/0) [storage.TaskManager.Task]
(Task='40683f67-d7b0-4105-aab8-6338deb54b00') aborting: Task is aborted:
u'[Errno 1] Operation not permitted' - code 100 (task:1
181)
2020-02-01 15:44:19,930+0600 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH
getStorageDomainInfo error=[Errno 1] Operation not permitted (dispatcher:87)
Seems like a reproduction of
https://bugzilla.redhat.com/show_bug.cgi?id=1777726#c1
Missing file creation/removal permissions on nfs storage.
But i see that this domain is mounted (by mount command):
storage:/volume3/ovirt-hosted on /rhev/data-center/mnt/storage:_volume3_ovirt-hosted
type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=
255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=
sys,clientaddr=192.168.2.251,local_lock=none,addr=192.168.2.248)
I didnt see storage directory in /var/run/vdsm? I see many differences
with another hosts. Here is listing of var/run/vdsm:
bonding-defaults.json
dhclientmon
nets_restored
payload
svdsm.sock
v2v
vhostuser
bonding-name2numeric.json
mom-vdsm.sock
ovirt-imageio-daemon.sock
supervdsmd.lock
trackedInterfaces
vdsmd.lock
What whe problem? Please help.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-
guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/
message/IBUDRUOETQ5WCTZQMGIVBZZZUAITDVHL/
6:38 p.m.
New subject: Can't connect vdsm storage: Command StorageDomain.getInfo with args failed: (code=350, message=Error in storage domain action
Ok, i will try to set 777 permissoin on NFS storage. But, why this issue starting from
updating 4.30.32-1 to 4.30.33-1? Withowt any another changes.
11:31 p.m.
New subject: Can't connect vdsm storage: Command StorageDomain.getInfo with args failed: (code=350, message=Error in storage domain action
On Sat, Feb 1, 2020 at 6:39 PM <asm(a)pioner.kz> wrote:
Ok, i will try to set 777 permissoin on NFS storage. But, why this
issue
starting from updating 4.30.32-1 to 4.30.33-1? Withowt any another
changes.
The differing commit for 4.30.33 over 4.30.32 is the transition into block
size probing done by ioprocess-1.3.0:
https://github.com/oVirt/vdsm/commit/9bd210e340be0855126d1620cdb94840ced5...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ICWRJ75Q7DI...
11:50 p.m.
New subject: Can't connect vdsm storage: Command StorageDomain.getInfo with args failed: (code=350, message=Error in storage domain action
On Sat, Feb 1, 2020 at 5:39 PM <asm(a)pioner.kz> wrote:
Ok, i will try to set 777 permissoin on NFS storage.
This is invalid configuration. See RHV docs for proper configuration:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/...
But, why this issue starting from updating 4.30.32-1 to 4.30.33-1?
Withowt any another changes.
I guess you had wrong permissions and ownership on the storage before, but
vdsm was not detecting
the issue because it was missing validations in older versions. Current
version is validating that creating
and deleting files and using direct I/O works with the storage when
creating and activating a storage
domain.
Nir
12:48 a.m.
New subject: Can't connect vdsm storage: Command StorageDomain.getInfo with args failed: (code=350, message=Error in storage domain action
One thing not in any of the documentation I have found are the extra options required for
the export. I followed all the docs and it was still failing.
I had to add, “sync,no_subtree_check,all_squash,anonuid=36,anongid=36”, to my export in
order to get it to work.
From: Nir Soffer <nsoffer(a)redhat.com>
Sent: Saturday, February 1, 2020 4:51 PM
To: asm(a)pioner.kz
Cc: users <users(a)ovirt.org>
Subject: [ovirt-users] Re: Can't connect vdsm storage: Command StorageDomain.getInfo
with args failed: (code=350, message=Error in storage domain action
On Sat, Feb 1, 2020 at 5:39 PM <asm@pioner.kz<mailto:asm@pioner.kz>> wrote:
Ok, i will try to set 777 permissoin on NFS storage.
This is invalid configuration. See RHV docs for proper configuration:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/...
But, why this issue starting from updating 4.30.32-1 to 4.30.33-1? Withowt any another
changes.
I guess you had wrong permissions and ownership on the storage before, but vdsm was not
detecting
the issue because it was missing validations in older versions. Current version is
validating that creating
and deleting files and using direct I/O works with the storage when creating and
activating a storage
domain.
Nir
12:20 p.m.
New subject: Can't connect vdsm storage: Command StorageDomain.getInfo with args failed: (code=350, message=Error in storage domain action
Yes. The problem with permission. But not with permissoin on export directory. Two ways
to
resolve this:
1) Make 777 permission on directory - dont right solution.
2) anonuid=36,anongid=36 in export - this right solution, very strange that this is not
in
any documentation, but very important!
I has right permission on my export directory, but this is not working before i mke 777
chmod. After i chande parameters in exports file i return chmod to 755 and all working
fine now!
Thank you and all very much.
1755
days inactive
1756
days old
8 comments
5 participants
participants (5)
-
Alexandr Mikhailov -
Amit Bawer -
asm＠pioner.kz
-
Nir Soffer -
Robert Webb