Hosted engine unaviable after storage rebuild
Hello all, My hosted engine does not start up anymore, because the node can't reach the storage. The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. What happend is that my NAS crashed and I had to rebuild it from the backup. I'm quit sure that I called all the NFS shares the same way as they bevore the issue. How can I figure out whats the issue with the storage? The logs wheren't realy helpful for me (but may it's me and not the logs to blame) Regards and thanks Sebi
On Wed, Jul 11, 2018 at 7:55 AM <sebastian.w@gmx.ch> wrote:
Hello all, My hosted engine does not start up anymore, because the node can't reach the storage.
The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
What happend is that my NAS crashed and I had to rebuild it from the backup. I'm quit sure that I called all the NFS shares the same way as they bevore the issue.
How can I figure out whats the issue with the storage? The logs wheren't realy helpful for me (but may it's me and not the logs to blame)
You have to check /var/log/ovirt-hosted-engine-ha/broker.log and /var/log/vdsm/vdsm.log for errors about mounting your NFS share. Did you tried manually mounting it?
Regards and thanks Sebi _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2TWEIRH45T6C3D...
sorry for the late replay, I was in holiday... I only checked the "normal" logs mentioned in the docs of oVirt. In /var/log/ovirt-hosted-engine-ha/broker.log I found MainThread::WARNING::2018-07-22 08:33:20,338::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: Command Image.prepare with args {'imageID': 'e14ecef8-28ec-4b93-b0a7-7f490358ddbf', 'storagepoolID': '00000000-0000-0000-0000-000000000000', 'volumeID': 'a6caa26a-6584-420f-8f55-2c396aac589a', 'storagedomainID': '2395042b-7031-4c83-85a4-a2bb325fdbfd'} failed: (code=100, message=[Errno 13] Permission denied) But the NFS share I can mount (as root) on /mnt. I changed the permission of the Folder to 777 and the owner is the user 36/vdsm drwxrwxrwx. 5 vdsm kvm 4096 Jul 22 09:19 2395042b-7031-4c83-85a4-a2bb325fdbfd But stil I get a Permission denied Thank for you help
On Sun, Jul 22, 2018 at 9:37 AM <sebastian.w@gmx.ch> wrote:
sorry for the late replay, I was in holiday...
I only checked the "normal" logs mentioned in the docs of oVirt.
In /var/log/ovirt-hosted-engine-ha/broker.log I found
MainThread::WARNING::2018-07-22 08:33:20,338::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: Command Image.prepare with args {'imageID': 'e14ecef8-28ec-4b93-b0a7-7f490358ddbf', 'storagepoolID': '00000000-0000-0000-0000-000000000000', 'volumeID': 'a6caa26a-6584-420f-8f55-2c396aac589a', 'storagedomainID': '2395042b-7031-4c83-85a4-a2bb325fdbfd'} failed: (code=100, message=[Errno 13] Permission denied)
But the NFS share I can mount (as root) on /mnt.
That error is not related to NFS mount but to the creation of a symlink for that on the local FS. Can you please share the whole relevant vdsm log?
I changed the permission of the Folder to 777 and the owner is the user 36/vdsm
drwxrwxrwx. 5 vdsm kvm 4096 Jul 22 09:19 2395042b-7031-4c83-85a4-a2bb325fdbfd
But stil I get a Permission denied
Thank for you help _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YH26KNTYJX2IQF...
the vdsm log is endless. These are the last lines Hope it helps OSError: [Errno 13] Permission denied 2018-07-29 16:56:17,069+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getStats failed (error 350) in 0.19 seconds (__init__:573) 2018-07-29 16:56:17,097+0200 INFO (jsonrpc/0) [vdsm.api] START prepareImage(sdUUID=u'2395042b-7031-4c83-85a4-a2bb325fdbfd', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'e14ecef8-28ec-4b93-b0a7-7f490358ddbf', leafUUID=u'a6caa26a-6584-420f-8f55-2c396aac589a', allowIllegal=False) from=::1,43892, task_id=286bd8bf-69a1-4c4e-b496-99efd7220fe0 (api:46) 2018-07-29 16:56:17,105+0200 INFO (jsonrpc/0) [vdsm.api] FINISH prepareImage error=[Errno 13] Permission denied from=::1,43892, task_id=286bd8bf-69a1-4c4e-b496-99efd7220fe0 (api:50) 2018-07-29 16:56:17,105+0200 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='286bd8bf-69a1-4c4e-b496-99efd7220fe0') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in prepareImage File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3156, in prepareImage dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 128, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 391, in __init__ validateFileSystemFeatures(manifest.sdUUID, manifest.mountpoint) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 104, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 567, in touch self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 451, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied 2018-07-29 16:56:17,105+0200 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='286bd8bf-69a1-4c4e-b496-99efd7220fe0') aborting: Task is aborted: u'[Errno 13] Permission denied' - code 100 (task:1181) 2018-07-29 16:56:17,105+0200 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH prepareImage error=[Errno 13] Permission denied (dispatcher:86) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 73, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error OSError: [Errno 13] Permission denied 2018-07-29 16:56:17,105+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Image.prepare failed (error 100) in 0.01 seconds (__init__:573) 2018-07-29 16:56:17,431+0200 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList(options=None) from=internal, task_id=48e0e011-5c17-466c-9a21-fe427dd5c8e3 (api:46) 2018-07-29 16:56:17,431+0200 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=48e0e011-5c17-466c-9a21-fe427dd5c8e3 (api:52) 2018-07-29 16:56:17,431+0200 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:707) 2018-07-29 16:56:21,930+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=19446066-361d-49ab-aef4-653f42c819ed (api:46) 2018-07-29 16:56:21,930+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=19446066-361d-49ab-aef4-653f42c819ed (api:52) 2018-07-29 16:56:21,931+0200 INFO (periodic/0) [vdsm.api] START multipath_health() from=internal, task_id=6fe54950-82f6-4881-9cdd-2363b002203e (api:46) 2018-07-29 16:56:21,931+0200 INFO (periodic/0) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=6fe54950-82f6-4881-9cdd-2363b002203e (api:52) 2018-07-29 16:56:21,937+0200 ERROR (periodic/0) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2018-07-29 16:56:22,437+0200 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList(options=None) from=internal, task_id=986cc8f1-e6df-4542-8f7e-e51a4f042fc8 (api:46) 2018-07-29 16:56:22,437+0200 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=986cc8f1-e6df-4542-8f7e-e51a4f042fc8 (api:52) 2018-07-29 16:56:22,437+0200 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:707) 2018-07-29 16:56:23,567+0200 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:43894 (protocoldetector:61) 2018-07-29 16:56:23,576+0200 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:43894 (protocoldetector:125) 2018-07-29 16:56:23,576+0200 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:103) 2018-07-29 16:56:23,577+0200 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:132) 2018-07-29 16:56:23,617+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.01 seconds (__init__:573) 2018-07-29 16:56:23,627+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-07-29 16:56:23,631+0200 INFO (jsonrpc/7) [vdsm.api] START getStorageDomainInfo(sdUUID=u'2395042b-7031-4c83-85a4-a2bb325fdbfd', options=None) from=::1,43894, task_id=f41da873-7c4f-49cd-94ae-55a8fb120fbd (api:46) 2018-07-29 16:56:23,646+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStorageDomainInfo error=[Errno 13] Permission denied from=::1,43894, task_id=f41da873-7c4f-49cd-94ae-55a8fb120fbd (api:50) 2018-07-29 16:56:23,646+0200 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='f41da873-7c4f-49cd-94ae-55a8fb120fbd') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in getStorageDomainInfo File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2718, in getStorageDomainInfo dom = self.validateSdUUID(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 304, in validateSdUUID sdDom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 128, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 391, in __init__ validateFileSystemFeatures(manifest.sdUUID, manifest.mountpoint) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 104, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 567, in touch self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 451, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied 2018-07-29 16:56:23,646+0200 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='f41da873-7c4f-49cd-94ae-55a8fb120fbd') aborting: Task is aborted: u'[Errno 13] Permission denied' - code 100 (task:1181) 2018-07-29 16:56:23,647+0200 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH getStorageDomainInfo error=[Errno 13] Permission denied (dispatcher:86) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 73, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error OSError: [Errno 13] Permission denied 2018-07-29 16:56:23,647+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getInfo failed (error 350) in 0.01 seconds (__init__:573) 2018-07-29 16:56:23,663+0200 INFO (jsonrpc/6) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'protocol_version': u'4', u'connection': u'10.11.1.20:/Multimedia', u'user': u'kvm', u'id': u'71f7ae1e-6b37-422a-a2c0-9d9b7a8b31cd'}], options=None) from=::1,43894, task_id=3b810f18-1dfa-4cc2-947c-e4806b9e8b24 (api:46) 2018-07-29 16:56:23,666+0200 INFO (jsonrpc/6) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'71f7ae1e-6b37-422a-a2c0-9d9b7a8b31cd'}]} from=::1,43894, task_id=3b810f18-1dfa-4cc2-947c-e4806b9e8b24 (api:52) 2018-07-29 16:56:23,666+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.00 seconds (__init__:573) 2018-07-29 16:56:23,670+0200 INFO (jsonrpc/1) [vdsm.api] START getStorageDomainStats(sdUUID=u'2395042b-7031-4c83-85a4-a2bb325fdbfd', options=None) from=::1,43894, task_id=b09dfc3f-9a54-4fa2-98f3-782473d07d7e (api:46) 2018-07-29 16:56:23,752+0200 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:43896 (protocoldetector:61) 2018-07-29 16:56:23,764+0200 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:43896 (protocoldetector:125) 2018-07-29 16:56:23,764+0200 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:103) 2018-07-29 16:56:23,766+0200 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:132) 2018-07-29 16:56:23,807+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-07-29 16:56:23,890+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getStorageDomainStats error=[Errno 13] Permission denied from=::1,43894, task_id=b09dfc3f-9a54-4fa2-98f3-782473d07d7e (api:50) 2018-07-29 16:56:23,890+0200 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='b09dfc3f-9a54-4fa2-98f3-782473d07d7e') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in getStorageDomainStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2740, in getStorageDomainStats dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 128, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 391, in __init__ validateFileSystemFeatures(manifest.sdUUID, manifest.mountpoint) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 104, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 567, in touch self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 451, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied 2018-07-29 16:56:23,891+0200 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='b09dfc3f-9a54-4fa2-98f3-782473d07d7e') aborting: Task is aborted: u'[Errno 13] Permission denied' - code 100 (task:1181) 2018-07-29 16:56:23,891+0200 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH getStorageDomainStats error=[Errno 13] Permission denied (dispatcher:86) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 73, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error OSError: [Errno 13] Permission denied 2018-07-29 16:56:23,891+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getStats failed (error 350) in 0.22 seconds (__init__:573) 2018-07-29 16:56:23,936+0200 INFO (jsonrpc/3) [vdsm.api] START prepareImage(sdUUID=u'2395042b-7031-4c83-85a4-a2bb325fdbfd', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'e14ecef8-28ec-4b93-b0a7-7f490358ddbf', leafUUID=u'a6caa26a-6584-420f-8f55-2c396aac589a', allowIllegal=False) from=::1,43894, task_id=d0df7850-b0f8-4069-9cc7-6965d3fbe128 (api:46) 2018-07-29 16:56:23,944+0200 INFO (jsonrpc/3) [vdsm.api] FINISH prepareImage error=[Errno 13] Permission denied from=::1,43894, task_id=d0df7850-b0f8-4069-9cc7-6965d3fbe128 (api:50) 2018-07-29 16:56:23,944+0200 ERROR (jsonrpc/3) [storage.TaskManager.Task] (Task='d0df7850-b0f8-4069-9cc7-6965d3fbe128') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in prepareImage File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3156, in prepareImage dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 128, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 391, in __init__ validateFileSystemFeatures(manifest.sdUUID, manifest.mountpoint) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 104, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 567, in touch self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 451, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied 2018-07-29 16:56:23,944+0200 INFO (jsonrpc/3) [storage.TaskManager.Task] (Task='d0df7850-b0f8-4069-9cc7-6965d3fbe128') aborting: Task is aborted: u'[Errno 13] Permission denied' - code 100 (task:1181) 2018-07-29 16:56:23,945+0200 ERROR (jsonrpc/3) [storage.Dispatcher] FINISH prepareImage error=[Errno 13] Permission denied (dispatcher:86) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 73, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error OSError: [Errno 13] Permission denied 2018-07-29 16:56:23,945+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Image.prepare failed (error 100) in 0.01 seconds (__init__:573) 2018-07-29 16:56:26,063+0200 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:43900 (protocoldetector:61) 2018-07-29 16:56:26,074+0200 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:43900 (protocoldetector:125) 2018-07-29 16:56:26,075+0200 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:103) 2018-07-29 16:56:26,077+0200 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:132) 2018-07-29 16:56:26,117+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-07-29 16:56:26,129+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-07-29 16:56:26,133+0200 INFO (jsonrpc/4) [vdsm.api] START getStorageDomainInfo(sdUUID=u'2395042b-7031-4c83-85a4-a2bb325fdbfd', options=None) from=::1,43900, task_id=4a6ffdda-848b-4893-b7e4-7cebb6491dd7 (api:46) 2018-07-29 16:56:26,143+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getStorageDomainInfo error=[Errno 13] Permission denied from=::1,43900, task_id=4a6ffdda-848b-4893-b7e4-7cebb6491dd7 (api:50) 2018-07-29 16:56:26,143+0200 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='4a6ffdda-848b-4893-b7e4-7cebb6491dd7') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in getStorageDomainInfo File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2718, in getStorageDomainInfo dom = self.validateSdUUID(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 304, in validateSdUUID sdDom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 128, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 391, in __init__ validateFileSystemFeatures(manifest.sdUUID, manifest.mountpoint) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 104, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 567, in touch self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 451, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied 2018-07-29 16:56:26,143+0200 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='4a6ffdda-848b-4893-b7e4-7cebb6491dd7') aborting: Task is aborted: u'[Errno 13] Permission denied' - code 100 (task:1181) 2018-07-29 16:56:26,143+0200 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH getStorageDomainInfo error=[Errno 13] Permission denied (dispatcher:86) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 73, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error OSError: [Errno 13] Permission denied 2018-07-29 16:56:26,144+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getInfo failed (error 350) in 0.01 seconds (__init__:573) 2018-07-29 16:56:26,149+0200 INFO (jsonrpc/7) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'protocol_version': u'4', u'connection': u'10.11.1.20:/Multimedia', u'user': u'kvm', u'id': u'71f7ae1e-6b37-422a-a2c0-9d9b7a8b31cd'}], options=None) from=::1,43900, task_id=339a49e4-b0f3-4a12-b162-0ce671bc8570 (api:46) 2018-07-29 16:56:26,153+0200 INFO (jsonrpc/7) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'71f7ae1e-6b37-422a-a2c0-9d9b7a8b31cd'}]} from=::1,43900, task_id=339a49e4-b0f3-4a12-b162-0ce671bc8570 (api:52) 2018-07-29 16:56:26,153+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.00 seconds (__init__:573) 2018-07-29 16:56:26,156+0200 INFO (jsonrpc/6) [vdsm.api] START getStorageDomainStats(sdUUID=u'2395042b-7031-4c83-85a4-a2bb325fdbfd', options=None) from=::1,43900, task_id=df624a63-3e8e-45f6-9986-7bec231aed9d (api:46) 2018-07-29 16:56:26,175+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,43540 (api:46) 2018-07-29 16:56:26,176+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,43540 (api:52) 2018-07-29 16:56:26,177+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 2018-07-29 16:56:26,370+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStorageDomainStats error=[Errno 13] Permission denied from=::1,43900, task_id=df624a63-3e8e-45f6-9986-7bec231aed9d (api:50) 2018-07-29 16:56:26,371+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='df624a63-3e8e-45f6-9986-7bec231aed9d') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in getStorageDomainStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2740, in getStorageDomainStats dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 128, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 391, in __init__ validateFileSystemFeatures(manifest.sdUUID, manifest.mountpoint) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 104, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 567, in touch self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 451, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied 2018-07-29 16:56:26,371+0200 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='df624a63-3e8e-45f6-9986-7bec231aed9d') aborting: Task is aborted: u'[Errno 13] Permission denied' - code 100 (task:1181) 2018-07-29 16:56:26,371+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getStorageDomainStats error=[Errno 13] Permission denied (dispatcher:86) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 73, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error OSError: [Errno 13] Permission denied 2018-07-29 16:56:26,371+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getStats failed (error 350) in 0.22 seconds (__init__:573) 2018-07-29 16:56:26,422+0200 INFO (jsonrpc/1) [vdsm.api] START prepareImage(sdUUID=u'2395042b-7031-4c83-85a4-a2bb325fdbfd', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'e14ecef8-28ec-4b93-b0a7-7f490358ddbf', leafUUID=u'a6caa26a-6584-420f-8f55-2c396aac589a', allowIllegal=False) from=::1,43900, task_id=8aff6d78-4877-4e4d-9649-e94587f924e3 (api:46) 2018-07-29 16:56:26,431+0200 INFO (jsonrpc/1) [vdsm.api] FINISH prepareImage error=[Errno 13] Permission denied from=::1,43900, task_id=8aff6d78-4877-4e4d-9649-e94587f924e3 (api:50) 2018-07-29 16:56:26,431+0200 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='8aff6d78-4877-4e4d-9649-e94587f924e3') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in prepareImage File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3156, in prepareImage dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 128, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 391, in __init__ validateFileSystemFeatures(manifest.sdUUID, manifest.mountpoint) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 104, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 567, in touch self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 451, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied 2018-07-29 16:56:26,431+0200 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='8aff6d78-4877-4e4d-9649-e94587f924e3') aborting: Task is aborted: u'[Errno 13] Permission denied' - code 100 (task:1181) 2018-07-29 16:56:26,432+0200 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH prepareImage error=[Errno 13] Permission denied (dispatcher:86) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 73, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error OSError: [Errno 13] Permission denied 2018-07-29 16:56:26,432+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Image.prepare failed (error 100) in 0.01 seconds (__init__:573)
participants (2)
-
sebastian.w＠gmx.ch -
Simone Tiraboschi