[ovirt-users] Host cannot connect to hosted storage domain
Alexander Witte
alexander.witte at baicanada.com
Thu Sep 28 03:23:45 UTC 2017
OK after a host reboot I was able to get the Engine VM up again and into the Web interface. Although continually when I try a second host to the datacenter within oVirt I run into the error:
"Host mennode2 cannot access the Storage Domain(s) hosted_storage attached to the Data Center Train1. Setting Host state to Non Operational."
Note: I can successfully read (and mount) the NFS exports oVirt is complaining about:
[root at mennode2 ~]# showmount -e menmaster.traindemo.local
Export list for menmaster.traindemo.local:
/shares *
/shares/exports *
/shares/data *
/shares/isos *
[root at mennode2 ~]#
[root at mennode2 tmp]# mount -t nfs menmaster.traindemo.local:/shares test
[root at mennode2 tmp]# cd test
[root at mennode2 test]# ls
7d18ff24-57a3-4b4a-9934-0263191fe2e4 data __DIRECT_IO_TEST__ exports isos
[root at mennode2 test]#
One thing I DO notice is the path of my exports is the difference in the path for the hosted_storage domain. I wonder if the second host would have issues resolving this?
Data. ==> menmaster.traindemo.local:/shares/data
Export ==> menmaster.traindemo.local:/shares/exports
Hosted_storage. ==> localhost:/shares
ISO. ==> menmaster.traindemo.local:/shares/exports
Below I have copied exports from the VDSM and OVIRT-ENGINE log and have copied the output of the hosted-engine.conf file. Any help in pinpointing the source of the problem is greatly appreciated!!
VSDM logs:
2017-09-27 23:10:01,700-0400 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.06 seconds (__init__:539)
2017-09-27 23:10:03,104-0400 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'59c7f8f3-0063-00a8-02c7-0000000000f3', options=None) from=::ffff:10.0.0.227,39748, flow_id=19e2dbb3, task_id=7a05a5bd-9b15-43be-890d-c6f5d7650e5c (api:46)
2017-09-27 23:10:03,109-0400 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8L}} from=::ffff:10.0.0.227,39748, flow_id=19e2dbb3, task_id=7a05a5bd-9b15-43be-890d-c6f5d7650e5c (api:52)
2017-09-27 23:10:03,110-0400 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:539)
2017-09-27 23:10:03,189-0400 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59c7f8f3-0063-00a8-02c7-0000000000f3', options=None) from=::ffff:10.0.0.227,39866, flow_id=19e2dbb3, task_id=0f8b49e9-9d82-457e-a2a2-39dc7ed9f022 (api:46)
2017-09-27 23:10:03,196-0400 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/menmaster.traindemo.local:_shares_isos/da001a29-eca5-44d6-a097-129dd9be623f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 8L, 'domains': u'da001a29-eca5-44d6-a097-129dd9be623f:Active,f36157cc-b25a-400a-ab0f-a071e8a8eea7:Active,7d18ff24-57a3-4b4a-9934-0263191fe2e4:Active,795d4a1d-3ceb-4773-99de-8e7cf05112f3:Active', 'master_uuid': u'f36157cc-b25a-400a-ab0f-a071e8a8eea7', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {u'da001a29-eca5-44d6-a097-129dd9be623f': {'status': u'Active', 'diskfree': '1044166737920', 'isoprefix': u'/rhev/data-center/mnt/menmaster.traindemo.local:_shares_isos/da001a29-eca5-44d6-a097-129dd9be623f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1049702170624', 'version': 0}, u'f36157cc-b25a-400a-ab0f-a071e8a8eea7': {'status': u'Active', 'diskfree': '1044166737920', 'isoprefix': '', 'alerts': [], 'disktotal': '1049702170624', 'version': 4}, u'7d18ff24-57a3-4b4a-9934-0263191fe2e4': {'status': u'Active', 'diskfree': '1044166737920', 'isoprefix': '', 'alerts': [], 'disktotal': '1049702170624', 'version': 4}, u'795d4a1d-3ceb-4773-99de-8e7cf05112f3': {'status': u'Active', 'diskfree': '1044166737920', 'isoprefix': '', 'alerts': [], 'disktotal': '1049702170624', 'version': 0}}} from=::ffff:10.0.0.227,39866, flow_id=19e2dbb3, task_id=0f8b49e9-9d82-457e-a2a2-39dc7ed9f022 (api:52)
2017-09-27 23:10:03,198-0400 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:539)
2017-09-27 23:10:04,212-0400 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:35980 (protocoldetector:72)
2017-09-27 23:10:04,223-0400 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:35980 (protocoldetector:127)
2017-09-27 23:10:04,224-0400 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:103)
2017-09-27 23:10:04,225-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:132)
2017-09-27 23:10:04,622-0400 INFO (monitor/7d18ff2) [storage.SANLock] Acquiring host id for domain 7d18ff24-57a3-4b4a-9934-0263191fe2e4 (id=1, async=True) (clusterlock:288)
2017-09-27 23:10:04,623-0400 ERROR (monitor/7d18ff2) [storage.Monitor] Error acquiring host id 1 for domain 7d18ff24-57a3-4b4a-9934-0263191fe2e4 (monitor:570)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/monitor.py", line 567, in _acquireHostId
self.domain.acquireHostId(self.hostId, async=True)
File "/usr/share/vdsm/storage/sd.py", line 785, in acquireHostId
self._manifest.acquireHostId(hostId, async)
File "/usr/share/vdsm/storage/sd.py", line 449, in acquireHostId
self._domainLock.acquireHostId(hostId, async)
File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", line 319, in acquireHostId
raise se.AcquireHostIdFailure(self._sdUUID, e)
AcquireHostIdFailure: Cannot acquire host id: (u'7d18ff24-57a3-4b4a-9934-0263191fe2e4', SanlockException(22, 'Sanlock lockspace add failure', 'Invalid argument'))
2017-09-27 23:10:04,647-0400 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539)
2017-09-27 23:10:04,654-0400 INFO (jsonrpc/3) [vdsm.api] START repoStats(options=None) from=::1,35980, task_id=9732972d-9a76-4349-9c65-53e57b2472d6 (api:46)
2017-09-27 23:10:04,655-0400 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'da001a29-eca5-44d6-a097-129dd9be623f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000357054', 'lastCheck': '4.1', 'valid': True}, u'f36157cc-b25a-400a-ab0f-a071e8a8eea7': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000531828', 'lastCheck': '4.0', 'valid': True}, u'7d18ff24-57a3-4b4a-9934-0263191fe2e4': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000205228', 'lastCheck': '0.0', 'valid': True}, u'795d4a1d-3ceb-4773-99de-8e7cf05112f3': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000352165', 'lastCheck': '4.1', 'valid': True}} from=::1,35980, task_id=9732972d-9a76-4349-9c65-53e57b2472d6 (api:52)
2017-09-27 23:10:04,656-0400 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStorageRepoStats succeeded in 0.00 seconds (__init__:539)
2017-09-27 23:10:07,717-0400 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:35982 (protocoldetector:72)
2017-09-27 23:10:07,728-0400 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:35982 (protocoldetector:127)
2017-09-27 23:10:07,729-0400 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:103)
2017-09-27 23:10:07,730-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:132)
2017-09-27 23:10:08,152-0400 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539)
2017-09-27 23:10:08,159-0400 INFO (jsonrpc/1) [vdsm.api] START repoStats(options=None) from=::1,35982, task_id=80eb282b-f297-4f98-a368-7d5627ac50f3 (api:46)
2017-09-27 23:10:08,160-0400 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'da001a29-eca5-44d6-a097-129dd9be623f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000357054', 'lastCheck': '7.6', 'valid': True}, u'f36157cc-b25a-400a-ab0f-a071e8a8eea7': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000531828', 'lastCheck': '7.5', 'valid': True}, u'7d18ff24-57a3-4b4a-9934-0263191fe2e4': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000205228', 'lastCheck': '3.5', 'valid': True}, u'795d4a1d-3ceb-4773-99de-8e7cf05112f3': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000352165', 'lastCheck': '7.6', 'valid': True}} from=::1,35982, task_id=80eb282b-f297-4f98-a368-7d5627ac50f3 (api:52)
2017-09-27 23:10:08,171-0400 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:35984 (protocoldetector:72)
2017-09-27 23:10:08,198-0400 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:35984 (protocoldetector:127)
2017-09-27 23:10:08,199-0400 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:103)
2017-09-27 23:10:08,201-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:132)
2017-09-27 23:10:08,374-0400 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.22 seconds (__init__:539)
2017-09-27 23:10:08,728-0400 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:35987 (protocoldetector:72)
2017-09-27 23:10:08,732-0400 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539)
2017-09-27 23:10:08,738-0400 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats succeeded in 0.00 seconds (__init__:539)
2017-09-27 23:10:08,756-0400 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:35987 (protocoldetector:127)
2017-09-27 23:10:08,757-0400 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:103)
2017-09-27 23:10:08,771-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:132)
2017-09-27 23:10:09,286-0400 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539)
2017-09-27 23:10:09,395-0400 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities succeeded in 0.10 seconds (__init__:539)
2017-09-27 23:10:10,590-0400 INFO (itmap/0) [IOProcessClient] Starting client ioprocess-37 (__init__:330)
ENGINE LOGS:
2017-09-27 23:07:58,154-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20170927230758-10.0.0.222-3f898655.log'
2017-09-27 23:07:58,316-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Stage: Termination
2017-09-27 23:07:58,458-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-1) [3f898655] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host mennode2 was completed successfully with message 'no updates found.'.
2017-09-27 23:07:58,874-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Stage: Misc configuration
2017-09-27 23:07:58,888-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Stage: Transaction commit
2017-09-27 23:07:59,029-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Stage: Closing up
2017-09-27 23:07:59,039-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Stage: Pre-termination
2017-09-27 23:07:59,092-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20170927230759-menmaster.traindemo.local-3f898655.log'
2017-09-27 23:07:59,262-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Stage: Termination
2017-09-27 23:07:59,375-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-2) [3f898655] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host menmaster was completed successfully with message 'no updates found.'.
2017-09-27 23:08:38,705-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-13) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[0ef7ec10-af19-4afa-980b-77a772c4e014=VDS]', sharedLocks=''}'
2017-09-27 23:08:38,776-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 0ef7ec10-af19-4afa-980b-77a772c4e014 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2017-09-27 23:08:38,777-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Before acquiring lock in order to prevent monitoring for host 'mennode2' from data-center 'Train1'
2017-09-27 23:08:38,778-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[0ef7ec10-af19-4afa-980b-77a772c4e014=VDS_INIT]', sharedLocks=''}'
2017-09-27 23:08:38,995-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Lock acquired, from now a monitoring of host will be skipped for host 'mennode2' from data-center 'Train1'
2017-09-27 23:08:39,003-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] START, SetVdsStatusVDSCommand(HostName = mennode2, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='0ef7ec10-af19-4afa-980b-77a772c4e014', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 78cc1caf
2017-09-27 23:08:39,027-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] FINISH, SetVdsStatusVDSCommand, log id: 78cc1caf
2017-09-27 23:08:39,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Activate host finished. Lock released. Monitoring can run now for host 'mennode2' from data-center 'Train1'
2017-09-27 23:08:39,050-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] EVENT_ID: VDS_ACTIVATE(16), Correlation ID: 75de01a6-efd2-418f-a6b1-b4ef1f38923f, Job ID: 2e40097e-e991-45cc-bd49-3176822ebb5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Activation of host mennode2 initiated by admin at internal-authz.
2017-09-27 23:08:39,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Lock freed to object 'EngineLock:{exclusiveLocks='[0ef7ec10-af19-4afa-980b-77a772c4e014=VDS]', sharedLocks=''}'
2017-09-27 23:08:43,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5437c1b7] START, GetHardwareInfoVDSCommand(HostName = mennode2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='0ef7ec10-af19-4afa-980b-77a772c4e014', vds='Host[mennode2,0ef7ec10-af19-4afa-980b-77a772c4e014]'}), log id: 2bcbe26e
2017-09-27 23:08:44,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5437c1b7] FINISH, GetHardwareInfoVDSCommand, log id: 2bcbe26e
2017-09-27 23:08:44,186-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [12ef27d8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 0ef7ec10-af19-4afa-980b-77a772c4e014 Type: VDS
2017-09-27 23:08:44,214-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [63cdc728] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 59c7f8f3-0063-00a8-02c7-0000000000f3 Type: StoragePool
2017-09-27 23:08:44,234-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7670f89e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 59c7f8f3-0063-00a8-02c7-0000000000f3 Type: StoragePool
2017-09-27 23:08:44,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7670f89e] START, ConnectStorageServerVDSCommand(HostName = mennode2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='0ef7ec10-af19-4afa-980b-77a772c4e014', storagePoolId='59c7f8f3-0063-00a8-02c7-0000000000f3', storageType='NFS', connectionList='[StorageServerConnections:{id='aac63433-88fb-4daf-8c83-871a85b3a49d', connection='menmaster.traindemo.local:/shares/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='AUTO', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='5adb7a4b-ebbd-43a4-b5cb-8c8eb3d6f1ee', connection='localhost:/shares', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='9eb3b1e2-4063-46cd-ba4e-440c4048e92a', connection='menmaster.traindemo.local:/shares/isos', iqn='null', vfsType='null', mountOptions='null', nfsVersion='AUTO', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='1c9215a1-4154-4a78-b8ff-565c3a237c58', connection='menmaster.traindemo.local:/shares/exports', iqn='null', vfsType='null', mountOptions='null', nfsVersion='AUTO', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5625b02f
2017-09-27 23:08:48,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7670f89e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host mennode2 to the Storage Domains hosted_storage.
2017-09-27 23:08:48,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7670f89e] FINISH, ConnectStorageServerVDSCommand, return: {aac63433-88fb-4daf-8c83-871a85b3a49d=0, 5adb7a4b-ebbd-43a4-b5cb-8c8eb3d6f1ee=477, 9eb3b1e2-4063-46cd-ba4e-440c4048e92a=0, 1c9215a1-4154-4a78-b8ff-565c3a237c58=0}, log id: 5625b02f
2017-09-27 23:08:48,386-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7670f89e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection localhost:/shares returned by VDSM was: Problem while trying to mount target
2017-09-27 23:08:48,389-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7670f89e] The connection with details 'localhost:/shares' failed because of error code '477' and error message is: problem while trying to mount target
2017-09-27 23:08:48,401-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7670f89e] Host 'mennode2' storage connection was failed
2017-09-27 23:08:48,411-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7670f89e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7670f89e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host mennode2 to Storage Servers
2017-09-27 23:08:48,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = mennode2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='0ef7ec10-af19-4afa-980b-77a772c4e014', vdsId='0ef7ec10-af19-4afa-980b-77a772c4e014', storagePoolId='59c7f8f3-0063-00a8-02c7-0000000000f3', masterVersion='1'}), log id: 3d5c5d79
2017-09-27 23:08:48,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {7d18ff24-57a3-4b4a-9934-0263191fe2e4=active, da001a29-eca5-44d6-a097-129dd9be623f=active, 795d4a1d-3ceb-4773-99de-8e7cf05112f3=active, f36157cc-b25a-400a-ab0f-a071e8a8eea7=active}
2017-09-27 23:08:49,656-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3d5c5d79
2017-09-27 23:08:50,681-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-8) [] Domain '7d18ff24-57a3-4b4a-9934-0263191fe2e4:hosted_storage' was reported with error code '358'
2017-09-27 23:08:50,682-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Storage Domain 'hosted_storage' of pool 'Train1' is in problem in host 'mennode2'
2017-09-27 23:08:50,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(189), Correlation ID: 63cdc728, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host mennode2 reports about one of the Active Storage Domains as Problematic.
2017-09-27 23:08:50,726-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [748ef73f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 0ef7ec10-af19-4afa-980b-77a772c4e014 Type: VDS
2017-09-27 23:08:50,729-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [748ef73f] START, SetVdsStatusVDSCommand(HostName = mennode2, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='0ef7ec10-af19-4afa-980b-77a772c4e014', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2589acd2
2017-09-27 23:08:50,736-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [748ef73f] FINISH, SetVdsStatusVDSCommand, log id: 2589acd2
2017-09-27 23:08:50,807-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [748ef73f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 748ef73f, Job ID: e80941e4-3dbf-4159-8421-96d2d90160ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host mennode2 cannot access the Storage Domain(s) hosted_storage attached to the Data Center Train1. Setting Host state to Non-Operational.
2017-09-27 23:08:50,820-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [748ef73f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 63cdc728, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host mennode2.
2017-09-27 23:08:50,826-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [748ef73f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 63cdc728, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host mennode2 to Storage Pool Train1
2017-09-27 23:08:50,856-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [42cbaee1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 0ef7ec10-af19-4afa-980b-77a772c4e014 Type: VDS
2017-09-27 23:08:50,863-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [42cbaee1] Host 'mennode2'(0ef7ec10-af19-4afa-980b-77a772c4e014) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
/etc/ovirt-hosted-engine/hosted-engine.conf
fqdn=engine.traindemo.local
vm_disk_id=0077cb58-7abe-4bdf-96b1-55cc85030d76
vm_disk_vol_id=d5bd2690-7cfb-4aa2-b898-41579114ec08
vmid=564d290f-ae66-4352-9770-f2e062000358
storage=menmaster.traindemo.local:/shares
mnt_options=
conf=/var/run/ovirt-hosted-engine-ha/vm.conf
host_id=1
console=vnc
domainType=nfs3
spUUID=00000000-0000-0000-0000-000000000000
sdUUID=7d18ff24-57a3-4b4a-9934-0263191fe2e4
connectionUUID=addfd5fc-8668-4d2a-9b0b-f801b21a98d8
ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem
ca_subject="C=EN, L=Test, O=Test, CN=Test"
vdsm_use_ssl=true
gateway=10.0.0.1
bridge=ovirtmgmt
metadata_volume_UUID=156c7549-8e2e-4580-9825-81c3b13387ce
metadata_image_UUID=c5db06ac-ea37-47bc-9065-b94eb230616a
lockspace_volume_UUID=5f4f4a63-5608-4daf-a034-2b8e83b5de42
lockspace_image_UUID=0d7f8661-3abd-44e8-87a7-3b85f2e29fda
conf_volume_UUID=ddffd499-294c-456f-bb0e-08e366380a22
conf_image_UUID=5f264ea1-4654-4e40-87aa-80fcaacc60d8
Thanks,
Alex Witte
On Sep 27, 2017, at 8:32 PM, Alexander Witte <alexander.witte at baicanada.com<mailto:alexander.witte at baicanada.com>> wrote:
/etc/ovirt-hosted-engine/hosted-engine.conf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170928/acba009a/attachment.html>
More information about the Users
mailing list