--_000_85068B30E9EA40D5883D6574AC1EC5FBbaicanadacom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable OK after a host reboot I was able to get the Engine VM up again and into th= e Web interface. Although continually when I try a second host to the data= center within oVirt I run into the error: "Host mennode2 cannot access the Storage Domain(s) hosted_storage attached = to the Data Center Train1. Setting Host state to Non Operational." Note: I can successfully read (and mount) the NFS exports oVirt is complai= ning about: [root@mennode2 ~]# showmount -e menmaster.traindemo.local Export list for menmaster.traindemo.local: /shares * /shares/exports * /shares/data * /shares/isos * [root@mennode2 ~]# [root@mennode2 tmp]# mount -t nfs menmaster.traindemo.local:/shares test [root@mennode2 tmp]# cd test [root@mennode2 test]# ls 7d18ff24-57a3-4b4a-9934-0263191fe2e4 data __DIRECT_IO_TEST__ exports is= os [root@mennode2 test]# One thing I DO notice is the path of my exports is the difference in the pa= th for the hosted_storage domain. I wonder if the second host would have i= ssues resolving this? Data. =3D=3D> menmaster.traindemo.local:/shares/data Export =3D=3D> menmaster.traindemo.local:/shares/exports Hosted_storage. =3D=3D> localhost:/shares ISO. =3D=3D> menmaster.traindemo.local:/shares/exports Below I have copied exports from the VDSM and OVIRT-ENGINE log and have cop= ied the output of the hosted-engine.conf file. Any help in pinpointing the= source of the problem is greatly appreciated!! VSDM logs: 2017-09-27 23:10:01,700-0400 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC = call Host.getStats succeeded in 0.06 seconds (__init__:539) 2017-09-27 23:10:03,104-0400 INFO (jsonrpc/5) [vdsm.api] START getSpmStatu= s(spUUID=3Du'59c7f8f3-0063-00a8-02c7-0000000000f3', options=3DNone) from=3D= ::ffff:10.0.0.227,39748, flow_id=3D19e2dbb3, task_id=3D7a05a5bd-9b15-43be-8= 90d-c6f5d7650e5c (api:46) 2017-09-27 23:10:03,109-0400 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStat= us return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8L}} fro= m=3D::ffff:10.0.0.227,39748, flow_id=3D19e2dbb3, task_id=3D7a05a5bd-9b15-43= be-890d-c6f5d7650e5c (api:52) 2017-09-27 23:10:03,110-0400 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC = call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:539) 2017-09-27 23:10:03,189-0400 INFO (jsonrpc/7) [vdsm.api] START getStorageP= oolInfo(spUUID=3Du'59c7f8f3-0063-00a8-02c7-0000000000f3', options=3DNone) f= rom=3D::ffff:10.0.0.227,39866, flow_id=3D19e2dbb3, task_id=3D0f8b49e9-9d82-= 457e-a2a2-39dc7ed9f022 (api:46) 2017-09-27 23:10:03,196-0400 INFO (jsonrpc/7) [vdsm.api] FINISH getStorage= PoolInfo return=3D{'info': {'name': 'No Description', 'isoprefix': u'/rhev/= data-center/mnt/menmaster.traindemo.local:_shares_isos/da001a29-eca5-44d6-a= 097-129dd9be623f/images/11111111-1111-1111-1111-111111111111', 'pool_status= ': 'connected', 'lver': 8L, 'domains': u'da001a29-eca5-44d6-a097-129dd9be62= 3f:Active,f36157cc-b25a-400a-ab0f-a071e8a8eea7:Active,7d18ff24-57a3-4b4a-99= 34-0263191fe2e4:Active,795d4a1d-3ceb-4773-99de-8e7cf05112f3:Active', 'maste= r_uuid': u'f36157cc-b25a-400a-ab0f-a071e8a8eea7', 'version': '4', 'spm_id':= 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {u'da001a29-eca5-44d6-a097-= 129dd9be623f': {'status': u'Active', 'diskfree': '1044166737920', 'isoprefi= x': u'/rhev/data-center/mnt/menmaster.traindemo.local:_shares_isos/da001a29= -eca5-44d6-a097-129dd9be623f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1049702170624', 'version': 0}, u'f36157cc-b25a-= 400a-ab0f-a071e8a8eea7': {'status': u'Active', 'diskfree': '1044166737920',= 'isoprefix': '', 'alerts': [], 'disktotal': '1049702170624', 'version': 4}= , u'7d18ff24-57a3-4b4a-9934-0263191fe2e4': {'status': u'Active', 'diskfree'= : '1044166737920', 'isoprefix': '', 'alerts': [], 'disktotal': '10497021706= 24', 'version': 4}, u'795d4a1d-3ceb-4773-99de-8e7cf05112f3': {'status': u'A= ctive', 'diskfree': '1044166737920', 'isoprefix': '', 'alerts': [], 'diskto= tal': '1049702170624', 'version': 0}}} from=3D::ffff:10.0.0.227,39866, flow= _id=3D19e2dbb3, task_id=3D0f8b49e9-9d82-457e-a2a2-39dc7ed9f022 (api:52) 2017-09-27 23:10:03,198-0400 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC = call StoragePool.getInfo succeeded in 0.01 seconds (__init__:539) 2017-09-27 23:10:04,212-0400 INFO (Reactor thread) [ProtocolDetector.Accep= torImpl] Accepted connection from ::1:35980 (protocoldetector:72) 2017-09-27 23:10:04,223-0400 INFO (Reactor thread) [ProtocolDetector.Detec= tor] Detected protocol stomp from ::1:35980 (protocoldetector:127) 2017-09-27 23:10:04,224-0400 INFO (Reactor thread) [Broker.StompAdapter] P= rocessing CONNECT request (stompreactor:103) 2017-09-27 23:10:04,225-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAd= apter] Subscribe command received (stompreactor:132) 2017-09-27 23:10:04,622-0400 INFO (monitor/7d18ff2) [storage.SANLock] Acqu= iring host id for domain 7d18ff24-57a3-4b4a-9934-0263191fe2e4 (id=3D1, asyn= c=3DTrue) (clusterlock:288) 2017-09-27 23:10:04,623-0400 ERROR (monitor/7d18ff2) [storage.Monitor] Erro= r acquiring host id 1 for domain 7d18ff24-57a3-4b4a-9934-0263191fe2e4 (moni= tor:570) Traceback (most recent call last): File "/usr/share/vdsm/storage/monitor.py", line 567, in _acquireHostId self.domain.acquireHostId(self.hostId, async=3DTrue) File "/usr/share/vdsm/storage/sd.py", line 785, in acquireHostId self._manifest.acquireHostId(hostId, async) File "/usr/share/vdsm/storage/sd.py", line 449, in acquireHostId self._domainLock.acquireHostId(hostId, async) File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", line= 319, in acquireHostId raise se.AcquireHostIdFailure(self._sdUUID, e) AcquireHostIdFailure: Cannot acquire host id: (u'7d18ff24-57a3-4b4a-9934-02= 63191fe2e4', SanlockException(22, 'Sanlock lockspace add failure', 'Invalid= argument')) 2017-09-27 23:10:04,647-0400 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC = call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539) 2017-09-27 23:10:04,654-0400 INFO (jsonrpc/3) [vdsm.api] START repoStats(o= ptions=3DNone) from=3D::1,35980, task_id=3D9732972d-9a76-4349-9c65-53e57b24= 72d6 (api:46) 2017-09-27 23:10:04,655-0400 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats = return=3D{u'da001a29-eca5-44d6-a097-129dd9be623f': {'code': 0, 'actual': Tr= ue, 'version': 0, 'acquired': True, 'delay': '0.000357054', 'lastCheck': '4= .1', 'valid': True}, u'f36157cc-b25a-400a-ab0f-a071e8a8eea7': {'code': 0, '= actual': True, 'version': 4, 'acquired': True, 'delay': '0.000531828', 'las= tCheck': '4.0', 'valid': True}, u'7d18ff24-57a3-4b4a-9934-0263191fe2e4': {'= code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.0002= 05228', 'lastCheck': '0.0', 'valid': True}, u'795d4a1d-3ceb-4773-99de-8e7cf= 05112f3': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'dela= y': '0.000352165', 'lastCheck': '4.1', 'valid': True}} from=3D::1,35980, ta= sk_id=3D9732972d-9a76-4349-9c65-53e57b2472d6 (api:52) 2017-09-27 23:10:04,656-0400 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC = call Host.getStorageRepoStats succeeded in 0.00 seconds (__init__:539) 2017-09-27 23:10:07,717-0400 INFO (Reactor thread) [ProtocolDetector.Accep= torImpl] Accepted connection from ::1:35982 (protocoldetector:72) 2017-09-27 23:10:07,728-0400 INFO (Reactor thread) [ProtocolDetector.Detec= tor] Detected protocol stomp from ::1:35982 (protocoldetector:127) 2017-09-27 23:10:07,729-0400 INFO (Reactor thread) [Broker.StompAdapter] P= rocessing CONNECT request (stompreactor:103) 2017-09-27 23:10:07,730-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAd= apter] Subscribe command received (stompreactor:132) 2017-09-27 23:10:08,152-0400 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC = call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539) 2017-09-27 23:10:08,159-0400 INFO (jsonrpc/1) [vdsm.api] START repoStats(o= ptions=3DNone) from=3D::1,35982, task_id=3D80eb282b-f297-4f98-a368-7d5627ac= 50f3 (api:46) 2017-09-27 23:10:08,160-0400 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats = return=3D{u'da001a29-eca5-44d6-a097-129dd9be623f': {'code': 0, 'actual': Tr= ue, 'version': 0, 'acquired': True, 'delay': '0.000357054', 'lastCheck': '7= .6', 'valid': True}, u'f36157cc-b25a-400a-ab0f-a071e8a8eea7': {'code': 0, '= actual': True, 'version': 4, 'acquired': True, 'delay': '0.000531828', 'las= tCheck': '7.5', 'valid': True}, u'7d18ff24-57a3-4b4a-9934-0263191fe2e4': {'= code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.0002= 05228', 'lastCheck': '3.5', 'valid': True}, u'795d4a1d-3ceb-4773-99de-8e7cf= 05112f3': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'dela= y': '0.000352165', 'lastCheck': '7.6', 'valid': True}} from=3D::1,35982, ta= sk_id=3D80eb282b-f297-4f98-a368-7d5627ac50f3 (api:52) 2017-09-27 23:10:08,171-0400 INFO (Reactor thread) [ProtocolDetector.Accep= torImpl] Accepted connection from ::1:35984 (protocoldetector:72) 2017-09-27 23:10:08,198-0400 INFO (Reactor thread) [ProtocolDetector.Detec= tor] Detected protocol stomp from ::1:35984 (protocoldetector:127) 2017-09-27 23:10:08,199-0400 INFO (Reactor thread) [Broker.StompAdapter] P= rocessing CONNECT request (stompreactor:103) 2017-09-27 23:10:08,201-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAd= apter] Subscribe command received (stompreactor:132) 2017-09-27 23:10:08,374-0400 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC = call Host.getStats succeeded in 0.22 seconds (__init__:539) 2017-09-27 23:10:08,728-0400 INFO (Reactor thread) [ProtocolDetector.Accep= torImpl] Accepted connection from ::1:35987 (protocoldetector:72) 2017-09-27 23:10:08,732-0400 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC = call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539) 2017-09-27 23:10:08,738-0400 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC = call VM.getStats succeeded in 0.00 seconds (__init__:539) 2017-09-27 23:10:08,756-0400 INFO (Reactor thread) [ProtocolDetector.Detec= tor] Detected protocol stomp from ::1:35987 (protocoldetector:127) 2017-09-27 23:10:08,757-0400 INFO (Reactor thread) [Broker.StompAdapter] P= rocessing CONNECT request (stompreactor:103) 2017-09-27 23:10:08,771-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAd= apter] Subscribe command received (stompreactor:132) 2017-09-27 23:10:09,286-0400 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC = call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539) 2017-09-27 23:10:09,395-0400 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC = call Host.getCapabilities succeeded in 0.10 seconds (__init__:539) 2017-09-27 23:10:10,590-0400 INFO (itmap/0) [IOProcessClient] Starting cli= ent ioprocess-37 (__init__:330) ENGINE LOGS: 2017-09-27 23:07:58,154-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsD= eployBase] (VdsDeploy) [3f898655] Retrieving installation logs to: '/var/lo= g/ovirt-engine/host-deploy/ovirt-host-mgmt-20170927230758-10.0.0.222-3f8986= 55.log' 2017-09-27 23:07:58,316-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsD= eployBase] (VdsDeploy) [3f898655] Stage: Termination 2017-09-27 23:07:58,458-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (pool-7-thread-1) [3f898655] EVENT_ID: HOST_AV= AILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Cust= om ID: null, Custom Event ID: -1, Message: Check for available updates on h= ost mennode2 was completed successfully with message 'no updates found.'. 2017-09-27 23:07:58,874-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsD= eployBase] (VdsDeploy) [3f898655] Stage: Misc configuration 2017-09-27 23:07:58,888-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsD= eployBase] (VdsDeploy) [3f898655] Stage: Transaction commit 2017-09-27 23:07:59,029-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsD= eployBase] (VdsDeploy) [3f898655] Stage: Closing up 2017-09-27 23:07:59,039-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsD= eployBase] (VdsDeploy) [3f898655] Stage: Pre-termination 2017-09-27 23:07:59,092-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsD= eployBase] (VdsDeploy) [3f898655] Retrieving installation logs to: '/var/lo= g/ovirt-engine/host-deploy/ovirt-host-mgmt-20170927230759-menmaster.trainde= mo.local-3f898655.log' 2017-09-27 23:07:59,262-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsD= eployBase] (VdsDeploy) [3f898655] Stage: Termination 2017-09-27 23:07:59,375-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (pool-7-thread-2) [3f898655] EVENT_ID: HOST_AV= AILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Cust= om ID: null, Custom Event ID: -1, Message: Check for available updates on h= ost menmaster was completed successfully with message 'no updates found.'. 2017-09-27 23:08:38,705-04 INFO [org.ovirt.engine.core.bll.ActivateVdsComm= and] (default task-13) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Lock Acquired= to object 'EngineLock:{exclusiveLocks=3D'[0ef7ec10-af19-4afa-980b-77a772c4= e014=3DVDS]', sharedLocks=3D''}' 2017-09-27 23:08:38,776-04 INFO [org.ovirt.engine.core.bll.ActivateVdsComm= and] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f3892= 3f] Running command: ActivateVdsCommand internal: false. Entities affected = : ID: 0ef7ec10-af19-4afa-980b-77a772c4e014 Type: VDSAction group MANIPULAT= E_HOST with role type ADMIN 2017-09-27 23:08:38,777-04 INFO [org.ovirt.engine.core.bll.ActivateVdsComm= and] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f3892= 3f] Before acquiring lock in order to prevent monitoring for host 'mennode2= ' from data-center 'Train1' 2017-09-27 23:08:38,778-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLo= ckManager] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef= 1f38923f] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLo= cks=3D'[0ef7ec10-af19-4afa-980b-77a772c4e014=3DVDS_INIT]', sharedLocks=3D''= }' 2017-09-27 23:08:38,995-04 INFO [org.ovirt.engine.core.bll.ActivateVdsComm= and] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f3892= 3f] Lock acquired, from now a monitoring of host will be skipped for host '= mennode2' from data-center 'Train1' 2017-09-27 23:08:39,003-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsSta= tusVDSCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-= b4ef1f38923f] START, SetVdsStatusVDSCommand(HostName =3D mennode2, SetVdsSt= atusVDSCommandParameters:{runAsync=3D'true', hostId=3D'0ef7ec10-af19-4afa-9= 80b-77a772c4e014', status=3D'Unassigned', nonOperationalReason=3D'NONE', st= opSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id: 78cc1ca= f 2017-09-27 23:08:39,027-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsSta= tusVDSCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-= b4ef1f38923f] FINISH, SetVdsStatusVDSCommand, log id: 78cc1caf 2017-09-27 23:08:39,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsComm= and] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f3892= 3f] Activate host finished. Lock released. Monitoring can run now for host = 'mennode2' from data-center 'Train1' 2017-09-27 23:08:39,050-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [75de01a6-e= fd2-418f-a6b1-b4ef1f38923f] EVENT_ID: VDS_ACTIVATE(16), Correlation ID: 75d= e01a6-efd2-418f-a6b1-b4ef1f38923f, Job ID: 2e40097e-e991-45cc-bd49-3176822e= bb5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Acti= vation of host mennode2 initiated by admin@internal-authz. 2017-09-27 23:08:39,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsComm= and] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f3892= 3f] Lock freed to object 'EngineLock:{exclusiveLocks=3D'[0ef7ec10-af19-4afa= -980b-77a772c4e014=3DVDS]', sharedLocks=3D''}' 2017-09-27 23:08:43,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5437c1b7] START, Get= HardwareInfoVDSCommand(HostName =3D mennode2, VdsIdAndVdsVDSCommandParamete= rsBase:{runAsync=3D'true', hostId=3D'0ef7ec10-af19-4afa-980b-77a772c4e014',= vds=3D'Host[mennode2,0ef7ec10-af19-4afa-980b-77a772c4e014]'}), log id: 2bc= be26e 2017-09-27 23:08:44,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5437c1b7] FINISH, Ge= tHardwareInfoVDSCommand, log id: 2bcbe26e 2017-09-27 23:08:44,186-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFla= gsOrClusterChangedCommand] (DefaultQuartzScheduler7) [12ef27d8] Running com= mand: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities aff= ected : ID: 0ef7ec10-af19-4afa-980b-77a772c4e014 Type: VDS 2017-09-27 23:08:44,214-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpComm= and] (DefaultQuartzScheduler7) [63cdc728] Running command: InitVdsOnUpComma= nd internal: true. Entities affected : ID: 59c7f8f3-0063-00a8-02c7-0000000= 000f3 Type: StoragePool 2017-09-27 23:08:44,234-04 INFO [org.ovirt.engine.core.bll.storage.pool.Co= nnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7670f89e] = Running command: ConnectHostToStoragePoolServersCommand internal: true. Ent= ities affected : ID: 59c7f8f3-0063-00a8-02c7-0000000000f3 Type: StoragePoo= l 2017-09-27 23:08:44,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7670f89e] START= , ConnectStorageServerVDSCommand(HostName =3D mennode2, StorageServerConnec= tionManagementVDSParameters:{runAsync=3D'true', hostId=3D'0ef7ec10-af19-4af= a-980b-77a772c4e014', storagePoolId=3D'59c7f8f3-0063-00a8-02c7-0000000000f3= ', storageType=3D'NFS', connectionList=3D'[StorageServerConnections:{id=3D'= aac63433-88fb-4daf-8c83-871a85b3a49d', connection=3D'menmaster.traindemo.lo= cal:/shares/data', iqn=3D'null', vfsType=3D'null', mountOptions=3D'null', n= fsVersion=3D'AUTO', nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null',= netIfaceName=3D'null'}, StorageServerConnections:{id=3D'5adb7a4b-ebbd-43a4= -b5cb-8c8eb3d6f1ee', connection=3D'localhost:/shares', iqn=3D'null', vfsTyp= e=3D'null', mountOptions=3D'null', nfsVersion=3D'null', nfsRetrans=3D'null'= , nfsTimeo=3D'null', iface=3D'null', netIfaceName=3D'null'}, StorageServerC= onnections:{id=3D'9eb3b1e2-4063-46cd-ba4e-440c4048e92a', connection=3D'menm= aster.traindemo.local:/shares/isos', iqn=3D'null', vfsType=3D'null', mountO= ptions=3D'null', nfsVersion=3D'AUTO', nfsRetrans=3D'null', nfsTimeo=3D'null= ', iface=3D'null', netIfaceName=3D'null'}, StorageServerConnections:{id=3D'= 1c9215a1-4154-4a78-b8ff-565c3a237c58', connection=3D'menmaster.traindemo.lo= cal:/shares/exports', iqn=3D'null', vfsType=3D'null', mountOptions=3D'null'= , nfsVersion=3D'AUTO', nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'nul= l', netIfaceName=3D'null'}]'}), log id: 5625b02f 2017-09-27 23:08:48,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7670f89e] EVENT_ID:= VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: nul= l, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host me= nnode2 to the Storage Domains hosted_storage. 2017-09-27 23:08:48,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7670f89e] FINIS= H, ConnectStorageServerVDSCommand, return: {aac63433-88fb-4daf-8c83-871a85b= 3a49d=3D0, 5adb7a4b-ebbd-43a4-b5cb-8c8eb3d6f1ee=3D477, 9eb3b1e2-4063-46cd-b= a4e-440c4048e92a=3D0, 1c9215a1-4154-4a78-b8ff-565c3a237c58=3D0}, log id: 56= 25b02f 2017-09-27 23:08:48,386-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7670f89e] EVENT_ID:= STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom = ID: null, Custom Event ID: -1, Message: The error message for connection lo= calhost:/shares returned by VDSM was: Problem while trying to mount target 2017-09-27 23:08:48,389-04 ERROR [org.ovirt.engine.core.bll.storage.connect= ion.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7670f89e] The connectio= n with details 'localhost:/shares' failed because of error code '477' and e= rror message is: problem while trying to mount target 2017-09-27 23:08:48,401-04 INFO [org.ovirt.engine.core.bll.storage.pool.Co= nnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7670f89e] = Host 'mennode2' storage connection was failed 2017-09-27 23:08:48,411-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7670f89e] EVENT_ID:= CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7670f89e, Call Stack:= null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Hos= t mennode2 to Storage Servers 2017-09-27 23:08:48,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START,= ConnectStoragePoolVDSCommand(HostName =3D mennode2, ConnectStoragePoolVDSC= ommandParameters:{runAsync=3D'true', hostId=3D'0ef7ec10-af19-4afa-980b-77a7= 72c4e014', vdsId=3D'0ef7ec10-af19-4afa-980b-77a772c4e014', storagePoolId=3D= '59c7f8f3-0063-00a8-02c7-0000000000f3', masterVersion=3D'1'}), log id: 3d5c= 5d79 2017-09-27 23:08:48,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Execut= ing with domain map: {7d18ff24-57a3-4b4a-9934-0263191fe2e4=3Dactive, da001a= 29-eca5-44d6-a097-129dd9be623f=3Dactive, 795d4a1d-3ceb-4773-99de-8e7cf05112= f3=3Dactive, f36157cc-b25a-400a-ab0f-a071e8a8eea7=3Dactive} 2017-09-27 23:08:49,656-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH= , ConnectStoragePoolVDSCommand, log id: 3d5c5d79 2017-09-27 23:08:50,681-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker= .IrsProxy] (org.ovirt.thread.pool-6-thread-8) [] Domain '7d18ff24-57a3-4b4a= -9934-0263191fe2e4:hosted_storage' was reported with error code '358' 2017-09-27 23:08:50,682-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpComm= and] (org.ovirt.thread.pool-6-thread-8) [] Storage Domain 'hosted_storage' = of pool 'Train1' is in problem in host 'mennode2' 2017-09-27 23:08:50,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID= : VDS_STORAGE_VDS_STATS_FAILED(189), Correlation ID: 63cdc728, Call Stack: = null, Custom ID: null, Custom Event ID: -1, Message: Host mennode2 reports = about one of the Active Storage Domains as Problematic. 2017-09-27 23:08:50,726-04 INFO [org.ovirt.engine.core.bll.SetNonOperation= alVdsCommand] (DefaultQuartzScheduler7) [748ef73f] Running command: SetNonO= perationalVdsCommand internal: true. Entities affected : ID: 0ef7ec10-af19= -4afa-980b-77a772c4e014 Type: VDS 2017-09-27 23:08:50,729-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsSta= tusVDSCommand] (DefaultQuartzScheduler7) [748ef73f] START, SetVdsStatusVDSC= ommand(HostName =3D mennode2, SetVdsStatusVDSCommandParameters:{runAsync=3D= 'true', hostId=3D'0ef7ec10-af19-4afa-980b-77a772c4e014', status=3D'NonOpera= tional', nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHABLE', stopSpmFailur= eLogged=3D'false', maintenanceReason=3D'null'}), log id: 2589acd2 2017-09-27 23:08:50,736-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsSta= tusVDSCommand] (DefaultQuartzScheduler7) [748ef73f] FINISH, SetVdsStatusVDS= Command, log id: 2589acd2 2017-09-27 23:08:50,807-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (DefaultQuartzScheduler7) [748ef73f] EVENT_ID:= VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 748ef73f, Job ID: e809= 41e4-3dbf-4159-8421-96d2d90160ca, Call Stack: null, Custom ID: null, Custom= Event ID: -1, Message: Host mennode2 cannot access the Storage Domain(s) h= osted_storage attached to the Data Center Train1. Setting Host state to Non= -Operational. 2017-09-27 23:08:50,820-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (DefaultQuartzScheduler7) [748ef73f] EVENT_ID:= VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 63cdc728, Call S= tack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify= Power Management configuration for Host mennode2. 2017-09-27 23:08:50,826-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (DefaultQuartzScheduler7) [748ef73f] EVENT_ID:= CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 63cdc728, Call Stack: nu= ll, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host m= ennode2 to Storage Pool Train1 2017-09-27 23:08:50,856-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersio= nCommand] (DefaultQuartzScheduler7) [42cbaee1] Running command: HandleVdsVe= rsionCommand internal: true. Entities affected : ID: 0ef7ec10-af19-4afa-98= 0b-77a772c4e014 Type: VDS 2017-09-27 23:08:50,863-04 INFO [org.ovirt.engine.core.vdsbroker.monitorin= g.HostMonitoring] (DefaultQuartzScheduler7) [42cbaee1] Host 'mennode2'(0ef7= ec10-af19-4afa-980b-77a772c4e014) is already in NonOperational status for r= eason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped= . /etc/ovirt-hosted-engine/hosted-engine.conf fqdn=3Dengine.traindemo.local vm_disk_id=3D0077cb58-7abe-4bdf-96b1-55cc85030d76 vm_disk_vol_id=3Dd5bd2690-7cfb-4aa2-b898-41579114ec08 vmid=3D564d290f-ae66-4352-9770-f2e062000358 storage=3Dmenmaster.traindemo.local:/shares mnt_options=3D conf=3D/var/run/ovirt-hosted-engine-ha/vm.conf host_id=3D1 console=3Dvnc domainType=3Dnfs3 spUUID=3D00000000-0000-0000-0000-000000000000 sdUUID=3D7d18ff24-57a3-4b4a-9934-0263191fe2e4 connectionUUID=3Daddfd5fc-8668-4d2a-9b0b-f801b21a98d8 ca_cert=3D/etc/pki/vdsm/libvirt-spice/ca-cert.pem ca_subject=3D"C=3DEN, L=3DTest, O=3DTest, CN=3DTest" vdsm_use_ssl=3Dtrue gateway=3D10.0.0.1 bridge=3Dovirtmgmt metadata_volume_UUID=3D156c7549-8e2e-4580-9825-81c3b13387ce metadata_image_UUID=3Dc5db06ac-ea37-47bc-9065-b94eb230616a lockspace_volume_UUID=3D5f4f4a63-5608-4daf-a034-2b8e83b5de42 lockspace_image_UUID=3D0d7f8661-3abd-44e8-87a7-3b85f2e29fda conf_volume_UUID=3Dddffd499-294c-456f-bb0e-08e366380a22 conf_image_UUID=3D5f264ea1-4654-4e40-87aa-80fcaacc60d8 Thanks, Alex Witte On Sep 27, 2017, at 8:32 PM, Alexander Witte <alexander.witte@baicanada.com= <mailto:alexander.witte@baicanada.com>> wrote: /etc/ovirt-hosted-engine/hosted-engine.conf --_000_85068B30E9EA40D5883D6574AC1EC5FBbaicanadacom_ Content-Type: text/html; charset="us-ascii" Content-ID: <4C836429D3E8524F9E2744C6E02DEC5D@baicanada.local> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
</head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin= e-break: after-white-space;" class=3D""> <div class=3D"">OK after a host reboot I was able to get the Engine VM up a= gain and into the Web interface. Although continually when I try a se= cond host to the datacenter within oVirt I run into the error:</div> <div class=3D""><br class=3D""> </div> <div class=3D"">"Host mennode2 cannot access the Storage Domain(s) hos= ted_storage attached to the Data Center Train1. Setting Host state to= Non Operational."</div> <div class=3D""><br class=3D""> </div> <div class=3D"">Note: I can successfully read (and mount) the NFS exp= orts oVirt is complaining about:</div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <i class=3D"">[root@mennode2 ~]# showmount -e menmaster.traindemo.local</i>= </div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <i class=3D"">Export list for menmaster.traindemo.local:</i></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <i class=3D"">/shares *</i></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <i class=3D"">/shares/exports *</i></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <i class=3D"">/shares/data *</i></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <i class=3D"">/shares/isos *</i></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <i class=3D"">[root@mennode2 ~]# </i></div> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> [root@mennode2 tmp]# mount -t nfs menmaster.traindemo.local:/shares test</d= iv> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> [root@mennode2 tmp]# cd test</div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> [root@mennode2 test]# ls</div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> 7d18ff24-57a3-4b4a-9934-0263191fe2e4 data __DIRECT_IO_TEST__&nb= sp; exports isos</div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> [root@mennode2 test]# </div> </div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <br class=3D""> </div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> One thing I DO notice is the path of my exports is the difference in the pa= th for the hosted_storage domain. I wonder if the second host would h= ave issues resolving this?</div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <br class=3D""> </div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> Data. =3D=3D> <span style=3D"color: rgb(119, 119, 119); font-family= : Arial, sans-serif; font-size: small; background-color: rgb(255, 255, 255)= ;" class=3D"">menmaster.traindemo.local:/shares/data</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> Export =3D=3D> <span style=3D"color: rgb(119, 119, 119); font= -family: Arial, sans-serif; font-size: small; background-color: rgb(255, 25= 5, 255);" class=3D"">menmaster.traindemo.local:/shares/exports</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> Hosted_storage. =3D=3D> <span style=3D"color: rgb(119, 119, 119); f= ont-family: Arial, sans-serif; font-size: small; background-color: rgb(255,= 255, 255);" class=3D"">localhost:/shares</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> ISO. =3D=3D> <span style=3D"color: rgb(119, 119, 119); font-family:= Arial, sans-serif; font-size: small; background-color: rgb(255, 255, 255);= " class=3D"">menmaster.traindemo.local:/shares/exports</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">Below I have copied exports from the VDSM and OVIRT-ENGINE = log and have copied the output of the hosted-engine.conf file. Any he= lp in pinpointing the source of the problem is greatly appreciated!!</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D""><b class=3D"">VSDM logs:</b></div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:01,700-0400 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC= call Host.getStats succeeded in 0.06 seconds (__init__:539)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:03,104-0400 INFO (jsonrpc/5) [vdsm.api] START getSpmStat= us(spUUID=3Du'59c7f8f3-0063-00a8-02c7-0000000000f3', options=3DNone) from= =3D::ffff:10.0.0.227,39748, flow_id=3D19e2dbb3, task_id=3D7a05a5bd-9b15-43b= e-890d-c6f5d7650e5c (api:46)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:03,109-0400 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmSta= tus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8L}} fr= om=3D::ffff:10.0.0.227,39748, flow_id=3D19e2dbb3, task_id=3D7a05a5bd-9b15-43be-890d-c6f5d7650e5c (api:52)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:03,110-0400 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC= call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:539)</sp= an></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:03,189-0400 INFO (jsonrpc/7) [vdsm.api] START getStorage= PoolInfo(spUUID=3Du'59c7f8f3-0063-00a8-02c7-0000000000f3', options=3DNone) = from=3D::ffff:10.0.0.227,39866, flow_id=3D19e2dbb3, task_id=3D0f8b49e9-9d82-457e-a2a2-39dc7ed9f022 (api:46)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:03,196-0400 INFO (jsonrpc/7) [vdsm.api] FINISH getStorag= ePoolInfo return=3D{'info': {'name': 'No Description', 'isoprefix': u'/rhev= /data-center/mnt/menmaster.traindemo.local:_shares_isos/da001a29-eca5-44d6-= a097-129dd9be623f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 8L, 'domains': u'da001a29-eca5-44d6-a0= 97-129dd9be623f:Active,f36157cc-b25a-400a-ab0f-a071e8a8eea7:Active,7d18ff24= -57a3-4b4a-9934-0263191fe2e4:Active,795d4a1d-3ceb-4773-99de-8e7cf05112f3:Ac= tive', 'master_uuid': u'f36157cc-b25a-400a-ab0f-a071e8a8eea7', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {= u'da001a29-eca5-44d6-a097-129dd9be623f': {'status': u'Active', 'diskfree': = '1044166737920', 'isoprefix': u'/rhev/data-center/mnt/menmaster.traindemo.l= ocal:_shares_isos/da001a29-eca5-44d6-a097-129dd9be623f/images/11111111-1111= -1111-1111-111111111111', 'alerts': [], 'disktotal': '1049702170624', 'version': 0}, u'f36157cc-b25a= -400a-ab0f-a071e8a8eea7': {'status': u'Active', 'diskfree': '1044166737920'= , 'isoprefix': '', 'alerts': [], 'disktotal': '1049702170624', 'version': 4= }, u'7d18ff24-57a3-4b4a-9934-0263191fe2e4': {'status': u'Active', 'diskfree': '1044166737920', 'isoprefix': '', 'alert= s': [], 'disktotal': '1049702170624', 'version': 4}, u'795d4a1d-3ceb-4773-9= 9de-8e7cf05112f3': {'status': u'Active', 'diskfree': '1044166737920', 'isop= refix': '', 'alerts': [], 'disktotal': '1049702170624', 'version': 0}}} from=3D::ffff:10.0.0.227,39866, flow_id= =3D19e2dbb3, task_id=3D0f8b49e9-9d82-457e-a2a2-39dc7ed9f022 (api:52)</span>= </div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:03,198-0400 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC= call StoragePool.getInfo succeeded in 0.01 seconds (__init__:539)</span></= div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:04,212-0400 INFO (Reactor thread) [ProtocolDetector.Acce= ptorImpl] Accepted connection from ::1:35980 (protocoldetector:72)</span></= div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:04,223-0400 INFO (Reactor thread) [ProtocolDetector.Dete= ctor] Detected protocol stomp from ::1:35980 (protocoldetector:127)</span><= /div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:04,224-0400 INFO (Reactor thread) [Broker.StompAdapter] = Processing CONNECT request (stompreactor:103)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:04,225-0400 INFO (JsonRpc (StompReactor)) [Broker.StompA= dapter] Subscribe command received (stompreactor:132)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:04,622-0400 INFO (monitor/7d18ff2) [storage.SANLock] Acq= uiring host id for domain 7d18ff24-57a3-4b4a-9934-0263191fe2e4 (id=3D1, asy= nc=3DTrue) (clusterlock:288)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:04,623-0400 ERROR (monitor/7d18ff2) [storage.Monitor] Error ac= quiring host id 1 for domain 7d18ff24-57a3-4b4a-9934-0263191fe2e4 (monitor:= 570)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">Trac= eback (most recent call last):</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">&nbs= p; File "/usr/share/vdsm/storage/monitor.py", line 567, in _acqui= reHostId</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">&nbs= p; self.domain.acquireHostId(self.hostId, async=3DTrue)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">&nbs= p; File "/usr/share/vdsm/storage/sd.py", line 785, in acquireHost= Id</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">&nbs= p; self._manifest.acquireHostId(hostId, async)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">&nbs= p; File "/usr/share/vdsm/storage/sd.py", line 449, in acquireHost= Id</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">&nbs= p; self._domainLock.acquireHostId(hostId, async)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">&nbs= p; File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py&= quot;, line 319, in acquireHostId</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">&nbs= p; raise se.AcquireHostIdFailure(self._sdUUID, e)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">Acqu= ireHostIdFailure: Cannot acquire host id: (u'7d18ff24-57a3-4b4a-9934-026319= 1fe2e4', SanlockException(22, 'Sanlock lockspace add failure', 'Invalid arg= ument'))</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:04,647-0400 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC= call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539)</span><= /div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:04,654-0400 INFO (jsonrpc/3) [vdsm.api] START repoStats(= options=3DNone) from=3D::1,35980, task_id=3D9732972d-9a76-4349-9c65-53e57b2= 472d6 (api:46)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:04,655-0400 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats= return=3D{u'da001a29-eca5-44d6-a097-129dd9be623f': {'code': 0, 'actual': T= rue, 'version': 0, 'acquired': True, 'delay': '0.000357054', 'lastCheck': '4.1', 'valid': True}, u'f36157cc-b25a-400a-ab= 0f-a071e8a8eea7': {'code': 0, 'actual': True, 'version': 4, 'acquired': Tru= e, 'delay': '0.000531828', 'lastCheck': '4.0', 'valid': True}, u'7d18ff24-5= 7a3-4b4a-9934-0263191fe2e4': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000205228'= , 'lastCheck': '0.0', 'valid': True}, u'795d4a1d-3ceb-4773-99de-8e7cf05112f= 3': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0= .000352165', 'lastCheck': '4.1', 'valid': True}} from=3D::1,35980, task_id=3D9732972d-9a76-4349-9c65-53e57b= 2472d6 (api:52)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:04,656-0400 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC= call Host.getStorageRepoStats succeeded in 0.00 seconds (__init__:539)</sp= an></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:07,717-0400 INFO (Reactor thread) [ProtocolDetector.Acce= ptorImpl] Accepted connection from ::1:35982 (protocoldetector:72)</span></= div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:07,728-0400 INFO (Reactor thread) [ProtocolDetector.Dete= ctor] Detected protocol stomp from ::1:35982 (protocoldetector:127)</span><= /div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:07,729-0400 INFO (Reactor thread) [Broker.StompAdapter] = Processing CONNECT request (stompreactor:103)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:07,730-0400 INFO (JsonRpc (StompReactor)) [Broker.StompA= dapter] Subscribe command received (stompreactor:132)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,152-0400 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC= call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539)</span><= /div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,159-0400 INFO (jsonrpc/1) [vdsm.api] START repoStats(= options=3DNone) from=3D::1,35982, task_id=3D80eb282b-f297-4f98-a368-7d5627a= c50f3 (api:46)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,160-0400 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats= return=3D{u'da001a29-eca5-44d6-a097-129dd9be623f': {'code': 0, 'actual': T= rue, 'version': 0, 'acquired': True, 'delay': '0.000357054', 'lastCheck': '7.6', 'valid': True}, u'f36157cc-b25a-400a-ab= 0f-a071e8a8eea7': {'code': 0, 'actual': True, 'version': 4, 'acquired': Tru= e, 'delay': '0.000531828', 'lastCheck': '7.5', 'valid': True}, u'7d18ff24-5= 7a3-4b4a-9934-0263191fe2e4': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000205228'= , 'lastCheck': '3.5', 'valid': True}, u'795d4a1d-3ceb-4773-99de-8e7cf05112f= 3': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0= .000352165', 'lastCheck': '7.6', 'valid': True}} from=3D::1,35982, task_id=3D80eb282b-f297-4f98-a368-7d5627= ac50f3 (api:52)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,171-0400 INFO (Reactor thread) [ProtocolDetector.Acce= ptorImpl] Accepted connection from ::1:35984 (protocoldetector:72)</span></= div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,198-0400 INFO (Reactor thread) [ProtocolDetector.Dete= ctor] Detected protocol stomp from ::1:35984 (protocoldetector:127)</span><= /div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,199-0400 INFO (Reactor thread) [Broker.StompAdapter] = Processing CONNECT request (stompreactor:103)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,201-0400 INFO (JsonRpc (StompReactor)) [Broker.StompA= dapter] Subscribe command received (stompreactor:132)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,374-0400 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC= call Host.getStats succeeded in 0.22 seconds (__init__:539)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,728-0400 INFO (Reactor thread) [ProtocolDetector.Acce= ptorImpl] Accepted connection from ::1:35987 (protocoldetector:72)</span></= div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,732-0400 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC= call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539)</span><= /div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,738-0400 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC= call VM.getStats succeeded in 0.00 seconds (__init__:539)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,756-0400 INFO (Reactor thread) [ProtocolDetector.Dete= ctor] Detected protocol stomp from ::1:35987 (protocoldetector:127)</span><= /div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,757-0400 INFO (Reactor thread) [Broker.StompAdapter] = Processing CONNECT request (stompreactor:103)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:08,771-0400 INFO (JsonRpc (StompReactor)) [Broker.StompA= dapter] Subscribe command received (stompreactor:132)</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:09,286-0400 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC= call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539)</span><= /div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:09,395-0400 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC= call Host.getCapabilities succeeded in 0.10 seconds (__init__:539)</span><= /div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:10:10,590-0400 INFO (itmap/0) [IOProcessClient] Starting cl= ient ioprocess-37 (__init__:330)</span></div> </div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <br class=3D""> </div> </div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <br class=3D""> </div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <b class=3D"">ENGINE LOGS:</b></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <br class=3D""> </div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Helvetica Neu= e'; color: rgb(69, 69, 69);" class=3D""> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:07:58,154-04 INFO [org.ovirt.engine.core.bll.hostdeploy.Vds= DeployBase] (VdsDeploy) [3f898655] Retrieving installation logs to: '/var/l= og/ovirt-engine/host-deploy/ovirt-host-mgmt-20170927230758-10.0.0.222-3f898= 655.log'</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:07:58,316-04 INFO [org.ovirt.engine.core.bll.hostdeploy.Vds= DeployBase] (VdsDeploy) [3f898655] Stage: Termination</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:07:58,458-04 INFO [org.ovirt.engine.core.dal.dbbroker.audit= loghandling.AuditLogDirector] (pool-7-thread-1) [3f898655] EVENT_ID: HOST_A= VAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message:= Check for available updates on host mennode2 was completed successfully wi= th message 'no updates found.'.</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:07:58,874-04 INFO [org.ovirt.engine.core.bll.hostdeploy.Vds= DeployBase] (VdsDeploy) [3f898655] Stage: Misc configuration</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:07:58,888-04 INFO [org.ovirt.engine.core.bll.hostdeploy.Vds= DeployBase] (VdsDeploy) [3f898655] Stage: Transaction commit</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:07:59,029-04 INFO [org.ovirt.engine.core.bll.hostdeploy.Vds= DeployBase] (VdsDeploy) [3f898655] Stage: Closing up</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:07:59,039-04 INFO [org.ovirt.engine.core.bll.hostdeploy.Vds= DeployBase] (VdsDeploy) [3f898655] Stage: Pre-termination</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:07:59,092-04 INFO [org.ovirt.engine.core.bll.hostdeploy.Vds= DeployBase] (VdsDeploy) [3f898655] Retrieving installation logs to: '/var/l= og/ovirt-engine/host-deploy/ovirt-host-mgmt-20170927230759-menmaster.traind= emo.local-3f898655.log'</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:07:59,262-04 INFO [org.ovirt.engine.core.bll.hostdeploy.Vds= DeployBase] (VdsDeploy) [3f898655] Stage: Termination</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:07:59,375-04 INFO [org.ovirt.engine.core.dal.dbbroker.audit= loghandling.AuditLogDirector] (pool-7-thread-2) [3f898655] EVENT_ID: HOST_A= VAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message:= Check for available updates on host menmaster was completed successfully w= ith message 'no updates found.'.</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:38,705-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCom= mand] (default task-13) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Lock Acquire= d to object 'EngineLock:{exclusiveLocks=3D'[0ef7ec10-af19-4afa-980b-77a772c= 4e014=3DVDS]', sharedLocks=3D''}'</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:38,776-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCom= mand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f389= 23f] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 0ef7ec10-af19-4afa-980b-77a= 772c4e014 Type: VDSAction group MANIPULATE_HOST with role type ADMIN</span>= </div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:38,777-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCom= mand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f389= 23f] Before acquiring lock in order to prevent monitoring for host 'mennode2' from data-center 'Train1'</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:38,778-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryL= ockManager] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4e= f1f38923f] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks=3D'[0ef7ec10-af19-4afa-980b-77a772c4e= 014=3DVDS_INIT]', sharedLocks=3D''}'</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:38,995-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCom= mand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f389= 23f] Lock acquired, from now a monitoring of host will be skipped for host 'mennode2' from data-center 'Train1'</spa= n></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:39,003-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsSt= atusVDSCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1= -b4ef1f38923f] START, SetVdsStatusVDSCommand(HostName =3D mennode2, SetVdsStatusVDSCommandParameters:{runAsync=3D'true', hostId= =3D'0ef7ec10-af19-4afa-980b-77a772c4e014', status=3D'Unassigned', nonOperat= ionalReason=3D'NONE', stopSpmFailureLogged=3D'false', maintenanceReason=3D'= null'}), log id: 78cc1caf</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:39,027-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsSt= atusVDSCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1= -b4ef1f38923f] FINISH, SetVdsStatusVDSCommand, log id: 78cc1caf</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:39,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCom= mand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f389= 23f] Activate host finished. Lock released. Monitoring can run now for host 'mennode2' from data-center 'Train1'</span=
</div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';=
color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:39,050-04 INFO [org.ovirt.engine.core.dal.dbbroker.audit= loghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [75de01a6-= efd2-418f-a6b1-b4ef1f38923f] EVENT_ID: VDS_ACTIVATE(16), Correlation ID: 75de01a6-efd2-418f-a6b1-b4ef1f38923f, Job ID: 2e40097e-e99= 1-45cc-bd49-3176822ebb5a, Call Stack: null, Custom ID: null, Custom Event I= D: -1, Message: Activation of host mennode2 initiated by admin@internal-aut= hz.</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:39,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCom= mand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f389= 23f] Lock freed to object 'EngineLock:{exclusiveLocks=3D'[0ef7ec10-af19-4af= a-980b-77a772c4e014=3DVDS]', sharedLocks=3D''}'</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:43,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroke= r.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5437c1b7] START, Ge= tHardwareInfoVDSCommand(HostName =3D mennode2, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId=3D'0ef7ec10= -af19-4afa-980b-77a772c4e014', vds=3D'Host[mennode2,0ef7ec10-af19-4afa-980b= -77a772c4e014]'}), log id: 2bcbe26e</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:44,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroke= r.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5437c1b7] FINISH, G= etHardwareInfoVDSCommand, log id: 2bcbe26e</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:44,186-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFl= agsOrClusterChangedCommand] (DefaultQuartzScheduler7) [12ef27d8] Running co= mmand: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 0ef7ec10-af19-4afa-980b-77a7= 72c4e014 Type: VDS</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:44,214-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCom= mand] (DefaultQuartzScheduler7) [63cdc728] Running command: InitVdsOnUpComm= and internal: true. Entities affected : ID: 59c7f8f3-0063-00a8-02c7-0000000000f3 Type: StoragePool</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:44,234-04 INFO [org.ovirt.engine.core.bll.storage.pool.C= onnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7670f89e]= Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 59c7f8f3-0063-00a8-02c7-0000= 000000f3 Type: StoragePool</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:44,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroke= r.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7670f89e] STAR= T, ConnectStorageServerVDSCommand(HostName =3D mennode2, StorageServerConnectionManagementVDSParameters:{runAsync=3D'= true', hostId=3D'0ef7ec10-af19-4afa-980b-77a772c4e014', storagePoolId=3D'59= c7f8f3-0063-00a8-02c7-0000000000f3', storageType=3D'NFS', connectionList=3D= '[StorageServerConnections:{id=3D'aac63433-88fb-4daf-8c83-871a85b3a49d', connection=3D'menmaster.traindemo.local:/shares/data', iqn=3D'null', vfsTy= pe=3D'null', mountOptions=3D'null', nfsVersion=3D'AUTO', nfsRetrans=3D'null= ', nfsTimeo=3D'null', iface=3D'null', netIfaceName=3D'null'}, StorageServer= Connections:{id=3D'5adb7a4b-ebbd-43a4-b5cb-8c8eb3d6f1ee', connection=3D'localhost:/shares', iqn=3D'null', vfsType=3D'null', mountOpt= ions=3D'null', nfsVersion=3D'null', nfsRetrans=3D'null', nfsTimeo=3D'null',= iface=3D'null', netIfaceName=3D'null'}, StorageServerConnections:{id=3D'9e= b3b1e2-4063-46cd-ba4e-440c4048e92a', connection=3D'menmaster.traindemo.loca= l:/shares/isos', iqn=3D'null', vfsType=3D'null', mountOptions=3D'null', nfsVersion=3D'AUTO'= , nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null', netIfaceName=3D'n= ull'}, StorageServerConnections:{id=3D'1c9215a1-4154-4a78-b8ff-565c3a237c58= ', connection=3D'menmaster.traindemo.local:/shares/exports', iqn=3D'null', vfsType=3D'null', mountOptions=3D'null', nfsVersion=3D'AUTO'= , nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null', netIfaceName=3D'n= ull'}]'}), log id: 5625b02f</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:48,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditlogha= ndling.AuditLogDirector] (DefaultQuartzScheduler7) [7670f89e] EVENT_ID: VDS= _STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message:= Failed to connect Host mennode2 to the Storage Domains hosted_storage.</sp= an></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:48,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroke= r.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7670f89e] FINI= SH, ConnectStorageServerVDSCommand, return: {aac63433-88fb-4daf-8c83-871a85b3a49d=3D0, 5adb7a4b-ebbd-43a4-b5cb-8c8eb3d= 6f1ee=3D477, 9eb3b1e2-4063-46cd-ba4e-440c4048e92a=3D0, 1c9215a1-4154-4a78-b= 8ff-565c3a237c58=3D0}, log id: 5625b02f</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:48,386-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditlogha= ndling.AuditLogDirector] (DefaultQuartzScheduler7) [7670f89e] EVENT_ID: STO= RAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message:= The error message for connection localhost:/shares returned by VDSM was: P= roblem while trying to mount target</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:48,389-04 ERROR [org.ovirt.engine.core.bll.storage.connection.= BaseFsStorageHelper] (DefaultQuartzScheduler7) [7670f89e] The connection wi= th details 'localhost:/shares' failed because of error code '477' and error message is: problem while trying to = mount target</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:48,401-04 INFO [org.ovirt.engine.core.bll.storage.pool.C= onnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7670f89e]= Host 'mennode2' storage connection was failed </span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:48,411-04 WARN [org.ovirt.engine.core.dal.dbbroker.audit= loghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7670f89e] EVENT_ID= : CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7670f89e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Mess= age: Failed to connect Host mennode2 to Storage Servers</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:48,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroke= r.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START= , ConnectStoragePoolVDSCommand(HostName =3D mennode2, ConnectStoragePoolVDSCommandParameters:{runAsync=3D'true', hostI= d=3D'0ef7ec10-af19-4afa-980b-77a772c4e014', vdsId=3D'0ef7ec10-af19-4afa-980= b-77a772c4e014', storagePoolId=3D'59c7f8f3-0063-00a8-02c7-0000000000f3', ma= sterVersion=3D'1'}), log id: 3d5c5d79</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:48,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroke= r.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Execu= ting with domain map: {7d18ff24-57a3-4b4a-9934-0263191fe2e4=3Dactive, da001a29-eca5-44d6-a097-129dd9be623f=3Dactive, 795d4a1d-3ceb-4773-99de-8e7= cf05112f3=3Dactive, f36157cc-b25a-400a-ab0f-a071e8a8eea7=3Dactive}</span></= div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:49,656-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroke= r.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINIS= H, ConnectStoragePoolVDSCommand, log id: 3d5c5d79</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:50,681-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.Irs= Proxy] (org.ovirt.thread.pool-6-thread-8) [] Domain '7d18ff24-57a3-4b4a-993= 4-0263191fe2e4:hosted_storage' was reported with error code '358'</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:50,682-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand]= (org.ovirt.thread.pool-6-thread-8) [] Storage Domain 'hosted_storage' of p= ool 'Train1' is in problem in host 'mennode2'</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:50,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditlogha= ndling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VD= S_STORAGE_VDS_STATS_FAILED(189), Correlation ID: 63cdc728, Call Stack: null, Custom ID: null, Custom Event ID: -1, Mess= age: Host mennode2 reports about one of the Active Storage Domains as Probl= ematic.</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:50,726-04 INFO [org.ovirt.engine.core.bll.SetNonOperatio= nalVdsCommand] (DefaultQuartzScheduler7) [748ef73f] Running command: SetNon= OperationalVdsCommand internal: true. Entities affected : ID: 0ef7ec10-af19-4afa-980b-77a772c4e014 Type: VDS</span>= </div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:50,729-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsSt= atusVDSCommand] (DefaultQuartzScheduler7) [748ef73f] START, SetVdsStatusVDS= Command(HostName =3D mennode2, SetVdsStatusVDSCommandParameters:{runAsync= =3D'true', hostId=3D'0ef7ec10-af19-4afa-980b-77a772c4e014', status=3D'NonOperational'= , nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged= =3D'false', maintenanceReason=3D'null'}), log id: 2589acd2</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:50,736-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsSt= atusVDSCommand] (DefaultQuartzScheduler7) [748ef73f] FINISH, SetVdsStatusVD= SCommand, log id: 2589acd2</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:50,807-04 WARN [org.ovirt.engine.core.dal.dbbroker.audit= loghandling.AuditLogDirector] (DefaultQuartzScheduler7) [748ef73f] EVENT_ID= : VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 748ef73f, Job ID: e80941e4-3dbf-4159-8421-96d2d90160ca, Call Stack: nu= ll, Custom ID: null, Custom Event ID: -1, Message: Host mennode2 cannot acc= ess the Storage Domain(s) hosted_storage attached to the Data Center Train1= . Setting Host state to Non-Operational.</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:50,820-04 WARN [org.ovirt.engine.core.dal.dbbroker.audit= loghandling.AuditLogDirector] (DefaultQuartzScheduler7) [748ef73f] EVENT_ID= : VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 63cdc728, Call Stack: null, Custom ID: null, Custom Event = ID: -1, Message: Failed to verify Power Management configuration for Host m= ennode2.</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:50,826-04 WARN [org.ovirt.engine.core.dal.dbbroker.audit= loghandling.AuditLogDirector] (DefaultQuartzScheduler7) [748ef73f] EVENT_ID= : CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 63cdc728, Call Stack: null, Custom ID: null, Custom Event ID: -1, Mess= age: Failed to connect Host mennode2 to Storage Pool Train1</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:50,856-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersi= onCommand] (DefaultQuartzScheduler7) [42cbaee1] Running command: HandleVdsV= ersionCommand internal: true. Entities affected : ID: 0ef7ec10-af19-4afa-980b-77a772c4e014 Type: VDS</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">2017= -09-27 23:08:50,863-04 INFO [org.ovirt.engine.core.vdsbroker.monitori= ng.HostMonitoring] (DefaultQuartzScheduler7) [42cbaee1] Host 'mennode2'(0ef= 7ec10-af19-4afa-980b-77a772c4e014) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOp= erationalVds command is skipped.</span></div> </div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <blockquote type=3D"cite" class=3D""><b class=3D""><span class=3D"" style= =3D"float: none; display: inline !important;">/</span><span class=3D"" styl= e=3D"color: rgb(69, 69, 69); font-family: 'Helvetica Neue';">etc/ovirt-host= ed-engine/hosted-engine.conf </span></b></blockquote> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">fqdn= =3Dengine.traindemo.local</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">vm_d= isk_id=3D0077cb58-7abe-4bdf-96b1-55cc85030d76</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">vm_d= isk_vol_id=3Dd5bd2690-7cfb-4aa2-b898-41579114ec08</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">vmid= =3D564d290f-ae66-4352-9770-f2e062000358</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">stor= age=3Dmenmaster.traindemo.local:/shares</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">mnt_= options=3D</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">conf= =3D/var/run/ovirt-hosted-engine-ha/vm.conf</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">host= _id=3D1</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">cons= ole=3Dvnc</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">doma= inType=3Dnfs3</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">spUU= ID=3D00000000-0000-0000-0000-000000000000</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">sdUU= ID=3D7d18ff24-57a3-4b4a-9934-0263191fe2e4</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">conn= ectionUUID=3Daddfd5fc-8668-4d2a-9b0b-f801b21a98d8</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">ca_c= ert=3D/etc/pki/vdsm/libvirt-spice/ca-cert.pem</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">ca_s= ubject=3D"C=3DEN, L=3DTest, O=3DTest, CN=3DTest"</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">vdsm= _use_ssl=3Dtrue</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">gate= way=3D10.0.0.1</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">brid= ge=3Dovirtmgmt</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">meta= data_volume_UUID=3D156c7549-8e2e-4580-9825-81c3b13387ce</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">meta= data_image_UUID=3Dc5db06ac-ea37-47bc-9065-b94eb230616a</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">lock= space_volume_UUID=3D5f4f4a63-5608-4daf-a034-2b8e83b5de42</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">lock= space_image_UUID=3D0d7f8661-3abd-44e8-87a7-3b85f2e29fda</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">conf= _volume_UUID=3Dddffd499-294c-456f-bb0e-08e366380a22</span></div> <div style=3D"margin: 0px; line-height: normal; font-family: 'Andale Mono';= color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class= =3D""> <span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">conf= _image_UUID=3D5f264ea1-4654-4e40-87aa-80fcaacc60d8</span></div> </div> <div class=3D"">Thanks,</div> <div class=3D""><br class=3D""> </div> <br class=3D""> <div class=3D""> <div style=3D"color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px;= font-style: normal; font-variant-caps: normal; font-weight: normal; letter= -spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-= transform: none; white-space: normal; widows: auto; word-spacing: 0px; -web= kit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: brea= k-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" c= lass=3D""> <div style=3D"color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px;= font-style: normal; font-variant-caps: normal; font-weight: normal; letter= -spacing: normal; text-align: start; text-indent: 0px; text-transform: none= ; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Alex Witte<br class=3D""> <br class=3D""> </div> </div> </div> <br class=3D""> <div> <blockquote type=3D"cite" class=3D""> <div class=3D"">On Sep 27, 2017, at 8:32 PM, Alexander Witte <<a href=3D= "mailto:alexander.witte@baicanada.com" class=3D"">alexander.witte@baicanada= .com</a>> wrote:</div> <br class=3D"Apple-interchange-newline"> <div class=3D""><span style=3D"font-family: Helvetica; font-size: 12px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float= : none; display: inline !important;" class=3D"">/</span><span class=3D"" st= yle=3D"font-size: 12px; font-style: normal; font-variant-caps: normal; font= -weight: normal; letter-spacing: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-widt= h: 0px; color: rgb(69, 69, 69); font-family: 'Helvetica Neue';">etc/ovirt-h= osted-engine/hosted-engine.conf<span class=3D"Apple-converted-space"> = </span></span></div> </blockquote> </div> <br class=3D""> </body> </html>
--_000_85068B30E9EA40D5883D6574AC1EC5FBbaicanadacom_--