<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">OK after a host reboot I was able to get the Engine VM up again and into the Web interface. Although continually when I try a second host to the datacenter within oVirt I run into the error:</div>
<div class=""><br class="">
</div>
<div class="">"Host mennode2 cannot access the Storage Domain(s) hosted_storage attached to the Data Center Train1. Setting Host state to Non Operational."</div>
<div class=""><br class="">
</div>
<div class="">Note: I can successfully read (and mount) the NFS exports oVirt is complaining about:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<i class="">[root@mennode2 ~]# showmount -e menmaster.traindemo.local</i></div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<i class="">Export list for menmaster.traindemo.local:</i></div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<i class="">/shares *</i></div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<i class="">/shares/exports *</i></div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<i class="">/shares/data *</i></div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<i class="">/shares/isos *</i></div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<i class="">[root@mennode2 ~]# </i></div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
[root@mennode2 tmp]# mount -t nfs menmaster.traindemo.local:/shares test</div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
[root@mennode2 tmp]# cd test</div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
[root@mennode2 test]# ls</div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
7d18ff24-57a3-4b4a-9934-0263191fe2e4 data __DIRECT_IO_TEST__ exports isos</div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
[root@mennode2 test]# </div>
</div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<br class="">
</div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
One thing I DO notice is the path of my exports is the difference in the path for the hosted_storage domain. I wonder if the second host would have issues resolving this?</div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<br class="">
</div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
Data. ==> <span style="color: rgb(119, 119, 119); font-family: Arial, sans-serif; font-size: small; background-color: rgb(255, 255, 255);" class="">menmaster.traindemo.local:/shares/data</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
Export ==> <span style="color: rgb(119, 119, 119); font-family: Arial, sans-serif; font-size: small; background-color: rgb(255, 255, 255);" class="">menmaster.traindemo.local:/shares/exports</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
Hosted_storage. ==> <span style="color: rgb(119, 119, 119); font-family: Arial, sans-serif; font-size: small; background-color: rgb(255, 255, 255);" class="">localhost:/shares</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
ISO. ==> <span style="color: rgb(119, 119, 119); font-family: Arial, sans-serif; font-size: small; background-color: rgb(255, 255, 255);" class="">menmaster.traindemo.local:/shares/exports</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Below I have copied exports from the VDSM and OVIRT-ENGINE log and have copied the output of the hosted-engine.conf file. Any help in pinpointing the source of the problem is greatly appreciated!!</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><b class="">VSDM logs:</b></div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:01,700-0400 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.06 seconds (__init__:539)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:03,104-0400 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'59c7f8f3-0063-00a8-02c7-0000000000f3', options=None) from=::ffff:10.0.0.227,39748, flow_id=19e2dbb3, task_id=7a05a5bd-9b15-43be-890d-c6f5d7650e5c
(api:46)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:03,109-0400 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8L}} from=::ffff:10.0.0.227,39748, flow_id=19e2dbb3,
task_id=7a05a5bd-9b15-43be-890d-c6f5d7650e5c (api:52)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:03,110-0400 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:539)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:03,189-0400 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59c7f8f3-0063-00a8-02c7-0000000000f3', options=None) from=::ffff:10.0.0.227,39866, flow_id=19e2dbb3,
task_id=0f8b49e9-9d82-457e-a2a2-39dc7ed9f022 (api:46)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:03,196-0400 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/menmaster.traindemo.local:_shares_isos/da001a29-eca5-44d6-a097-129dd9be623f/images/11111111-1111-1111-1111-111111111111',
'pool_status': 'connected', 'lver': 8L, 'domains': u'da001a29-eca5-44d6-a097-129dd9be623f:Active,f36157cc-b25a-400a-ab0f-a071e8a8eea7:Active,7d18ff24-57a3-4b4a-9934-0263191fe2e4:Active,795d4a1d-3ceb-4773-99de-8e7cf05112f3:Active', 'master_uuid': u'f36157cc-b25a-400a-ab0f-a071e8a8eea7',
'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {u'da001a29-eca5-44d6-a097-129dd9be623f': {'status': u'Active', 'diskfree': '1044166737920', 'isoprefix': u'/rhev/data-center/mnt/menmaster.traindemo.local:_shares_isos/da001a29-eca5-44d6-a097-129dd9be623f/images/11111111-1111-1111-1111-111111111111',
'alerts': [], 'disktotal': '1049702170624', 'version': 0}, u'f36157cc-b25a-400a-ab0f-a071e8a8eea7': {'status': u'Active', 'diskfree': '1044166737920', 'isoprefix': '', 'alerts': [], 'disktotal': '1049702170624', 'version': 4}, u'7d18ff24-57a3-4b4a-9934-0263191fe2e4':
{'status': u'Active', 'diskfree': '1044166737920', 'isoprefix': '', 'alerts': [], 'disktotal': '1049702170624', 'version': 4}, u'795d4a1d-3ceb-4773-99de-8e7cf05112f3': {'status': u'Active', 'diskfree': '1044166737920', 'isoprefix': '', 'alerts': [], 'disktotal':
'1049702170624', 'version': 0}}} from=::ffff:10.0.0.227,39866, flow_id=19e2dbb3, task_id=0f8b49e9-9d82-457e-a2a2-39dc7ed9f022 (api:52)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:03,198-0400 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:539)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:04,212-0400 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:35980 (protocoldetector:72)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:04,223-0400 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:35980 (protocoldetector:127)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:04,224-0400 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:103)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:04,225-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:132)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:04,622-0400 INFO (monitor/7d18ff2) [storage.SANLock] Acquiring host id for domain 7d18ff24-57a3-4b4a-9934-0263191fe2e4 (id=1, async=True) (clusterlock:288)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:04,623-0400 ERROR (monitor/7d18ff2) [storage.Monitor] Error acquiring host id 1 for domain 7d18ff24-57a3-4b4a-9934-0263191fe2e4 (monitor:570)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Traceback (most recent call last):</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> File "/usr/share/vdsm/storage/monitor.py", line 567, in _acquireHostId</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> self.domain.acquireHostId(self.hostId, async=True)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> File "/usr/share/vdsm/storage/sd.py", line 785, in acquireHostId</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> self._manifest.acquireHostId(hostId, async)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> File "/usr/share/vdsm/storage/sd.py", line 449, in acquireHostId</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> self._domainLock.acquireHostId(hostId, async)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", line 319, in acquireHostId</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> raise se.AcquireHostIdFailure(self._sdUUID, e)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">AcquireHostIdFailure: Cannot acquire host id: (u'7d18ff24-57a3-4b4a-9934-0263191fe2e4', SanlockException(22, 'Sanlock lockspace add failure', 'Invalid argument'))</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:04,647-0400 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:04,654-0400 INFO (jsonrpc/3) [vdsm.api] START repoStats(options=None) from=::1,35980, task_id=9732972d-9a76-4349-9c65-53e57b2472d6 (api:46)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:04,655-0400 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'da001a29-eca5-44d6-a097-129dd9be623f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay':
'0.000357054', 'lastCheck': '4.1', 'valid': True}, u'f36157cc-b25a-400a-ab0f-a071e8a8eea7': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000531828', 'lastCheck': '4.0', 'valid': True}, u'7d18ff24-57a3-4b4a-9934-0263191fe2e4': {'code':
0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000205228', 'lastCheck': '0.0', 'valid': True}, u'795d4a1d-3ceb-4773-99de-8e7cf05112f3': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000352165', 'lastCheck': '4.1',
'valid': True}} from=::1,35980, task_id=9732972d-9a76-4349-9c65-53e57b2472d6 (api:52)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:04,656-0400 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStorageRepoStats succeeded in 0.00 seconds (__init__:539)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:07,717-0400 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:35982 (protocoldetector:72)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:07,728-0400 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:35982 (protocoldetector:127)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:07,729-0400 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:103)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:07,730-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:132)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,152-0400 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,159-0400 INFO (jsonrpc/1) [vdsm.api] START repoStats(options=None) from=::1,35982, task_id=80eb282b-f297-4f98-a368-7d5627ac50f3 (api:46)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,160-0400 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'da001a29-eca5-44d6-a097-129dd9be623f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay':
'0.000357054', 'lastCheck': '7.6', 'valid': True}, u'f36157cc-b25a-400a-ab0f-a071e8a8eea7': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000531828', 'lastCheck': '7.5', 'valid': True}, u'7d18ff24-57a3-4b4a-9934-0263191fe2e4': {'code':
0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000205228', 'lastCheck': '3.5', 'valid': True}, u'795d4a1d-3ceb-4773-99de-8e7cf05112f3': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000352165', 'lastCheck': '7.6',
'valid': True}} from=::1,35982, task_id=80eb282b-f297-4f98-a368-7d5627ac50f3 (api:52)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,171-0400 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:35984 (protocoldetector:72)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,198-0400 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:35984 (protocoldetector:127)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,199-0400 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:103)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,201-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:132)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,374-0400 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.22 seconds (__init__:539)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,728-0400 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:35987 (protocoldetector:72)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,732-0400 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,738-0400 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats succeeded in 0.00 seconds (__init__:539)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,756-0400 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:35987 (protocoldetector:127)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,757-0400 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:103)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:08,771-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:132)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:09,286-0400 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:09,395-0400 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities succeeded in 0.10 seconds (__init__:539)</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:10:10,590-0400 INFO (itmap/0) [IOProcessClient] Starting client ioprocess-37 (__init__:330)</span></div>
</div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<br class="">
</div>
</div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<br class="">
</div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<b class="">ENGINE LOGS:</b></div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<br class="">
</div>
<div style="margin: 0px; line-height: normal; font-family: 'Helvetica Neue'; color: rgb(69, 69, 69);" class="">
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:07:58,154-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20170927230758-10.0.0.222-3f898655.log'</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:07:58,316-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Stage: Termination</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:07:58,458-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-1) [3f898655] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation
ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host mennode2 was completed successfully with message 'no updates found.'.</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:07:58,874-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Stage: Misc configuration</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:07:58,888-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Stage: Transaction commit</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:07:59,029-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Stage: Closing up</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:07:59,039-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Stage: Pre-termination</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:07:59,092-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20170927230759-menmaster.traindemo.local-3f898655.log'</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:07:59,262-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [3f898655] Stage: Termination</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:07:59,375-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-2) [3f898655] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation
ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host menmaster was completed successfully with message 'no updates found.'.</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:38,705-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-13) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[0ef7ec10-af19-4afa-980b-77a772c4e014=VDS]',
sharedLocks=''}'</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:38,776-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Running command: ActivateVdsCommand
internal: false. Entities affected : ID: 0ef7ec10-af19-4afa-980b-77a772c4e014 Type: VDSAction group MANIPULATE_HOST with role type ADMIN</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:38,777-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Before acquiring lock in order to prevent
monitoring for host 'mennode2' from data-center 'Train1'</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:38,778-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Failed to acquire lock and wait
lock 'HostEngineLock:{exclusiveLocks='[0ef7ec10-af19-4afa-980b-77a772c4e014=VDS_INIT]', sharedLocks=''}'</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:38,995-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Lock acquired, from now a monitoring
of host will be skipped for host 'mennode2' from data-center 'Train1'</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:39,003-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] START, SetVdsStatusVDSCommand(HostName
= mennode2, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='0ef7ec10-af19-4afa-980b-77a772c4e014', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 78cc1caf</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:39,027-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] FINISH, SetVdsStatusVDSCommand,
log id: 78cc1caf</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:39,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Activate host finished. Lock released.
Monitoring can run now for host 'mennode2' from data-center 'Train1'</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:39,050-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] EVENT_ID: VDS_ACTIVATE(16),
Correlation ID: 75de01a6-efd2-418f-a6b1-b4ef1f38923f, Job ID: 2e40097e-e991-45cc-bd49-3176822ebb5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Activation of host mennode2 initiated by admin@internal-authz.</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:39,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-2) [75de01a6-efd2-418f-a6b1-b4ef1f38923f] Lock freed to object 'EngineLock:{exclusiveLocks='[0ef7ec10-af19-4afa-980b-77a772c4e014=VDS]',
sharedLocks=''}'</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:43,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5437c1b7] START, GetHardwareInfoVDSCommand(HostName = mennode2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='0ef7ec10-af19-4afa-980b-77a772c4e014', vds='Host[mennode2,0ef7ec10-af19-4afa-980b-77a772c4e014]'}), log id: 2bcbe26e</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:44,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5437c1b7] FINISH, GetHardwareInfoVDSCommand, log id: 2bcbe26e</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:44,186-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [12ef27d8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand
internal: true. Entities affected : ID: 0ef7ec10-af19-4afa-980b-77a772c4e014 Type: VDS</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:44,214-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [63cdc728] Running command: InitVdsOnUpCommand internal: true. Entities affected :
ID: 59c7f8f3-0063-00a8-02c7-0000000000f3 Type: StoragePool</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:44,234-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7670f89e] Running command: ConnectHostToStoragePoolServersCommand
internal: true. Entities affected : ID: 59c7f8f3-0063-00a8-02c7-0000000000f3 Type: StoragePool</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:44,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7670f89e] START, ConnectStorageServerVDSCommand(HostName
= mennode2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='0ef7ec10-af19-4afa-980b-77a772c4e014', storagePoolId='59c7f8f3-0063-00a8-02c7-0000000000f3', storageType='NFS', connectionList='[StorageServerConnections:{id='aac63433-88fb-4daf-8c83-871a85b3a49d',
connection='menmaster.traindemo.local:/shares/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='AUTO', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='5adb7a4b-ebbd-43a4-b5cb-8c8eb3d6f1ee',
connection='localhost:/shares', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='9eb3b1e2-4063-46cd-ba4e-440c4048e92a', connection='menmaster.traindemo.local:/shares/isos',
iqn='null', vfsType='null', mountOptions='null', nfsVersion='AUTO', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='1c9215a1-4154-4a78-b8ff-565c3a237c58', connection='menmaster.traindemo.local:/shares/exports',
iqn='null', vfsType='null', mountOptions='null', nfsVersion='AUTO', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5625b02f</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:48,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7670f89e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation
ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host mennode2 to the Storage Domains hosted_storage.</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:48,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7670f89e] FINISH, ConnectStorageServerVDSCommand, return:
{aac63433-88fb-4daf-8c83-871a85b3a49d=0, 5adb7a4b-ebbd-43a4-b5cb-8c8eb3d6f1ee=477, 9eb3b1e2-4063-46cd-ba4e-440c4048e92a=0, 1c9215a1-4154-4a78-b8ff-565c3a237c58=0}, log id: 5625b02f</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:48,386-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7670f89e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation
ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection localhost:/shares returned by VDSM was: Problem while trying to mount target</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:48,389-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7670f89e] The connection with details 'localhost:/shares' failed
because of error code '477' and error message is: problem while trying to mount target</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:48,401-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7670f89e] Host 'mennode2' storage connection was failed </span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:48,411-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7670f89e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation
ID: 7670f89e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host mennode2 to Storage Servers</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:48,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName =
mennode2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='0ef7ec10-af19-4afa-980b-77a772c4e014', vdsId='0ef7ec10-af19-4afa-980b-77a772c4e014', storagePoolId='59c7f8f3-0063-00a8-02c7-0000000000f3', masterVersion='1'}), log id: 3d5c5d79</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:48,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {7d18ff24-57a3-4b4a-9934-0263191fe2e4=active,
da001a29-eca5-44d6-a097-129dd9be623f=active, 795d4a1d-3ceb-4773-99de-8e7cf05112f3=active, f36157cc-b25a-400a-ab0f-a071e8a8eea7=active}</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:49,656-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id:
3d5c5d79</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:50,681-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-8) [] Domain '7d18ff24-57a3-4b4a-9934-0263191fe2e4:hosted_storage' was reported
with error code '358'</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:50,682-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Storage Domain 'hosted_storage' of pool 'Train1' is in problem in host 'mennode2'</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:50,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(189), Correlation
ID: 63cdc728, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host mennode2 reports about one of the Active Storage Domains as Problematic.</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:50,726-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [748ef73f] Running command: SetNonOperationalVdsCommand internal: true. Entities
affected : ID: 0ef7ec10-af19-4afa-980b-77a772c4e014 Type: VDS</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:50,729-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [748ef73f] START, SetVdsStatusVDSCommand(HostName = mennode2, SetVdsStatusVDSCommandParameters:{runAsync='true',
hostId='0ef7ec10-af19-4afa-980b-77a772c4e014', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2589acd2</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:50,736-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [748ef73f] FINISH, SetVdsStatusVDSCommand, log id: 2589acd2</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:50,807-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [748ef73f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation
ID: 748ef73f, Job ID: e80941e4-3dbf-4159-8421-96d2d90160ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host mennode2 cannot access the Storage Domain(s) hosted_storage attached to the Data Center Train1. Setting Host state to Non-Operational.</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:50,820-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [748ef73f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000),
Correlation ID: 63cdc728, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host mennode2.</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:50,826-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [748ef73f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation
ID: 63cdc728, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host mennode2 to Storage Pool Train1</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:50,856-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [42cbaee1] Running command: HandleVdsVersionCommand internal: true. Entities affected
: ID: 0ef7ec10-af19-4afa-980b-77a772c4e014 Type: VDS</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">2017-09-27 23:08:50,863-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [42cbaee1] Host 'mennode2'(0ef7ec10-af19-4afa-980b-77a772c4e014) is already
in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.</span></div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class=""><b class=""><span class="" style="float: none; display: inline !important;">/</span><span class="" style="color: rgb(69, 69, 69); font-family: 'Helvetica Neue';">etc/ovirt-hosted-engine/hosted-engine.conf </span></b></blockquote>
</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">fqdn=engine.traindemo.local</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">vm_disk_id=0077cb58-7abe-4bdf-96b1-55cc85030d76</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">vm_disk_vol_id=d5bd2690-7cfb-4aa2-b898-41579114ec08</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">vmid=564d290f-ae66-4352-9770-f2e062000358</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">storage=menmaster.traindemo.local:/shares</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">mnt_options=</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">conf=/var/run/ovirt-hosted-engine-ha/vm.conf</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">host_id=1</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">console=vnc</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">domainType=nfs3</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">spUUID=00000000-0000-0000-0000-000000000000</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">sdUUID=7d18ff24-57a3-4b4a-9934-0263191fe2e4</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">connectionUUID=addfd5fc-8668-4d2a-9b0b-f801b21a98d8</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">ca_subject="C=EN, L=Test, O=Test, CN=Test"</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">vdsm_use_ssl=true</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">gateway=10.0.0.1</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">bridge=ovirtmgmt</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">metadata_volume_UUID=156c7549-8e2e-4580-9825-81c3b13387ce</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">metadata_image_UUID=c5db06ac-ea37-47bc-9065-b94eb230616a</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">lockspace_volume_UUID=5f4f4a63-5608-4daf-a034-2b8e83b5de42</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">lockspace_image_UUID=0d7f8661-3abd-44e8-87a7-3b85f2e29fda</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">conf_volume_UUID=ddffd499-294c-456f-bb0e-08e366380a22</span></div>
<div style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: rgb(40, 254, 20); background-color: rgba(0, 0, 0, 0.901961);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">conf_image_UUID=5f264ea1-4654-4e40-87aa-80fcaacc60d8</span></div>
</div>
<div class="">Thanks,</div>
<div class=""><br class="">
</div>
<br class="">
<div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
Alex Witte<br class="">
<br class="">
</div>
</div>
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Sep 27, 2017, at 8:32 PM, Alexander Witte <<a href="mailto:alexander.witte@baicanada.com" class="">alexander.witte@baicanada.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">/</span><span class="" style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: rgb(69, 69, 69); font-family: 'Helvetica Neue';">etc/ovirt-hosted-engine/hosted-engine.conf<span class="Apple-converted-space"> </span></span></div>
</blockquote>
</div>
<br class="">
</body>
</html>