[ovirt-users] Hosted engine install failed; vdsm upset about broker (revised)
Jamie Lawrence
jlawrence at squaretrade.com
Thu Apr 20 00:14:10 UTC 2017
So, tracing this further, I’m pretty sure this is something about sanlock.
As best I can tell this[1] seems to be the failure that is blocking importing the pool, creating storage domains, importing the HE, etc. Contrary to the log, sanlock is running; I verified it starts on system-boot and restarts just fine.
I found one reference to someone having a similar problem in 3.6, but that appeared to have been a permission issue I’m not afflicted with.
How can I move past this?
TIA,
-j
[1] agent.log:
MainThread::WARNING::2017-04-19 17:07:13,537::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '6'
MainThread::INFO::2017-04-19 17:07:13,567::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: sc5-ovirt-2.squaretrade.com
MainThread::INFO::2017-04-19 17:07:13,569::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-04-19 17:07:16,044::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-04-19 17:07:16,045::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-04-19 17:07:20,876::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-04-19 17:07:20,893::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-04-19 17:07:21,160::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-04-19 17:07:21,160::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-04-19 17:07:23,954::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-04-19 17:07:23,955::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-04-19 17:07:23,955::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::WARNING::2017-04-19 17:07:26,741::ovf_store::107::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE
MainThread::ERROR::2017-04-19 17:07:26,744::config::450::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs
MainThread::INFO::2017-04-19 17:07:26,770::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-04-19 17:07:26,771::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '10.181.26.1'}
MainThread::INFO::2017-04-19 17:07:26,774::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140621269798096
MainThread::INFO::2017-04-19 17:07:26,774::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-04-19 17:07:26,791::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140621269798544
MainThread::INFO::2017-04-19 17:07:26,792::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-04-19 17:07:26,793::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140621269798224
MainThread::INFO::2017-04-19 17:07:26,794::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
MainThread::INFO::2017-04-19 17:07:26,796::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140621269796816
MainThread::INFO::2017-04-19 17:07:26,796::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
MainThread::INFO::2017-04-19 17:07:26,798::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140621269797840
MainThread::INFO::2017-04-19 17:07:31,513::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140621269798480
MainThread::INFO::2017-04-19 17:07:31,513::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::ERROR::2017-04-19 17:07:31,556::agent::196::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Service sanlock is not running and the admin is responsible for starting it. Waiting...
MainThread::WARNING::2017-04-19 17:07:36,562::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '7'
MainThread::INFO::2017-04-19 17:07:36,592::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: sc5-ovirt-2.squaretrade.com
- - - prior message below - - -
I trialed installing the hosted engine, following the instructions at http://www.ovirt.org/documentation/self-hosted/chap-Deploying_Self-Hosted_Engine/ . This is using Gluster as the backend storage subsystem.
Answer file at the end.
Per the docs,
"When the hosted-engine deployment script completes successfully, the oVirt Engine is configured and running on your host. The Engine has already configured the data center, cluster, host, the Engine virtual machine, and a shared storage domain dedicated to the Engine virtual machine.”
In my case, this is false. The installation claims success, but the hosted engine VM stays stopped, unless I start it manually. If I start it manually, the default DC is down, the default cluster has the installation host in the cluster, there is no storage, and the VM doesn’t show up in the GUI. In this install run, I have not yet started the engine manually.
I assume this is related to the errors in ovirt-hosted-engine-setup.log, below. (The timestamps are confusing; it looks like the Python errors are logged some time after they’re captured or something.) The HA broker and agent logs just show them looping in the sequence below.
Is there a decent way to pick this up and continue? If not, how do I make this work?
Thanks,
-j
- - - - ovirt-hosted-engine-setup.log snippet: - - - -
2017-04-19 12:29:55 DEBUG otopi.context context._executeMethod:128 Stage late_setup METHOD otopi.plugins.gr_he_setup.system.vdsmenv.Plugin._late_setup
2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd systemd.status:90 check service vdsmd status
2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/bin/systemctl', 'status', 'vdsmd.service'), executable='None', cwd='None', env=None
2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/bin/systemctl', 'status', 'vdsmd.service'), rc=0
2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/bin/systemctl', 'status', 'vdsmd.service') stdout:
● vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2017-04-19 12:26:59 PDT; 2min 55s ago
Process: 67370 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS)
Process: 69995 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS)
Main PID: 70062 (vdsm)
CGroup: /system.slice/vdsmd.service
└─70062 /usr/bin/python2 /usr/share/vdsm/vdsm
Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1)
Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed to retrieve Hosted Engine HA info
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
stats = instance.get_all_stats()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats
with broker.connection(self._retries, self._wait):
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection
self.connect(retries, wait)
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect
raise BrokerConnectionError(error_msg)
BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1)
Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1)
Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed to retrieve Hosted Engine HA info
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
stats = instance.get_all_stats()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats
with broker.connection(self._retries, self._wait):
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection
self.connect(retries, wait)
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect
raise BrokerConnectionError(error_msg)
BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1)
Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN File: /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.com.redhat.rhevm.vdsm already removed
Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN File: /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.org.qemu.guest_agent.0 already removed
Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1)
Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed to retrieve Hosted Engine HA info
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
stats = instance.get_all_stats()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats
with broker.connection(self._retries, self._wait):
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection
self.connect(retries, wait)
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect
raise BrokerConnectionError(error_msg)
BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1)
Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1)
Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed to retrieve Hosted Engine HA info
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
stats = instance.get_all_stats()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats
with broker.connection(self._retries, self._wait):
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection
self.connect(retries, wait)
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect
raise BrokerConnectionError(error_msg)
BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1)
2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/bin/systemctl', 'status', 'vdsmd.service') stderr:
2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.system.vdsmenv util.connect_vdsm_json_rpc:209 {'status': {'message': 'Done', 'code': 0}, u'systemProductName': u'PowerEdge R630', u'systemSerialNumber': u'GTH5942', u'systemFamily': u'PowerEdge', u'systemUUID': u'4C4C4544-0054-4810-8035-C7C04F393432', u'systemManufacturer': u'Dell Inc.'}
2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN
2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:770 ENV OVEHOSTED_VDSM/vdscli=_Server:'<vdsm.jsonrpcvdscli._Server object at 0x34a8290>'
2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:774 ENVIRONMENT DUMP - END
2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage late_setup METHOD otopi.plugins.gr_he_common.vm.misc.Plugin._late_setup
2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:134 condition False
2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage late_setup METHOD otopi.plugins.gr_he_setup.pki.vdsmpki.Plugin._late_setup
2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki plugin.executeRaw:813 execute: ('/bin/openssl', 'x509', '-noout', '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), executable='None', cwd='None', env=None
2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki plugin.executeRaw:863 execute-result: ('/bin/openssl', 'x509', '-noout', '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), rc=0
2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki plugin.execute:921 execute-output: ('/bin/openssl', 'x509', '-noout', '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem') stdout:
- - - - broker.log: - - - -
Thread-2970::INFO::2017-04-19 14:29:51,591::cpu_load_no_engine::113::cpu_load_no_engine.EngineHealth::(calculate_load) System load total=0.0117, engine=0.0000, non-engine=0.0117
Thread-2976::INFO::2017-04-19 14:29:56,347::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
Thread-2976::INFO::2017-04-19 14:29:56,361::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
Thread-2977::INFO::2017-04-19 14:29:57,628::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
Thread-2977::INFO::2017-04-19 14:29:57,629::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor ping
Thread-2977::INFO::2017-04-19 14:29:57,631::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor ping, id 140603614328464
Thread-2977::INFO::2017-04-19 14:29:57,662::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor mgmt-bridge
Thread-2977::INFO::2017-04-19 14:29:57,667::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor mgmt-bridge, id 140603614326992
Thread-2977::INFO::2017-04-19 14:29:57,688::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor mem-free
Thread-2977::INFO::2017-04-19 14:29:57,735::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor mem-free, id 140604548742096
Thread-2977::INFO::2017-04-19 14:29:57,756::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor cpu-load-no-engine
Thread-2977::INFO::2017-04-19 14:29:57,798::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor cpu-load-no-engine, id 43463568
Thread-2977::INFO::2017-04-19 14:29:57,841::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor engine-health
Thread-2977::INFO::2017-04-19 14:29:57,863::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor engine-health, id 140604881915728
Thread-2983::INFO::2017-04-19 14:29:58,153::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
Thread-2984::INFO::2017-04-19 14:30:02,179::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
Thread-2977::INFO::2017-04-19 14:30:02,249::storage_backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks) Cleaning up stale LV link '/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_agent/hosted-engine.lockspace'
Thread-2977::INFO::2017-04-19 14:30:02,321::storage_backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks) Cleaning up stale LV link '/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_agent/hosted-engine.metadata'
Thread-2983::INFO::2017-04-19 14:30:02,334::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
Thread-2984::INFO::2017-04-19 14:30:02,340::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
Thread-2969::INFO::2017-04-19 14:30:02,343::mem_free::50::mem_free.MemFree::(action) memFree: 254533
Thread-2980::INFO::2017-04-19 14:30:02,382::mem_free::50::mem_free.MemFree::(action) memFree: 254530
Thread-2985::INFO::2017-04-19 14:30:07,991::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
Thread-2985::INFO::2017-04-19 14:30:08,082::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
Thread-2969::INFO::2017-04-19 14:30:08,129::mem_free::50::mem_free.MemFree::(action) memFree: 254534
Thread-2966::INFO::2017-04-19 14:30:09,719::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor ping, id 140603546975120
Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor ping, id 140603546975120
Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor mgmt-bridge, id 140603546973136
Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor mgmt-bridge, id 140603546973136
Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor mem-free, id 140603546973264
Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor mem-free, id 140603546973264
Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor cpu-load-no-engine, id 140603546976144
Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor cpu-load-no-engine, id 140603546976144
Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor engine-health, id 140603546974096
Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor engine-health, id 140603546974096
Thread-2979::INFO::2017-04-19 14:30:11,145::mgmt_bridge::55::mgmt_bridge.MgmtBridge::(action) Found bridge ovirtmgmt with ports
Thread-2986::INFO::2017-04-19 14:30:11,462::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
Thread-2986::INFO::2017-04-19 14:30:11,480::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
Thread-2987::INFO::2017-04-19 14:30:11,481::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
Thread-2987::INFO::2017-04-19 14:30:11,488::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
Thread-2980::INFO::2017-04-19 14:30:11,502::mem_free::50::mem_free.MemFree::(action) memFree: 254535
- - - - ha agent.log: - - - -
MainThread::WARNING::2017-04-19 14:31:16,352::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '6'
MainThread::INFO::2017-04-19 14:31:16,383::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: sc5-ovirt-2.squaretrade.com
MainThread::INFO::2017-04-19 14:31:16,385::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-04-19 14:31:18,863::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-04-19 14:31:18,864::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-04-19 14:31:23,672::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-04-19 14:31:23,689::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-04-19 14:31:23,956::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-04-19 14:31:23,957::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-04-19 14:31:26,768::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-04-19 14:31:26,769::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-04-19 14:31:26,769::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::WARNING::2017-04-19 14:31:29,548::ovf_store::107::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE
MainThread::ERROR::2017-04-19 14:31:29,551::config::450::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs
MainThread::INFO::2017-04-19 14:31:29,576::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-04-19 14:31:29,577::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '10.181.26.1'}
MainThread::INFO::2017-04-19 14:31:29,596::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881712208
MainThread::INFO::2017-04-19 14:31:29,596::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-04-19 14:31:29,600::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881712016
MainThread::INFO::2017-04-19 14:31:29,600::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-04-19 14:31:29,602::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881711568
MainThread::INFO::2017-04-19 14:31:29,602::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
MainThread::INFO::2017-04-19 14:31:29,605::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881711312
MainThread::INFO::2017-04-19 14:31:29,605::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
MainThread::INFO::2017-04-19 14:31:29,608::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881714896
MainThread::INFO::2017-04-19 14:31:34,326::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140604881713168
MainThread::INFO::2017-04-19 14:31:34,327::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::ERROR::2017-04-19 14:31:34,372::agent::196::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Service sanlock is not running and the admin is responsible for starting it. Waiting…
- - - - answers.conf - - - -
[environment:default]
OVEHOSTED_CORE/rollbackProceed=none:None
OVEHOSTED_CORE/screenProceed=none:None
OVEHOSTED_CORE/deployProceed=bool:True
OVEHOSTED_CORE/upgradeProceed=none:None
OVEHOSTED_CORE/confirmSettings=bool:True
OVEHOSTED_NETWORK/fqdn=str:sc5-ovirt-engine-1.squaretrade.com
OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
OVEHOSTED_NETWORK/firewallManager=none:None
OVEHOSTED_NETWORK/gateway=str:10.181.26.1
OVEHOSTED_ENGINE/insecureSSL=none:None
OVEHOSTED_ENGINE/clusterName=str:Default
OVEHOSTED_STORAGE/storageDatacenterName=str:hosted_datacenter
OVEHOSTED_STORAGE/domainType=str:glusterfs
OVEHOSTED_STORAGE/connectionUUID=str:c2dd62cb-79aa-4547-b191-b74e55fe5311
OVEHOSTED_STORAGE/LunID=none:None
OVEHOSTED_STORAGE/imgSizeGB=str:50
OVEHOSTED_STORAGE/mntOptions=none:None
OVEHOSTED_STORAGE/iSCSIPortalIPAddress=none:None
OVEHOSTED_STORAGE/metadataVolumeUUID=str:ac9d7fbf-fb41-4b96-8940-b95698e58b18
OVEHOSTED_STORAGE/sdUUID=str:a81b229c-2422-4110-af45-1bc4d9a01bab
OVEHOSTED_STORAGE/iSCSITargetName=none:None
OVEHOSTED_STORAGE/metadataImageUUID=str:298fef26-3039-4c9e-bee8-39770fb2e821
OVEHOSTED_STORAGE/lockspaceVolumeUUID=str:c90a1959-369d-4109-8a6e-5c81f735e43b
OVEHOSTED_STORAGE/iSCSIPortalPort=none:None
OVEHOSTED_STORAGE/imgUUID=str:284c61f2-a901-46f0-a05a-2e5bf220efc6
OVEHOSTED_STORAGE/confImageUUID=str:8bcc7fae-2fbc-40aa-b75a-6c6adb8ba7dc
OVEHOSTED_STORAGE/spUUID=str:00000000-0000-0000-0000-000000000000
OVEHOSTED_STORAGE/lockspaceImageUUID=str:0ce0f2f9-b9ea-45b3-b091-d4f0e46e150f
OVEHOSTED_ENGINE/enableHcGlusterService=bool:False
OVEHOSTED_STORAGE/storageDomainName=str:hosted_storage
OVEHOSTED_STORAGE/iSCSIPortal=none:None
OVEHOSTED_STORAGE/volUUID=str:16304df2-2d95-4d75-932d-21d7c6de000d
OVEHOSTED_STORAGE/vgUUID=none:None
OVEHOSTED_STORAGE/confVolUUID=str:3b42f578-5d38-4820-b977-fa37b49926f7
OVEHOSTED_STORAGE/storageDomainConnection=str:sc5-gluster-1:/ovirt_engine
OVEHOSTED_STORAGE/iSCSIPortalUser=none:None
OVEHOSTED_VDSM/consoleType=str:qxl
OVEHOSTED_VM/vmMemSizeMB=int:16384
OVEHOSTED_VM/vmUUID=str:04ff4cf1-135a-4918-9a1f-8023322f89a3
OVEHOSTED_VM/vmMACAddr=str:00:16:3e:6e:86:b2
OVEHOSTED_VM/emulatedMachine=str:pc
OVEHOSTED_VM/consoleUUID=str:7d26f23e-59b6-4378-a0a4-f50e9653ac78
OVEHOSTED_VM/vmVCpus=str:4
OVEHOSTED_VM/nicUUID=str:a2b7dc56-02f6-4585-bd50-079b17aafe1f
OVEHOSTED_VM/cdromUUID=str:cde48946-a7dc-41a5-87ee-854eb4d4dc22
OVEHOSTED_VM/ovfArchive=str:/usr/share/ovirt-engine-appliance/ovirt-engine-appliance-4.1-20170406.1.el7.centos.ova
OVEHOSTED_VM/vmCDRom=none:None
OVEHOSTED_VM/automateVMShutdown=bool:True
OVEHOSTED_VM/cloudInitISO=str:generate
OVEHOSTED_VM/cloudinitInstanceDomainName=str:squaretrade.com
OVEHOSTED_VM/cloudinitInstanceHostName=str:sc5-ovirt-engine-1.squaretrade.com
OVEHOSTED_VM/rootSshPubkey=str:ssh-rsa [snip]
OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True
OVEHOSTED_VM/cloudinitVMStaticCIDR=[snip]
OVEHOSTED_VM/cloudinitVMTZ=str:America/Los_Angeles
OVEHOSTED_VM/rootSshAccess=str:yes
OVEHOSTED_VM/cloudinitVMETCHOSTS=bool:False
OVEHOSTED_VM/cloudinitVMDNS=[snip]
OVEHOSTED_VDSM/spicePkiSubject=str:O=squaretrade.com, CN=sc5-ovirt-2.squaretrade.com
OVEHOSTED_VDSM/pkiSubject=str:/C=EN/L=Test/O=Test/CN=Test
OVEHOSTED_VDSM/caSubject=str:/C=EN/L=Test/O=Test/CN=TestCA
OVEHOSTED_VDSM/cpu=str:model_Haswell-noTSX
OVEHOSTED_NOTIF/smtpPort=str:25
OVEHOSTED_NOTIF/smtpServer=[snip]
OVEHOSTED_NOTIF/sourceEmail=[snip]
OVEHOSTED_NOTIF/destEmail=[snip]
More information about the Users
mailing list