[ovirt-users] Hosted engine install failed; vdsm upset about broker
Jamie Lawrence
jlawrence at squaretrade.com
Thu Apr 20 17:18:38 UTC 2017
> On Apr 19, 2017, at 11:35 PM, knarra <knarra at redhat.com> wrote:
>
> On 04/20/2017 03:15 AM, Jamie Lawrence wrote:
>> I trialed installing the hosted engine, following the instructions at http://www.ovirt.org/documentation/self-hosted/chap-Deploying_Self-Hosted_Engine/ . This is using Gluster as the backend storage subsystem.
>>
>> Answer file at the end.
>>
>> Per the docs,
>>
>> "When the hosted-engine deployment script completes successfully, the oVirt Engine is configured and running on your host. The Engine has already configured the data center, cluster, host, the Engine virtual machine, and a shared storage domain dedicated to the Engine virtual machine.”
>>
>> In my case, this is false. The installation claims success, but the hosted engine VM stays stopped, unless I start it manually.
> During the install process there is a step where HE vm is stopped and started. Can you check if this has happened correctly ?
The installer claimed it did, but I believe it didn’t. Below the error from my original email, there’s the below (apologies for not including it earlier; I missed it). Note: 04ff4cf1-135a-4918-9a1f-8023322f89a3 is the HE - I’m pretty sure it is complaining about itself. (In any case, I verified that there are no other VMs running with both virsh and vdsClient.)
2017-04-19 12:27:02 DEBUG otopi.context context._executeMethod:128 Stage late_setup METHOD otopi.plugins.gr_he_setup.vm.runvm.Plugin._late_setup
2017-04-19 12:27:02 DEBUG otopi.plugins.gr_he_setup.vm.runvm runvm._late_setup:83 {'status': {'message': 'Done', 'code': 0}, 'items': [u'04ff4cf1-135a-4918-9a1f-8023322f89a3']}
2017-04-19 12:27:02 ERROR otopi.plugins.gr_he_setup.vm.runvm runvm._late_setup:91 The following VMs have been found: 04ff4cf1-135a-4918-9a1f-8023322f89a3
2017-04-19 12:27:02 DEBUG otopi.context context._executeMethod:142 method exception
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod
method['method']()
File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-setup/vm/runvm.py", line 95, in _late_setup
_('Cannot setup Hosted Engine with other VMs running')
RuntimeError: Cannot setup Hosted Engine with other VMs running
2017-04-19 12:27:02 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Environment setup': Cannot setup Hosted Engine with other VMs running
2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN
2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/error=bool:'True'
2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>, RuntimeError('Cannot setup Hosted Engine with other VMs running',), <traceback object at 0x4358098>)]'
2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:774 ENVIRONMENT DUMP - END
>> If I start it manually, the default DC is down, the default cluster has the installation host in the cluster, there is no storage, and the VM doesn’t show up in the GUI. In this install run, I have not yet started the engine manually.
> you wont be seeing HE vm until HE storage is imported into the UI. HE storage will be automatically imported into the UI (which will import HE vm too )once a master domain is present .
Sure; I’m just attempting to provide context.
>> I assume this is related to the errors in ovirt-hosted-engine-setup.log, below. (The timestamps are confusing; it looks like the Python errors are logged some time after they’re captured or something.) The HA broker and agent logs just show them looping in the sequence below.
>>
>> Is there a decent way to pick this up and continue? If not, how do I make this work?
> Can you please check the following things.
>
> 1) is glusterd running on all the nodes ? 'systemctl status glistered’
> 2) Are you able to connect to your storage server which is ovirt_engine in your case.
> 3) Can you check if all the brick process in the volume is up ?
1) Verified that glusterd is running on all three nodes.
2)
[root at sc5-thing-1]# mount -tglusterfs sc5-gluster-1:/ovirt_engine /mnt/ovirt_engine
[root at sc5-thing-1]# df -h
Filesystem Size Used Avail Use% Mounted on
[…]
sc5-gluster-1:/ovirt_engine 300G 2.6G 298G 1% /mnt/ovirt_engine
3)
[root at sc5-gluster-1 jlawrence]# gluster volume status
Status of volume: ovirt_engine
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sc5-gluster-1:/gluster-bricks/ovirt_e
ngine/ovirt_engine-1 49217 0 Y 22102
Brick sc5-gluster-2:/gluster-bricks/ovirt_e
ngine/ovirt_engine-1 49157 0 Y 37842
Brick sc5-gluster-3:/gluster-bricks/ovirt_e
ngine/ovirt_engine-1 49157 0 Y 112018
Self-heal Daemon on localhost N/A N/A Y 113364
Self-heal Daemon on sc5-gluster-2.squaretra
de.com N/A N/A Y 19344
Self-heal Daemon on sc5-gluster-3.squaretra
de.com N/A N/A Y 37956
Task Status of Volume ovirt_engine
------------------------------------------------------------------------------
There are no active volume tasks
Volume ovirt_images is not started
> Thanks
> kasturi.
Thank you!
-j
>>
>> Thanks,
>>
>> -j
>>
>> - - - - ovirt-hosted-engine-setup.log snippet: - - - -
>>
>> 2017-04-19 12:29:55 DEBUG otopi.context context._executeMethod:128 Stage late_setup METHOD otopi.plugins.gr_he_setup.system.vdsmenv.Plugin._late_setup
>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd systemd.status:90 check service vdsmd status
>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/bin/systemctl', 'status', 'vdsmd.service'), executable='None', cwd='None', env=None
>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/bin/systemctl', 'status', 'vdsmd.service'), rc=0
>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/bin/systemctl', 'status', 'vdsmd.service') stdout:
>> ● vdsmd.service - Virtual Desktop Server Manager
>> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
>> Active: active (running) since Wed 2017-04-19 12:26:59 PDT; 2min 55s ago
>> Process: 67370 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS)
>> Process: 69995 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS)
>> Main PID: 70062 (vdsm)
>> CGroup: /system.slice/vdsmd.service
>> └─70062 /usr/bin/python2 /usr/share/vdsm/vdsm
>>
>> Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1)
>> Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed to retrieve Hosted Engine HA info
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
>> stats = instance.get_all_stats()
>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats
>> with broker.connection(self._retries, self._wait):
>> File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
>> return self.gen.next()
>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection
>> self.connect(retries, wait)
>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect
>> raise BrokerConnectionError(error_msg)
>> BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1)
>> Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1)
>> Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed to retrieve Hosted Engine HA info
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
>> stats = instance.get_all_stats()
>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats
>> with broker.connection(self._retries, self._wait):
>> File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
>> return self.gen.next()
>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection
>> self.connect(retries, wait)
>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect
>> raise BrokerConnectionError(error_msg)
>> BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1)
>> Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN File: /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.com.redhat.rhevm.vdsm already removed
>> Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN File: /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.org.qemu.guest_agent.0 already removed
>> Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1)
>> Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed to retrieve Hosted Engine HA info
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
>> stats = instance.get_all_stats()
>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats
>> with broker.connection(self._retries, self._wait):
>> File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
>> return self.gen.next()
>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection
>> self.connect(retries, wait)
>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect
>> raise BrokerConnectionError(error_msg)
>> BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1)
>> Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1)
>> Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed to retrieve Hosted Engine HA info
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
>> stats = instance.get_all_stats()
>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats
>> with broker.connection(self._retries, self._wait):
>> File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
>> return self.gen.next()
>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection
>> self.connect(retries, wait)
>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect
>> raise BrokerConnectionError(error_msg)
>> BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1)
>>
>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/bin/systemctl', 'status', 'vdsmd.service') stderr:
>>
>>
>> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.system.vdsmenv util.connect_vdsm_json_rpc:209 {'status': {'message': 'Done', 'code': 0}, u'systemProductName': u'PowerEdge R630', u'systemSerialNumber': u'GTH5942', u'systemFamily': u'PowerEdge', u'systemUUID': u'4C4C4544-0054-4810-8035-C7C04F393432', u'systemManufacturer': u'Dell Inc.'}
>> 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN
>> 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:770 ENV OVEHOSTED_VDSM/vdscli=_Server:'<vdsm.jsonrpcvdscli._Server object at 0x34a8290>'
>> 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:774 ENVIRONMENT DUMP - END
>> 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage late_setup METHOD otopi.plugins.gr_he_common.vm.misc.Plugin._late_setup
>> 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:134 condition False
>> 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage late_setup METHOD otopi.plugins.gr_he_setup.pki.vdsmpki.Plugin._late_setup
>> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki plugin.executeRaw:813 execute: ('/bin/openssl', 'x509', '-noout', '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), executable='None', cwd='None', env=None
>> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki plugin.executeRaw:863 execute-result: ('/bin/openssl', 'x509', '-noout', '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), rc=0
>> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki plugin.execute:921 execute-output: ('/bin/openssl', 'x509', '-noout', '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem') stdout:
>>
>> - - - - broker.log: - - - -
>>
>> Thread-2970::INFO::2017-04-19 14:29:51,591::cpu_load_no_engine::113::cpu_load_no_engine.EngineHealth::(calculate_load) System load total=0.0117, engine=0.0000, non-engine=0.0117
>> Thread-2976::INFO::2017-04-19 14:29:56,347::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
>> Thread-2976::INFO::2017-04-19 14:29:56,361::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
>> Thread-2977::INFO::2017-04-19 14:29:57,628::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
>> Thread-2977::INFO::2017-04-19 14:29:57,629::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor ping
>> Thread-2977::INFO::2017-04-19 14:29:57,631::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor ping, id 140603614328464
>> Thread-2977::INFO::2017-04-19 14:29:57,662::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor mgmt-bridge
>> Thread-2977::INFO::2017-04-19 14:29:57,667::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor mgmt-bridge, id 140603614326992
>> Thread-2977::INFO::2017-04-19 14:29:57,688::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor mem-free
>> Thread-2977::INFO::2017-04-19 14:29:57,735::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor mem-free, id 140604548742096
>> Thread-2977::INFO::2017-04-19 14:29:57,756::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor cpu-load-no-engine
>> Thread-2977::INFO::2017-04-19 14:29:57,798::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor cpu-load-no-engine, id 43463568
>> Thread-2977::INFO::2017-04-19 14:29:57,841::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor engine-health
>> Thread-2977::INFO::2017-04-19 14:29:57,863::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor engine-health, id 140604881915728
>> Thread-2983::INFO::2017-04-19 14:29:58,153::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
>> Thread-2984::INFO::2017-04-19 14:30:02,179::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
>> Thread-2977::INFO::2017-04-19 14:30:02,249::storage_backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks) Cleaning up stale LV link '/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_agent/hosted-engine.lockspace'
>> Thread-2977::INFO::2017-04-19 14:30:02,321::storage_backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks) Cleaning up stale LV link '/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_agent/hosted-engine.metadata'
>> Thread-2983::INFO::2017-04-19 14:30:02,334::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
>> Thread-2984::INFO::2017-04-19 14:30:02,340::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
>> Thread-2969::INFO::2017-04-19 14:30:02,343::mem_free::50::mem_free.MemFree::(action) memFree: 254533
>> Thread-2980::INFO::2017-04-19 14:30:02,382::mem_free::50::mem_free.MemFree::(action) memFree: 254530
>> Thread-2985::INFO::2017-04-19 14:30:07,991::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
>> Thread-2985::INFO::2017-04-19 14:30:08,082::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
>> Thread-2969::INFO::2017-04-19 14:30:08,129::mem_free::50::mem_free.MemFree::(action) memFree: 254534
>> Thread-2966::INFO::2017-04-19 14:30:09,719::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
>> Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor ping, id 140603546975120
>> Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor ping, id 140603546975120
>> Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor mgmt-bridge, id 140603546973136
>> Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor mgmt-bridge, id 140603546973136
>> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor mem-free, id 140603546973264
>> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor mem-free, id 140603546973264
>> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor cpu-load-no-engine, id 140603546976144
>> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor cpu-load-no-engine, id 140603546976144
>> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor engine-health, id 140603546974096
>> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor engine-health, id 140603546974096
>> Thread-2979::INFO::2017-04-19 14:30:11,145::mgmt_bridge::55::mgmt_bridge.MgmtBridge::(action) Found bridge ovirtmgmt with ports
>> Thread-2986::INFO::2017-04-19 14:30:11,462::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
>> Thread-2986::INFO::2017-04-19 14:30:11,480::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
>> Thread-2987::INFO::2017-04-19 14:30:11,481::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established
>> Thread-2987::INFO::2017-04-19 14:30:11,488::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed
>> Thread-2980::INFO::2017-04-19 14:30:11,502::mem_free::50::mem_free.MemFree::(action) memFree: 254535
>>
>> - - - - ha agent.log: - - - -
>>
>> MainThread::WARNING::2017-04-19 14:31:16,352::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '6'
>> MainThread::INFO::2017-04-19 14:31:16,383::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: sc5-ovirt-2.squaretrade.com
>> MainThread::INFO::2017-04-19 14:31:16,385::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
>> MainThread::INFO::2017-04-19 14:31:18,863::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
>> MainThread::INFO::2017-04-19 14:31:18,864::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
>> MainThread::INFO::2017-04-19 14:31:23,672::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
>> MainThread::INFO::2017-04-19 14:31:23,689::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
>> MainThread::INFO::2017-04-19 14:31:23,956::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
>> MainThread::INFO::2017-04-19 14:31:23,957::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
>> MainThread::INFO::2017-04-19 14:31:26,768::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
>> MainThread::INFO::2017-04-19 14:31:26,769::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
>> MainThread::INFO::2017-04-19 14:31:26,769::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
>> MainThread::WARNING::2017-04-19 14:31:29,548::ovf_store::107::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE
>> MainThread::ERROR::2017-04-19 14:31:29,551::config::450::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs
>> MainThread::INFO::2017-04-19 14:31:29,576::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
>> MainThread::INFO::2017-04-19 14:31:29,577::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '10.181.26.1'}
>> MainThread::INFO::2017-04-19 14:31:29,596::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881712208
>> MainThread::INFO::2017-04-19 14:31:29,596::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
>> MainThread::INFO::2017-04-19 14:31:29,600::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881712016
>> MainThread::INFO::2017-04-19 14:31:29,600::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
>> MainThread::INFO::2017-04-19 14:31:29,602::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881711568
>> MainThread::INFO::2017-04-19 14:31:29,602::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
>> MainThread::INFO::2017-04-19 14:31:29,605::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881711312
>> MainThread::INFO::2017-04-19 14:31:29,605::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
>> MainThread::INFO::2017-04-19 14:31:29,608::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881714896
>> MainThread::INFO::2017-04-19 14:31:34,326::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140604881713168
>> MainThread::INFO::2017-04-19 14:31:34,327::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
>> MainThread::ERROR::2017-04-19 14:31:34,372::agent::196::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Service sanlock is not running and the admin is responsible for starting it. Waiting…
>>
>>
>> - - - - answers.conf - - - -
>>
>> [environment:default]
>> OVEHOSTED_CORE/rollbackProceed=none:None
>> OVEHOSTED_CORE/screenProceed=none:None
>> OVEHOSTED_CORE/deployProceed=bool:True
>> OVEHOSTED_CORE/upgradeProceed=none:None
>> OVEHOSTED_CORE/confirmSettings=bool:True
>> OVEHOSTED_NETWORK/fqdn=str:sc5-ovirt-engine-1.squaretrade.com
>> OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
>> OVEHOSTED_NETWORK/firewallManager=none:None
>> OVEHOSTED_NETWORK/gateway=str:10.181.26.1
>> OVEHOSTED_ENGINE/insecureSSL=none:None
>> OVEHOSTED_ENGINE/clusterName=str:Default
>> OVEHOSTED_STORAGE/storageDatacenterName=str:hosted_datacenter
>> OVEHOSTED_STORAGE/domainType=str:glusterfs
>> OVEHOSTED_STORAGE/connectionUUID=str:c2dd62cb-79aa-4547-b191-b74e55fe5311
>> OVEHOSTED_STORAGE/LunID=none:None
>> OVEHOSTED_STORAGE/imgSizeGB=str:50
>> OVEHOSTED_STORAGE/mntOptions=none:None
>> OVEHOSTED_STORAGE/iSCSIPortalIPAddress=none:None
>> OVEHOSTED_STORAGE/metadataVolumeUUID=str:ac9d7fbf-fb41-4b96-8940-b95698e58b18
>> OVEHOSTED_STORAGE/sdUUID=str:a81b229c-2422-4110-af45-1bc4d9a01bab
>> OVEHOSTED_STORAGE/iSCSITargetName=none:None
>> OVEHOSTED_STORAGE/metadataImageUUID=str:298fef26-3039-4c9e-bee8-39770fb2e821
>> OVEHOSTED_STORAGE/lockspaceVolumeUUID=str:c90a1959-369d-4109-8a6e-5c81f735e43b
>> OVEHOSTED_STORAGE/iSCSIPortalPort=none:None
>> OVEHOSTED_STORAGE/imgUUID=str:284c61f2-a901-46f0-a05a-2e5bf220efc6
>> OVEHOSTED_STORAGE/confImageUUID=str:8bcc7fae-2fbc-40aa-b75a-6c6adb8ba7dc
>> OVEHOSTED_STORAGE/spUUID=str:00000000-0000-0000-0000-000000000000
>> OVEHOSTED_STORAGE/lockspaceImageUUID=str:0ce0f2f9-b9ea-45b3-b091-d4f0e46e150f
>> OVEHOSTED_ENGINE/enableHcGlusterService=bool:False
>> OVEHOSTED_STORAGE/storageDomainName=str:hosted_storage
>> OVEHOSTED_STORAGE/iSCSIPortal=none:None
>> OVEHOSTED_STORAGE/volUUID=str:16304df2-2d95-4d75-932d-21d7c6de000d
>> OVEHOSTED_STORAGE/vgUUID=none:None
>> OVEHOSTED_STORAGE/confVolUUID=str:3b42f578-5d38-4820-b977-fa37b49926f7
>> OVEHOSTED_STORAGE/storageDomainConnection=str:sc5-gluster-1:/ovirt_engine
>> OVEHOSTED_STORAGE/iSCSIPortalUser=none:None
>> OVEHOSTED_VDSM/consoleType=str:qxl
>> OVEHOSTED_VM/vmMemSizeMB=int:16384
>> OVEHOSTED_VM/vmUUID=str:04ff4cf1-135a-4918-9a1f-8023322f89a3
>> OVEHOSTED_VM/vmMACAddr=str:00:16:3e:6e:86:b2
>> OVEHOSTED_VM/emulatedMachine=str:pc
>> OVEHOSTED_VM/consoleUUID=str:7d26f23e-59b6-4378-a0a4-f50e9653ac78
>> OVEHOSTED_VM/vmVCpus=str:4
>> OVEHOSTED_VM/nicUUID=str:a2b7dc56-02f6-4585-bd50-079b17aafe1f
>> OVEHOSTED_VM/cdromUUID=str:cde48946-a7dc-41a5-87ee-854eb4d4dc22
>> OVEHOSTED_VM/ovfArchive=str:/usr/share/ovirt-engine-appliance/ovirt-engine-appliance-4.1-20170406.1.el7.centos.ova
>> OVEHOSTED_VM/vmCDRom=none:None
>> OVEHOSTED_VM/automateVMShutdown=bool:True
>> OVEHOSTED_VM/cloudInitISO=str:generate
>> OVEHOSTED_VM/cloudinitInstanceDomainName=str:squaretrade.com
>> OVEHOSTED_VM/cloudinitInstanceHostName=str:sc5-ovirt-engine-1.squaretrade.com
>> OVEHOSTED_VM/rootSshPubkey=str:ssh-rsa [snip]
>> OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True
>> OVEHOSTED_VM/cloudinitVMStaticCIDR=[snip]
>> OVEHOSTED_VM/cloudinitVMTZ=str:America/Los_Angeles
>> OVEHOSTED_VM/rootSshAccess=str:yes
>> OVEHOSTED_VM/cloudinitVMETCHOSTS=bool:False
>> OVEHOSTED_VM/cloudinitVMDNS=[snip]
>> OVEHOSTED_VDSM/spicePkiSubject=str:O=squaretrade.com, CN=sc5-ovirt-2.squaretrade.com
>> OVEHOSTED_VDSM/pkiSubject=str:/C=EN/L=Test/O=Test/CN=Test
>> OVEHOSTED_VDSM/caSubject=str:/C=EN/L=Test/O=Test/CN=TestCA
>> OVEHOSTED_VDSM/cpu=str:model_Haswell-noTSX
>> OVEHOSTED_NOTIF/smtpPort=str:25
>> OVEHOSTED_NOTIF/smtpServer=[snip]
>> OVEHOSTED_NOTIF/sourceEmail=[snip]
>> OVEHOSTED_NOTIF/destEmail=[snip]
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
More information about the Users
mailing list