Re: [ovirt-users] Hosted engine install failed; vdsm upset about broker

On Apr 21, 2017, at 6:38 AM, knarra <knarra(a)redhat.com&gt; wrote: On 04/21/2017 06:34 PM, Jamie Lawrence wrote: >> On Apr 20, 2017, at 10:36 PM, knarra <knarra(a)redhat.com&gt; wrote: >>> The installer claimed it did, but I believe it didn’t. Below the error from my original email, there’s the below (apologies for not including it earlier; I missed it). Note: 04ff4cf1-135a-4918-9a1f-8023322f89a3 is the HE - I’m pretty sure it is complaining about itself. (In any case, I verified that there are no other VMs running with both virsh and vdsClient.) > ^^^ > >>> 2017-04-19 12:27:02 DEBUG otopi.context context._executeMethod:128 Stage late_setup METHOD otopi.plugins.gr_he_setup.vm.runvm.Plugin._late_setup >>> 2017-04-19 12:27:02 DEBUG otopi.plugins.gr_he_setup.vm.runvm runvm._late_setup:83 {'status': {'message': 'Done', 'code': 0}, 'items': [u'04ff4cf1-135a-4918-9a1f-8023322f89a3']} >>> 2017-04-19 12:27:02 ERROR otopi.plugins.gr_he_setup.vm.runvm runvm._late_setup:91 The following VMs have been found: 04ff4cf1-135a-4918-9a1f-8023322f89a3 >>> 2017-04-19 12:27:02 DEBUG otopi.context context._executeMethod:142 method exception >>> Traceback (most recent call last): >>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod >>> method['method']() >>> File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-setup/vm/runvm.py", line 95, in _late_setup >>> _('Cannot setup Hosted Engine with other VMs running') >>> RuntimeError: Cannot setup Hosted Engine with other VMs running >>> 2017-04-19 12:27:02 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Environment setup': Cannot setup Hosted Engine with other VMs running >>> 2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN >>> 2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/error=bool:'True' >>> 2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>, RuntimeError('Cannot setup Hosted Engine with other VMs running',), <traceback object at 0x4358098>)]' >>> 2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:774 ENVIRONMENT DUMP - END >> James, generally this issue happens when the setup failed once and you tried re running it again. Can you clean it and deploy it again? HE should come up successfully. Below are the steps for cleaning it up. > Knarra, > > I realize that. However, that is not the situation in my case. See above, at the mark - the UUID it is complaining about is the UUID of the hosted-engine it just installed. From the answers file generated from the run (whole thing below): > >>>>> OVEHOSTED_VM/vmUUID=str:04ff4cf1-135a-4918-9a1f-8023322f89a3 > Also see the WARNs I mentioned previously, quoted below. Excerpt: > >>>>> Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN File: /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.com.redhat.rhevm.vdsm already removed >>>>> Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN File: /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.org.qemu.guest_agent.0 already removed >>>>> Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1) > I’m not clear on what it is attempting to do there, but it seems relevant. I remember that you said HE vm was not started when the installation was successful. Is Local Maintenance enabled on that host? can you please check if the services 'ovirt-ha-agent' and 'ovirt-ha-broker' running fine and try to restart them once ?

> > I know there is no failed install left on the gluster volume, because when I attempt an install, part of my scripted prep process is deleting and recreating the Gluster volume. The below instructions are more or less what I’m doing already in a script[1]. (the gluster portion of the script process is: stop the volume, delete the volume, remove the mount point directory to avoid Gluster’s xattr problem with recycling directories, recreate the directory, change perms, create the volume, start the volume, set Ovirt-recc’ed volume options.) > > -j > > [1] We have a requirement for automated setup of all production resources, so all of this ends up being scripted. > >> 1) vdsClient -s 0 list table | awk '{print $1}' | xargs vdsClient -s 0 destroy >> >> 2) stop the volume and delete all the information inside the bricks from all the hosts >> >> 3) try to umount storage from /rhev/data-center/mnt/ - umount -f /rhev/data-center/mnt/ if it is mounted >> >> 4) remove all dirs from /rhev/data-center/mnt/ - rm -rf /rhev/data-center/mnt/* >> >> 5) start volume again and start the deployment. >> >> Thanks >> kasturi >>> >>>>> If I start it manually, the default DC is down, the default cluster has the installation host in the cluster, there is no storage, and the VM doesn’t show up in the GUI. In this install run, I have not yet started the engine manually. >>>> you wont be seeing HE vm until HE storage is imported into the UI. HE storage will be automatically imported into the UI (which will import HE vm too )once a master domain is present . >>> Sure; I’m just attempting to provide context. >>> >>>>> I assume this is related to the errors in ovirt-hosted-engine-setup.log, below. (The timestamps are confusing; it looks like the Python errors are logged some time after they’re captured or something.) The HA broker and agent logs just show them looping in the sequence below. >>>>> >>>>> Is there a decent way to pick this up and continue? If not, how do I make this work? >>>> Can you please check the following things. >>>> >>>> 1) is glusterd running on all the nodes ? 'systemctl status glistered’ >>>> 2) Are you able to connect to your storage server which is ovirt_engine in your case. >>>> 3) Can you check if all the brick process in the volume is up ? >>> 1) Verified that glusterd is running on all three nodes. >>> >>> 2) >>> [root@sc5-thing-1]# mount -tglusterfs sc5-gluster-1:/ovirt_engine /mnt/ovirt_engine >>> [root@sc5-thing-1]# df -h >>> Filesystem Size Used Avail Use% Mounted on >>> […] >>> sc5-gluster-1:/ovirt_engine 300G 2.6G 298G 1% /mnt/ovirt_engine >>> >>> >>> 3) >>> [root@sc5-gluster-1 jlawrence]# gluster volume status >>> Status of volume: ovirt_engine >>> Gluster process TCP Port RDMA Port Online Pid >>> ------------------------------------------------------------------------------ >>> Brick sc5-gluster-1:/gluster-bricks/ovirt_e >>> ngine/ovirt_engine-1 49217 0 Y 22102 >>> Brick sc5-gluster-2:/gluster-bricks/ovirt_e >>> ngine/ovirt_engine-1 49157 0 Y 37842 >>> Brick sc5-gluster-3:/gluster-bricks/ovirt_e >>> ngine/ovirt_engine-1 49157 0 Y 112018 >>> Self-heal Daemon on localhost N/A N/A Y 113364 >>> Self-heal Daemon on sc5-gluster-2.squaretra >>> de.com N/A N/A Y 19344 >>> Self-heal Daemon on sc5-gluster-3.squaretra >>> de.com N/A N/A Y 37956 >>> >>> Task Status of Volume ovirt_engine >>> ------------------------------------------------------------------------------ >>> There are no active volume tasks >>> >>> Volume ovirt_images is not started >>> >>> >>> >>> >>>> Thanks >>>> kasturi. >>> Thank you! >>> >>> -j >>> >>> >>>>> Thanks, >>>>> >>>>> -j >>>>> >>>>> - - - - ovirt-hosted-engine-setup.log snippet: - - - - >>>>> >>>>> 2017-04-19 12:29:55 DEBUG otopi.context context._executeMethod:128 Stage late_setup METHOD otopi.plugins.gr_he_setup.system.vdsmenv.Plugin._late_setup >>>>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd systemd.status:90 check service vdsmd status >>>>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/bin/systemctl', 'status', 'vdsmd.service'), executable='None', cwd='None', env=None >>>>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/bin/systemctl', 'status', 'vdsmd.service'), rc=0 >>>>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/bin/systemctl', 'status', 'vdsmd.service') stdout: >>>>> ● vdsmd.service - Virtual Desktop Server Manager >>>>> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) >>>>> Active: active (running) since Wed 2017-04-19 12:26:59 PDT; 2min 55s ago >>>>> Process: 67370 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) >>>>> Process: 69995 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) >>>>> Main PID: 70062 (vdsm) >>>>> CGroup: /system.slice/vdsmd.service >>>>> └─70062 /usr/bin/python2 /usr/share/vdsm/vdsm >>>>> >>>>> Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1) >>>>> Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed to retrieve Hosted Engine HA info >>>>> Traceback (most recent call last): >>>>> File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo >>>>> stats = instance.get_all_stats() >>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats >>>>> with broker.connection(self._retries, self._wait): >>>>> File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ >>>>> return self.gen.next() >>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection >>>>> self.connect(retries, wait) >>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect >>>>> raise BrokerConnectionError(error_msg) >>>>> BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1) >>>>> Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1) >>>>> Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed to retrieve Hosted Engine HA info >>>>> Traceback (most recent call last): >>>>> File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo >>>>> stats = instance.get_all_stats() >>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats >>>>> with broker.connection(self._retries, self._wait): >>>>> File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ >>>>> return self.gen.next() >>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection >>>>> self.connect(retries, wait) >>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect >>>>> raise BrokerConnectionError(error_msg) >>>>> BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1) >>>>> Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN File: /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.com.redhat.rhevm.vdsm already removed >>>>> Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN File: /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.org.qemu.guest_agent.0 already removed >>>>> Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1) >>>>> Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed to retrieve Hosted Engine HA info >>>>> Traceback (most recent call last): >>>>> File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo >>>>> stats = instance.get_all_stats() >>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats >>>>> with broker.connection(self._retries, self._wait): >>>>> File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ >>>>> return self.gen.next() >>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection >>>>> self.connect(retries, wait) >>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect >>>>> raise BrokerConnectionError(error_msg) >>>>> BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1) >>>>> Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to broker, the number of errors has exceeded the limit (1) >>>>> Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed to retrieve Hosted Engine HA info >>>>> Traceback (most recent call last): >>>>> File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo >>>>> stats = instance.get_all_stats() >>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats >>>>> with broker.connection(self._retries, self._wait): >>>>> File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ >>>>> return self.gen.next() >>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection >>>>> self.connect(retries, wait) >>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect >>>>> raise BrokerConnectionError(error_msg) >>>>> BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1) >>>>> >>>>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/bin/systemctl', 'status', 'vdsmd.service') stderr: >>>>> >>>>> >>>>> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.system.vdsmenv util.connect_vdsm_json_rpc:209 {'status': {'message': 'Done', 'code': 0}, u'systemProductName': u'PowerEdge R630', u'systemSerialNumber': u'GTH5942', u'systemFamily': u'PowerEdge', u'systemUUID': u'4C4C4544-0054-4810-8035-C7C04F393432', u'systemManufacturer': u'Dell Inc.'} >>>>> 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN >>>>> 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:770 ENV OVEHOSTED_VDSM/vdscli=_Server:'<vdsm.jsonrpcvdscli._Server object at 0x34a8290>' >>>>> 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:774 ENVIRONMENT DUMP - END >>>>> 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage late_setup METHOD otopi.plugins.gr_he_common.vm.misc.Plugin._late_setup >>>>> 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:134 condition False >>>>> 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage late_setup METHOD otopi.plugins.gr_he_setup.pki.vdsmpki.Plugin._late_setup >>>>> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki plugin.executeRaw:813 execute: ('/bin/openssl', 'x509', '-noout', '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), executable='None', cwd='None', env=None >>>>> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki plugin.executeRaw:863 execute-result: ('/bin/openssl', 'x509', '-noout', '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), rc=0 >>>>> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki plugin.execute:921 execute-output: ('/bin/openssl', 'x509', '-noout', '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem') stdout: >>>>> >>>>> - - - - broker.log: - - - - >>>>> >>>>> Thread-2970::INFO::2017-04-19 14:29:51,591::cpu_load_no_engine::113::cpu_load_no_engine.EngineHealth::(calculate_load) System load total=0.0117, engine=0.0000, non-engine=0.0117 >>>>> Thread-2976::INFO::2017-04-19 14:29:56,347::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established >>>>> Thread-2976::INFO::2017-04-19 14:29:56,361::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed >>>>> Thread-2977::INFO::2017-04-19 14:29:57,628::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established >>>>> Thread-2977::INFO::2017-04-19 14:29:57,629::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor ping >>>>> Thread-2977::INFO::2017-04-19 14:29:57,631::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor ping, id 140603614328464 >>>>> Thread-2977::INFO::2017-04-19 14:29:57,662::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor mgmt-bridge >>>>> Thread-2977::INFO::2017-04-19 14:29:57,667::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor mgmt-bridge, id 140603614326992 >>>>> Thread-2977::INFO::2017-04-19 14:29:57,688::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor mem-free >>>>> Thread-2977::INFO::2017-04-19 14:29:57,735::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor mem-free, id 140604548742096 >>>>> Thread-2977::INFO::2017-04-19 14:29:57,756::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor cpu-load-no-engine >>>>> Thread-2977::INFO::2017-04-19 14:29:57,798::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor cpu-load-no-engine, id 43463568 >>>>> Thread-2977::INFO::2017-04-19 14:29:57,841::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Starting submonitor engine-health >>>>> Thread-2977::INFO::2017-04-19 14:29:57,863::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started submonitor engine-health, id 140604881915728 >>>>> Thread-2983::INFO::2017-04-19 14:29:58,153::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established >>>>> Thread-2984::INFO::2017-04-19 14:30:02,179::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established >>>>> Thread-2977::INFO::2017-04-19 14:30:02,249::storage_backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks) Cleaning up stale LV link '/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_agent/hosted-engine.lockspace' >>>>> Thread-2977::INFO::2017-04-19 14:30:02,321::storage_backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks) Cleaning up stale LV link '/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_agent/hosted-engine.metadata' >>>>> Thread-2983::INFO::2017-04-19 14:30:02,334::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed >>>>> Thread-2984::INFO::2017-04-19 14:30:02,340::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed >>>>> Thread-2969::INFO::2017-04-19 14:30:02,343::mem_free::50::mem_free.MemFree::(action) memFree: 254533 >>>>> Thread-2980::INFO::2017-04-19 14:30:02,382::mem_free::50::mem_free.MemFree::(action) memFree: 254530 >>>>> Thread-2985::INFO::2017-04-19 14:30:07,991::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established >>>>> Thread-2985::INFO::2017-04-19 14:30:08,082::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed >>>>> Thread-2969::INFO::2017-04-19 14:30:08,129::mem_free::50::mem_free.MemFree::(action) memFree: 254534 >>>>> Thread-2966::INFO::2017-04-19 14:30:09,719::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed >>>>> Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor ping, id 140603546975120 >>>>> Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor ping, id 140603546975120 >>>>> Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor mgmt-bridge, id 140603546973136 >>>>> Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor mgmt-bridge, id 140603546973136 >>>>> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor mem-free, id 140603546973264 >>>>> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor mem-free, id 140603546973264 >>>>> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor cpu-load-no-engine, id 140603546976144 >>>>> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor cpu-load-no-engine, id 140603546976144 >>>>> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping submonitor engine-health, id 140603546974096 >>>>> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped submonitor engine-health, id 140603546974096 >>>>> Thread-2979::INFO::2017-04-19 14:30:11,145::mgmt_bridge::55::mgmt_bridge.MgmtBridge::(action) Found bridge ovirtmgmt with ports >>>>> Thread-2986::INFO::2017-04-19 14:30:11,462::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established >>>>> Thread-2986::INFO::2017-04-19 14:30:11,480::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed >>>>> Thread-2987::INFO::2017-04-19 14:30:11,481::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) Connection established >>>>> Thread-2987::INFO::2017-04-19 14:30:11,488::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Connection closed >>>>> Thread-2980::INFO::2017-04-19 14:30:11,502::mem_free::50::mem_free.MemFree::(action) memFree: 254535 >>>>> >>>>> - - - - ha agent.log: - - - - >>>>> >>>>> MainThread::WARNING::2017-04-19 14:31:16,352::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '6' >>>>> MainThread::INFO::2017-04-19 14:31:16,383::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: sc5-ovirt-2.squaretrade.com >>>>> MainThread::INFO::2017-04-19 14:31:16,385::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM >>>>> MainThread::INFO::2017-04-19 14:31:18,863::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage >>>>> MainThread::INFO::2017-04-19 14:31:18,864::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server >>>>> MainThread::INFO::2017-04-19 14:31:23,672::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server >>>>> MainThread::INFO::2017-04-19 14:31:23,689::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain >>>>> MainThread::INFO::2017-04-19 14:31:23,956::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images >>>>> MainThread::INFO::2017-04-19 14:31:23,957::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images >>>>> MainThread::INFO::2017-04-19 14:31:26,768::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf >>>>> MainThread::INFO::2017-04-19 14:31:26,769::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain >>>>> MainThread::INFO::2017-04-19 14:31:26,769::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE >>>>> MainThread::WARNING::2017-04-19 14:31:29,548::ovf_store::107::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE >>>>> MainThread::ERROR::2017-04-19 14:31:29,551::config::450::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs >>>>> MainThread::INFO::2017-04-19 14:31:29,576::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection >>>>> MainThread::INFO::2017-04-19 14:31:29,577::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '10.181.26.1'} >>>>> MainThread::INFO::2017-04-19 14:31:29,596::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881712208 >>>>> MainThread::INFO::2017-04-19 14:31:29,596::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'} >>>>> MainThread::INFO::2017-04-19 14:31:29,600::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881712016 >>>>> MainThread::INFO::2017-04-19 14:31:29,600::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'} >>>>> MainThread::INFO::2017-04-19 14:31:29,602::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881711568 >>>>> MainThread::INFO::2017-04-19 14:31:29,602::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'} >>>>> MainThread::INFO::2017-04-19 14:31:29,605::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881711312 >>>>> MainThread::INFO::2017-04-19 14:31:29,605::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'} >>>>> MainThread::INFO::2017-04-19 14:31:29,608::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140604881714896 >>>>> MainThread::INFO::2017-04-19 14:31:34,326::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140604881713168 >>>>> MainThread::INFO::2017-04-19 14:31:34,327::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started >>>>> MainThread::ERROR::2017-04-19 14:31:34,372::agent::196::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Service sanlock is not running and the admin is responsible for starting it. Waiting… >>>>> >>>>> >>>>> - - - - answers.conf - - - - >>>>> >>>>> [environment:default] >>>>> OVEHOSTED_CORE/rollbackProceed=none:None >>>>> OVEHOSTED_CORE/screenProceed=none:None >>>>> OVEHOSTED_CORE/deployProceed=bool:True >>>>> OVEHOSTED_CORE/upgradeProceed=none:None >>>>> OVEHOSTED_CORE/confirmSettings=bool:True >>>>> OVEHOSTED_NETWORK/fqdn=str:sc5-ovirt-engine-1.squaretrade.com >>>>> OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt >>>>> OVEHOSTED_NETWORK/firewallManager=none:None >>>>> OVEHOSTED_NETWORK/gateway=str:10.181.26.1 >>>>> OVEHOSTED_ENGINE/insecureSSL=none:None >>>>> OVEHOSTED_ENGINE/clusterName=str:Default >>>>> OVEHOSTED_STORAGE/storageDatacenterName=str:hosted_datacenter >>>>> OVEHOSTED_STORAGE/domainType=str:glusterfs >>>>> OVEHOSTED_STORAGE/connectionUUID=str:c2dd62cb-79aa-4547-b191-b74e55fe5311 >>>>> OVEHOSTED_STORAGE/LunID=none:None >>>>> OVEHOSTED_STORAGE/imgSizeGB=str:50 >>>>> OVEHOSTED_STORAGE/mntOptions=none:None >>>>> OVEHOSTED_STORAGE/iSCSIPortalIPAddress=none:None >>>>> OVEHOSTED_STORAGE/metadataVolumeUUID=str:ac9d7fbf-fb41-4b96-8940-b95698e58b18 >>>>> OVEHOSTED_STORAGE/sdUUID=str:a81b229c-2422-4110-af45-1bc4d9a01bab >>>>> OVEHOSTED_STORAGE/iSCSITargetName=none:None >>>>> OVEHOSTED_STORAGE/metadataImageUUID=str:298fef26-3039-4c9e-bee8-39770fb2e821 >>>>> OVEHOSTED_STORAGE/lockspaceVolumeUUID=str:c90a1959-369d-4109-8a6e-5c81f735e43b >>>>> OVEHOSTED_STORAGE/iSCSIPortalPort=none:None >>>>> OVEHOSTED_STORAGE/imgUUID=str:284c61f2-a901-46f0-a05a-2e5bf220efc6 >>>>> OVEHOSTED_STORAGE/confImageUUID=str:8bcc7fae-2fbc-40aa-b75a-6c6adb8ba7dc >>>>> OVEHOSTED_STORAGE/spUUID=str:00000000-0000-0000-0000-000000000000 >>>>> OVEHOSTED_STORAGE/lockspaceImageUUID=str:0ce0f2f9-b9ea-45b3-b091-d4f0e46e150f >>>>> OVEHOSTED_ENGINE/enableHcGlusterService=bool:False >>>>> OVEHOSTED_STORAGE/storageDomainName=str:hosted_storage >>>>> OVEHOSTED_STORAGE/iSCSIPortal=none:None >>>>> OVEHOSTED_STORAGE/volUUID=str:16304df2-2d95-4d75-932d-21d7c6de000d >>>>> OVEHOSTED_STORAGE/vgUUID=none:None >>>>> OVEHOSTED_STORAGE/confVolUUID=str:3b42f578-5d38-4820-b977-fa37b49926f7 >>>>> OVEHOSTED_STORAGE/storageDomainConnection=str:sc5-gluster-1:/ovirt_engine >>>>> OVEHOSTED_STORAGE/iSCSIPortalUser=none:None >>>>> OVEHOSTED_VDSM/consoleType=str:qxl >>>>> OVEHOSTED_VM/vmMemSizeMB=int:16384 >>>>> OVEHOSTED_VM/vmUUID=str:04ff4cf1-135a-4918-9a1f-8023322f89a3 >>>>> OVEHOSTED_VM/vmMACAddr=str:00:16:3e:6e:86:b2 >>>>> OVEHOSTED_VM/emulatedMachine=str:pc >>>>> OVEHOSTED_VM/consoleUUID=str:7d26f23e-59b6-4378-a0a4-f50e9653ac78 >>>>> OVEHOSTED_VM/vmVCpus=str:4 >>>>> OVEHOSTED_VM/nicUUID=str:a2b7dc56-02f6-4585-bd50-079b17aafe1f >>>>> OVEHOSTED_VM/cdromUUID=str:cde48946-a7dc-41a5-87ee-854eb4d4dc22 >>>>> OVEHOSTED_VM/ovfArchive=str:/usr/share/ovirt-engine-appliance/ovirt-engine-appliance-4.1-20170406.1.el7.centos.ova >>>>> OVEHOSTED_VM/vmCDRom=none:None >>>>> OVEHOSTED_VM/automateVMShutdown=bool:True >>>>> OVEHOSTED_VM/cloudInitISO=str:generate >>>>> OVEHOSTED_VM/cloudinitInstanceDomainName=str:squaretrade.com >>>>> OVEHOSTED_VM/cloudinitInstanceHostName=str:sc5-ovirt-engine-1.squaretrade... >>>>> OVEHOSTED_VM/rootSshPubkey=str:ssh-rsa [snip] >>>>> OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True >>>>> OVEHOSTED_VM/cloudinitVMStaticCIDR=[snip] >>>>> OVEHOSTED_VM/cloudinitVMTZ=str:America/Los_Angeles >>>>> OVEHOSTED_VM/rootSshAccess=str:yes >>>>> OVEHOSTED_VM/cloudinitVMETCHOSTS=bool:False >>>>> OVEHOSTED_VM/cloudinitVMDNS=[snip] >>>>> OVEHOSTED_VDSM/spicePkiSubject=str:O=squaretrade.com, CN=sc5-ovirt-2.squaretrade.com >>>>> OVEHOSTED_VDSM/pkiSubject=str:/C=EN/L=Test/O=Test/CN=Test >>>>> OVEHOSTED_VDSM/caSubject=str:/C=EN/L=Test/O=Test/CN=TestCA >>>>> OVEHOSTED_VDSM/cpu=str:model_Haswell-noTSX >>>>> OVEHOSTED_NOTIF/smtpPort=str:25 >>>>> OVEHOSTED_NOTIF/smtpServer=[snip] >>>>> OVEHOSTED_NOTIF/sourceEmail=[snip] >>>>> OVEHOSTED_NOTIF/destEmail=[snip] >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users