[ovirt-users] Hosted engine install failed; vdsm upset about broker (revised)
Simone Tiraboschi
stirabos at redhat.com
Thu Apr 20 16:18:51 UTC 2017
On Thu, Apr 20, 2017 at 2:14 AM, Jamie Lawrence <jlawrence at squaretrade.com>
wrote:
>
> So, tracing this further, I’m pretty sure this is something about sanlock.
>
> As best I can tell this[1] seems to be the failure that is blocking
> importing the pool, creating storage domains, importing the HE, etc.
> Contrary to the log, sanlock is running; I verified it starts on
> system-boot and restarts just fine.
>
> I found one reference to someone having a similar problem in 3.6, but that
> appeared to have been a permission issue I’m not afflicted with.
>
> How can I move past this?
>
Could you please share the output of
sudo -u vdsm sudo service sanlock status
?
>
> TIA,
>
> -j
>
>
> [1] agent.log:
> MainThread::WARNING::2017-04-19 17:07:13,537::agent::209::
> ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent,
> attempt '6'
> MainThread::INFO::2017-04-19 17:07:13,567::hosted_engine::
> 242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname)
> Found certificate common name: sc5-ovirt-2.squaretrade.com
> MainThread::INFO::2017-04-19 17:07:13,569::hosted_engine::
> 604::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_vdsm) Initializing VDSM
> MainThread::INFO::2017-04-19 17:07:16,044::hosted_engine::
> 630::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Connecting the storage
> MainThread::INFO::2017-04-19 17:07:16,045::storage_server::
> 219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-04-19 17:07:20,876::storage_server::
> 226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-04-19 17:07:20,893::storage_server::
> 233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
> MainThread::INFO::2017-04-19 17:07:21,160::hosted_engine::
> 657::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Preparing images
> MainThread::INFO::2017-04-19 17:07:21,160::image::126::
> ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
> MainThread::INFO::2017-04-19 17:07:23,954::hosted_engine::
> 660::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Refreshing vm.conf
> MainThread::INFO::2017-04-19 17:07:23,955::config::485::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-04-19 17:07:23,955::config::412::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.
> config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher
> copy of vm configuration from the OVF_STORE
> MainThread::WARNING::2017-04-19 17:07:26,741::ovf_store::107::
> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find
> OVF_STORE
> MainThread::ERROR::2017-04-19 17:07:26,744::config::450::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.
> config::(_get_vm_conf_content_from_ovf_store) Unable to identify the
> OVF_STORE volume, falling back to initial vm.conf. Please ensure you
> already added your first data domain for regular VMs
> MainThread::INFO::2017-04-19 17:07:26,770::hosted_engine::
> 509::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_broker) Initializing ha-broker connection
> MainThread::INFO::2017-04-19 17:07:26,771::brokerlink::130:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Starting monitor ping, options {'addr': '10.181.26.1'}
> MainThread::INFO::2017-04-19 17:07:26,774::brokerlink::141:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Success, id 140621269798096
> MainThread::INFO::2017-04-19 17:07:26,774::brokerlink::130:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name':
> 'ovirtmgmt', 'address': '0'}
> MainThread::INFO::2017-04-19 17:07:26,791::brokerlink::141:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Success, id 140621269798544
> MainThread::INFO::2017-04-19 17:07:26,792::brokerlink::130:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
> MainThread::INFO::2017-04-19 17:07:26,793::brokerlink::141:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Success, id 140621269798224
> MainThread::INFO::2017-04-19 17:07:26,794::brokerlink::130:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid':
> '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
> MainThread::INFO::2017-04-19 17:07:26,796::brokerlink::141:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Success, id 140621269796816
> MainThread::INFO::2017-04-19 17:07:26,796::brokerlink::130:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid':
> '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
> MainThread::INFO::2017-04-19 17:07:26,798::brokerlink::141:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Success, id 140621269797840
> MainThread::INFO::2017-04-19 17:07:31,513::brokerlink::179:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain)
> Success, id 140621269798480
> MainThread::INFO::2017-04-19 17:07:31,513::hosted_engine::
> 601::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_broker) Broker initialized, all submonitors
> started
> MainThread::ERROR::2017-04-19 17:07:31,556::agent::196::
> ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Service sanlock is
> not running and the admin is responsible for starting it. Waiting...
> MainThread::WARNING::2017-04-19 17:07:36,562::agent::209::
> ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent,
> attempt '7'
> MainThread::INFO::2017-04-19 17:07:36,592::hosted_engine::
> 242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname)
> Found certificate common name: sc5-ovirt-2.squaretrade.com
>
>
>
>
> - - - prior message below - - -
>
> I trialed installing the hosted engine, following the instructions at
> http://www.ovirt.org/documentation/self-hosted/chap-Deploying_Self-Hosted_
> Engine/ . This is using Gluster as the backend storage subsystem.
>
> Answer file at the end.
>
> Per the docs,
>
> "When the hosted-engine deployment script completes successfully, the
> oVirt Engine is configured and running on your host. The Engine has already
> configured the data center, cluster, host, the Engine virtual machine, and
> a shared storage domain dedicated to the Engine virtual machine.”
>
> In my case, this is false. The installation claims success, but the
> hosted engine VM stays stopped, unless I start it manually. If I start it
> manually, the default DC is down, the default cluster has the installation
> host in the cluster, there is no storage, and the VM doesn’t show up in
> the GUI. In this install run, I have not yet started the engine manually.
>
> I assume this is related to the errors in ovirt-hosted-engine-setup.log,
> below. (The timestamps are confusing; it looks like the Python errors are
> logged some time after they’re captured or something.) The HA broker and
> agent logs just show them looping in the sequence below.
>
> Is there a decent way to pick this up and continue? If not, how do I make
> this work?
>
> Thanks,
>
> -j
>
> - - - - ovirt-hosted-engine-setup.log snippet: - - - -
>
> 2017-04-19 12:29:55 DEBUG otopi.context context._executeMethod:128 Stage
> late_setup METHOD otopi.plugins.gr_he_setup.system.vdsmenv.Plugin._late_
> setup
> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd
> systemd.status:90 check service vdsmd status
> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd
> plugin.executeRaw:813 execute: ('/bin/systemctl', 'status',
> 'vdsmd.service'), executable='None', cwd='None', env=None
> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd
> plugin.executeRaw:863 execute-result: ('/bin/systemctl', 'status',
> 'vdsmd.service'), rc=0
> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd
> plugin.execute:921 execute-output: ('/bin/systemctl', 'status',
> 'vdsmd.service') stdout:
> ● vdsmd.service - Virtual Desktop Server Manager
> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor
> preset: enabled)
> Active: active (running) since Wed 2017-04-19 12:26:59 PDT; 2min 55s ago
> Process: 67370 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh
> --post-stop (code=exited, status=0/SUCCESS)
> Process: 69995 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
> --pre-start (code=exited, status=0/SUCCESS)
> Main PID: 70062 (vdsm)
> CGroup: /system.slice/vdsmd.service
> └─70062 /usr/bin/python2 /usr/share/vdsm/vdsm
>
> Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm
> ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect
> to broker, the number of errors has exceeded the limit (1)
> Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR
> failed to retrieve Hosted Engine HA info
> Traceback (most
> recent call last):
> File
> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in
> _getHaInfo
> stats =
> instance.get_all_stats()
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 102, in get_all_stats
> with
> broker.connection(self._retries, self._wait):
> File
> "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> return
> self.gen.next()
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 99, in connection
>
> self.connect(retries, wait)
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 78, in connect
> raise
> BrokerConnectionError(error_msg)
>
> BrokerConnectionError: Failed to connect to broker, the number of errors
> has exceeded the limit (1)
> Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm
> ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect
> to broker, the number of errors has exceeded the limit (1)
> Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR
> failed to retrieve Hosted Engine HA info
> Traceback (most
> recent call last):
> File
> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in
> _getHaInfo
> stats =
> instance.get_all_stats()
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 102, in get_all_stats
> with
> broker.connection(self._retries, self._wait):
> File
> "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> return
> self.gen.next()
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 99, in connection
>
> self.connect(retries, wait)
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 78, in connect
> raise
> BrokerConnectionError(error_msg)
>
> BrokerConnectionError: Failed to connect to broker, the number of errors
> has exceeded the limit (1)
> Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN
> File: /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-
> 9a1f-8023322f89a3.com.redhat.rhevm.vdsm already removed
> Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN
> File: /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-
> 9a1f-8023322f89a3.org.qemu.guest_agent.0 already removed
> Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm
> ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect
> to broker, the number of errors has exceeded the limit (1)
> Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR
> failed to retrieve Hosted Engine HA info
> Traceback (most
> recent call last):
> File
> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in
> _getHaInfo
> stats =
> instance.get_all_stats()
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 102, in get_all_stats
> with
> broker.connection(self._retries, self._wait):
> File
> "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> return
> self.gen.next()
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 99, in connection
>
> self.connect(retries, wait)
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 78, in connect
> raise
> BrokerConnectionError(error_msg)
>
> BrokerConnectionError: Failed to connect to broker, the number of errors
> has exceeded the limit (1)
> Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm
> ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect
> to broker, the number of errors has exceeded the limit (1)
> Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR
> failed to retrieve Hosted Engine HA info
> Traceback (most
> recent call last):
> File
> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in
> _getHaInfo
> stats =
> instance.get_all_stats()
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 102, in get_all_stats
> with
> broker.connection(self._retries, self._wait):
> File
> "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> return
> self.gen.next()
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 99, in connection
>
> self.connect(retries, wait)
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 78, in connect
> raise
> BrokerConnectionError(error_msg)
>
> BrokerConnectionError: Failed to connect to broker, the number of errors
> has exceeded the limit (1)
>
> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd
> plugin.execute:926 execute-output: ('/bin/systemctl', 'status',
> 'vdsmd.service') stderr:
>
>
> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.system.vdsmenv
> util.connect_vdsm_json_rpc:209 {'status': {'message': 'Done', 'code': 0},
> u'systemProductName': u'PowerEdge R630', u'systemSerialNumber': u'GTH5942',
> u'systemFamily': u'PowerEdge', u'systemUUID': u'4C4C4544-0054-4810-8035-C7C04F393432',
> u'systemManufacturer': u'Dell Inc.'}
> 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:760
> ENVIRONMENT DUMP - BEGIN
> 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:770 ENV
> OVEHOSTED_VDSM/vdscli=_Server:'<vdsm.jsonrpcvdscli._Server object at
> 0x34a8290>'
> 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:774
> ENVIRONMENT DUMP - END
> 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage
> late_setup METHOD otopi.plugins.gr_he_common.vm.misc.Plugin._late_setup
> 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:134
> condition False
> 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage
> late_setup METHOD otopi.plugins.gr_he_setup.pki.vdsmpki.Plugin._late_setup
> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki
> plugin.executeRaw:813 execute: ('/bin/openssl', 'x509', '-noout', '-text',
> '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), executable='None',
> cwd='None', env=None
> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki
> plugin.executeRaw:863 execute-result: ('/bin/openssl', 'x509', '-noout',
> '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), rc=0
> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki
> plugin.execute:921 execute-output: ('/bin/openssl', 'x509', '-noout',
> '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem') stdout:
>
> - - - - broker.log: - - - -
>
> Thread-2970::INFO::2017-04-19 14:29:51,591::cpu_load_no_
> engine::113::cpu_load_no_engine.EngineHealth::(calculate_load) System
> load total=0.0117, engine=0.0000, non-engine=0.0117
> Thread-2976::INFO::2017-04-19 14:29:56,347::listener::134::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
> Connection established
> Thread-2976::INFO::2017-04-19 14:29:56,361::listener::186::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
> Connection closed
> Thread-2977::INFO::2017-04-19 14:29:57,628::listener::134::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
> Connection established
> Thread-2977::INFO::2017-04-19 14:29:57,629::monitor::66::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
> Starting submonitor ping
> Thread-2977::INFO::2017-04-19 14:29:57,631::monitor::79::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started
> submonitor ping, id 140603614328464
> Thread-2977::INFO::2017-04-19 14:29:57,662::monitor::66::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
> Starting submonitor mgmt-bridge
> Thread-2977::INFO::2017-04-19 14:29:57,667::monitor::79::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started
> submonitor mgmt-bridge, id 140603614326992
> Thread-2977::INFO::2017-04-19 14:29:57,688::monitor::66::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
> Starting submonitor mem-free
> Thread-2977::INFO::2017-04-19 14:29:57,735::monitor::79::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started
> submonitor mem-free, id 140604548742096
> Thread-2977::INFO::2017-04-19 14:29:57,756::monitor::66::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
> Starting submonitor cpu-load-no-engine
> Thread-2977::INFO::2017-04-19 14:29:57,798::monitor::79::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started
> submonitor cpu-load-no-engine, id 43463568
> Thread-2977::INFO::2017-04-19 14:29:57,841::monitor::66::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
> Starting submonitor engine-health
> Thread-2977::INFO::2017-04-19 14:29:57,863::monitor::79::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started
> submonitor engine-health, id 140604881915728
> Thread-2983::INFO::2017-04-19 14:29:58,153::listener::134::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
> Connection established
> Thread-2984::INFO::2017-04-19 14:30:02,179::listener::134::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
> Connection established
> Thread-2977::INFO::2017-04-19 14:30:02,249::storage_
> backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks)
> Cleaning up stale LV link '/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_
> ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_
> agent/hosted-engine.lockspace'
> Thread-2977::INFO::2017-04-19 14:30:02,321::storage_
> backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks)
> Cleaning up stale LV link '/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_
> ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_
> agent/hosted-engine.metadata'
> Thread-2983::INFO::2017-04-19 14:30:02,334::listener::186::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
> Connection closed
> Thread-2984::INFO::2017-04-19 14:30:02,340::listener::186::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
> Connection closed
> Thread-2969::INFO::2017-04-19 14:30:02,343::mem_free::50::mem_free.MemFree::(action)
> memFree: 254533
> Thread-2980::INFO::2017-04-19 14:30:02,382::mem_free::50::mem_free.MemFree::(action)
> memFree: 254530
> Thread-2985::INFO::2017-04-19 14:30:07,991::listener::134::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
> Connection established
> Thread-2985::INFO::2017-04-19 14:30:08,082::listener::186::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
> Connection closed
> Thread-2969::INFO::2017-04-19 14:30:08,129::mem_free::50::mem_free.MemFree::(action)
> memFree: 254534
> Thread-2966::INFO::2017-04-19 14:30:09,719::listener::186::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
> Connection closed
> Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::90::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping
> submonitor ping, id 140603546975120
> Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::99::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped
> submonitor ping, id 140603546975120
> Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::90::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping
> submonitor mgmt-bridge, id 140603546973136
> Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::99::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped
> submonitor mgmt-bridge, id 140603546973136
> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping
> submonitor mem-free, id 140603546973264
> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped
> submonitor mem-free, id 140603546973264
> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping
> submonitor cpu-load-no-engine, id 140603546976144
> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped
> submonitor cpu-load-no-engine, id 140603546976144
> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping
> submonitor engine-health, id 140603546974096
> Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99::
> ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped
> submonitor engine-health, id 140603546974096
> Thread-2979::INFO::2017-04-19 14:30:11,145::mgmt_bridge::55:
> :mgmt_bridge.MgmtBridge::(action) Found bridge ovirtmgmt with ports
> Thread-2986::INFO::2017-04-19 14:30:11,462::listener::134::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
> Connection established
> Thread-2986::INFO::2017-04-19 14:30:11,480::listener::186::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
> Connection closed
> Thread-2987::INFO::2017-04-19 14:30:11,481::listener::134::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
> Connection established
> Thread-2987::INFO::2017-04-19 14:30:11,488::listener::186::
> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
> Connection closed
> Thread-2980::INFO::2017-04-19 14:30:11,502::mem_free::50::mem_free.MemFree::(action)
> memFree: 254535
>
> - - - - ha agent.log: - - - -
>
> MainThread::WARNING::2017-04-19 14:31:16,352::agent::209::
> ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent,
> attempt '6'
> MainThread::INFO::2017-04-19 14:31:16,383::hosted_engine::
> 242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname)
> Found certificate common name: sc5-ovirt-2.squaretrade.com
> MainThread::INFO::2017-04-19 14:31:16,385::hosted_engine::
> 604::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_vdsm) Initializing VDSM
> MainThread::INFO::2017-04-19 14:31:18,863::hosted_engine::
> 630::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Connecting the storage
> MainThread::INFO::2017-04-19 14:31:18,864::storage_server::
> 219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-04-19 14:31:23,672::storage_server::
> 226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-04-19 14:31:23,689::storage_server::
> 233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
> MainThread::INFO::2017-04-19 14:31:23,956::hosted_engine::
> 657::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Preparing images
> MainThread::INFO::2017-04-19 14:31:23,957::image::126::
> ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
> MainThread::INFO::2017-04-19 14:31:26,768::hosted_engine::
> 660::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Refreshing vm.conf
> MainThread::INFO::2017-04-19 14:31:26,769::config::485::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-04-19 14:31:26,769::config::412::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.
> config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher
> copy of vm configuration from the OVF_STORE
> MainThread::WARNING::2017-04-19 14:31:29,548::ovf_store::107::
> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find
> OVF_STORE
> MainThread::ERROR::2017-04-19 14:31:29,551::config::450::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.
> config::(_get_vm_conf_content_from_ovf_store) Unable to identify the
> OVF_STORE volume, falling back to initial vm.conf. Please ensure you
> already added your first data domain for regular VMs
> MainThread::INFO::2017-04-19 14:31:29,576::hosted_engine::
> 509::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_broker) Initializing ha-broker connection
> MainThread::INFO::2017-04-19 14:31:29,577::brokerlink::130:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Starting monitor ping, options {'addr': '10.181.26.1'}
> MainThread::INFO::2017-04-19 14:31:29,596::brokerlink::141:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Success, id 140604881712208
> MainThread::INFO::2017-04-19 14:31:29,596::brokerlink::130:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name':
> 'ovirtmgmt', 'address': '0'}
> MainThread::INFO::2017-04-19 14:31:29,600::brokerlink::141:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Success, id 140604881712016
> MainThread::INFO::2017-04-19 14:31:29,600::brokerlink::130:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
> MainThread::INFO::2017-04-19 14:31:29,602::brokerlink::141:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Success, id 140604881711568
> MainThread::INFO::2017-04-19 14:31:29,602::brokerlink::130:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid':
> '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
> MainThread::INFO::2017-04-19 14:31:29,605::brokerlink::141:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Success, id 140604881711312
> MainThread::INFO::2017-04-19 14:31:29,605::brokerlink::130:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid':
> '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
> MainThread::INFO::2017-04-19 14:31:29,608::brokerlink::141:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Success, id 140604881714896
> MainThread::INFO::2017-04-19 14:31:34,326::brokerlink::179:
> :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain)
> Success, id 140604881713168
> MainThread::INFO::2017-04-19 14:31:34,327::hosted_engine::
> 601::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_broker) Broker initialized, all submonitors
> started
> MainThread::ERROR::2017-04-19 14:31:34,372::agent::196::
> ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Service sanlock is
> not running and the admin is responsible for starting it. Waiting…
>
>
> - - - - answers.conf - - - -
>
> [environment:default]
> OVEHOSTED_CORE/rollbackProceed=none:None
> OVEHOSTED_CORE/screenProceed=none:None
> OVEHOSTED_CORE/deployProceed=bool:True
> OVEHOSTED_CORE/upgradeProceed=none:None
> OVEHOSTED_CORE/confirmSettings=bool:True
> OVEHOSTED_NETWORK/fqdn=str:sc5-ovirt-engine-1.squaretrade.com
> OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
> OVEHOSTED_NETWORK/firewallManager=none:None
> OVEHOSTED_NETWORK/gateway=str:10.181.26.1
> OVEHOSTED_ENGINE/insecureSSL=none:None
> OVEHOSTED_ENGINE/clusterName=str:Default
> OVEHOSTED_STORAGE/storageDatacenterName=str:hosted_datacenter
> OVEHOSTED_STORAGE/domainType=str:glusterfs
> OVEHOSTED_STORAGE/connectionUUID=str:c2dd62cb-79aa-4547-b191-b74e55fe5311
> OVEHOSTED_STORAGE/LunID=none:None
> OVEHOSTED_STORAGE/imgSizeGB=str:50
> OVEHOSTED_STORAGE/mntOptions=none:None
> OVEHOSTED_STORAGE/iSCSIPortalIPAddress=none:None
> OVEHOSTED_STORAGE/metadataVolumeUUID=str:ac9d7fbf-fb41-4b96-8940-
> b95698e58b18
> OVEHOSTED_STORAGE/sdUUID=str:a81b229c-2422-4110-af45-1bc4d9a01bab
> OVEHOSTED_STORAGE/iSCSITargetName=none:None
> OVEHOSTED_STORAGE/metadataImageUUID=str:298fef26-3039-4c9e-bee8-
> 39770fb2e821
> OVEHOSTED_STORAGE/lockspaceVolumeUUID=str:c90a1959-369d-4109-8a6e-
> 5c81f735e43b
> OVEHOSTED_STORAGE/iSCSIPortalPort=none:None
> OVEHOSTED_STORAGE/imgUUID=str:284c61f2-a901-46f0-a05a-2e5bf220efc6
> OVEHOSTED_STORAGE/confImageUUID=str:8bcc7fae-2fbc-40aa-b75a-6c6adb8ba7dc
> OVEHOSTED_STORAGE/spUUID=str:00000000-0000-0000-0000-000000000000
> OVEHOSTED_STORAGE/lockspaceImageUUID=str:0ce0f2f9-b9ea-45b3-b091-
> d4f0e46e150f
> OVEHOSTED_ENGINE/enableHcGlusterService=bool:False
> OVEHOSTED_STORAGE/storageDomainName=str:hosted_storage
> OVEHOSTED_STORAGE/iSCSIPortal=none:None
> OVEHOSTED_STORAGE/volUUID=str:16304df2-2d95-4d75-932d-21d7c6de000d
> OVEHOSTED_STORAGE/vgUUID=none:None
> OVEHOSTED_STORAGE/confVolUUID=str:3b42f578-5d38-4820-b977-fa37b49926f7
> OVEHOSTED_STORAGE/storageDomainConnection=str:sc5-gluster-1:/ovirt_engine
> OVEHOSTED_STORAGE/iSCSIPortalUser=none:None
> OVEHOSTED_VDSM/consoleType=str:qxl
> OVEHOSTED_VM/vmMemSizeMB=int:16384
> OVEHOSTED_VM/vmUUID=str:04ff4cf1-135a-4918-9a1f-8023322f89a3
> OVEHOSTED_VM/vmMACAddr=str:00:16:3e:6e:86:b2
> OVEHOSTED_VM/emulatedMachine=str:pc
> OVEHOSTED_VM/consoleUUID=str:7d26f23e-59b6-4378-a0a4-f50e9653ac78
> OVEHOSTED_VM/vmVCpus=str:4
> OVEHOSTED_VM/nicUUID=str:a2b7dc56-02f6-4585-bd50-079b17aafe1f
> OVEHOSTED_VM/cdromUUID=str:cde48946-a7dc-41a5-87ee-854eb4d4dc22
> OVEHOSTED_VM/ovfArchive=str:/usr/share/ovirt-engine-
> appliance/ovirt-engine-appliance-4.1-20170406.1.el7.centos.ova
> OVEHOSTED_VM/vmCDRom=none:None
> OVEHOSTED_VM/automateVMShutdown=bool:True
> OVEHOSTED_VM/cloudInitISO=str:generate
> OVEHOSTED_VM/cloudinitInstanceDomainName=str:squaretrade.com
> OVEHOSTED_VM/cloudinitInstanceHostName=str:sc5-ovirt-engine-1.
> squaretrade.com
> OVEHOSTED_VM/rootSshPubkey=str:ssh-rsa [snip]
> OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True
> OVEHOSTED_VM/cloudinitVMStaticCIDR=[snip]
> OVEHOSTED_VM/cloudinitVMTZ=str:America/Los_Angeles
> OVEHOSTED_VM/rootSshAccess=str:yes
> OVEHOSTED_VM/cloudinitVMETCHOSTS=bool:False
> OVEHOSTED_VM/cloudinitVMDNS=[snip]
> OVEHOSTED_VDSM/spicePkiSubject=str:O=squaretrade.com, CN=
> sc5-ovirt-2.squaretrade.com
> OVEHOSTED_VDSM/pkiSubject=str:/C=EN/L=Test/O=Test/CN=Test
> OVEHOSTED_VDSM/caSubject=str:/C=EN/L=Test/O=Test/CN=TestCA
> OVEHOSTED_VDSM/cpu=str:model_Haswell-noTSX
> OVEHOSTED_NOTIF/smtpPort=str:25
> OVEHOSTED_NOTIF/smtpServer=[snip]
> OVEHOSTED_NOTIF/sourceEmail=[snip]
> OVEHOSTED_NOTIF/destEmail=[snip]
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170420/a0fbe001/attachment-0001.html>
More information about the Users
mailing list