[ovirt-users] /var/run/ovirt-hosted-engine-ha/vm.conf not found

Thu Sep 1 12:41:08 EDT 2016

I seem to be unable to restart my ovirt hosted engine.....

I'd swear I had this issue once before, but couldn't find any notes on 
my end.

# cat /etc/ovirt-hosted-engine/hosted-engine.conf
fqdn=vmadmin.fnal.gov
vm_disk_id=36a09f1d-0923-4b3b-87aa-4788ca64064e
vm_disk_vol_id=c4244bdd-80c5-4f68-83c2-9494d9d05723
vmid=823d3e5b-60c2-4e53-a9e8-313aedcaf808
storage=None
conf=/var/run/ovirt-hosted-engine-ha/vm.conf
host_id=4
console=qxl
domainType=fc
spUUID=00000000-0000-0000-0000-000000000000
sdUUID=81f19871-4d91-4698-a97d-36452bfae281
connectionUUID=73b61b0a-85f8-4fb7-8faf-c687ef7cc5d8
ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem
ca_subject="C=EN, L=Test, O=Test, CN=Test"
vdsm_use_ssl=true
gateway=131.225.193.200
bridge=ovirtmgmt
metadata_volume_UUID=93cf12c5-d5e6-4ea6-bee7-de64ee52d7a5
metadata_image_UUID=22a9849b-c551-4783-ad0c-530464df47f3
lockspace_volume_UUID=0593a3b8-1d75-4be3-b65b-ce9a164d0309
lockspace_image_UUID=2c8c56f2-1711-4867-8ed0-3c502bb635ff
conf_volume_UUID=07c72aa5-7fd0-4159-b83e-4d078ae9c351
conf_image_UUID=f9e59ec5-6903-4b2d-8164-4fce3d901bdd

# The following are used only for iSCSI storage
iqn=
portal=
user=
password=
port=

-- 
Pat Riehecky

Fermi National Accelerator Laboratory
www.fnal.gov
www.scientificlinux.org

-------------- next part --------------
MainThread::INFO::2016-09-01 11:36:05,537::hosted_engine::517::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2016-09-01 11:36:05,538::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '131.225.193.200'}
MainThread::INFO::2016-09-01 11:36:05,541::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 23350672
MainThread::INFO::2016-09-01 11:36:05,542::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2016-09-01 11:36:05,549::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 23350736
MainThread::INFO::2016-09-01 11:36:05,550::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2016-09-01 11:36:05,554::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 23351120
MainThread::INFO::2016-09-01 11:36:05,555::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '823d3e5b-60c2-4e53-a9e8-313aedcaf808', 'address': '0'}
MainThread::INFO::2016-09-01 11:36:05,558::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140278866605776
MainThread::INFO::2016-09-01 11:36:05,558::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '823d3e5b-60c2-4e53-a9e8-313aedcaf808', 'address': '0'}
MainThread::INFO::2016-09-01 11:36:05,563::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140278866742032
MainThread::INFO::2016-09-01 11:36:05,756::brokerlink::178::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140278732398288
MainThread::INFO::2016-09-01 11:36:05,756::hosted_engine::609::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2016-09-01 11:36:05,816::hosted_engine::708::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 4 is acquired (file: /var/run/vdsm/storage/81f19871-4d91-4698-a97d-36452bfae281/2c8c56f2-1711-4867-8ed0-3c502bb635ff/0593a3b8-1d75-4be3-b65b-ce9a164d0309)
MainThread::INFO::2016-09-01 11:36:10,961::upgrade::1001::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Host configuration is already up-to-date
MainThread::INFO::2016-09-01 11:36:10,961::hosted_engine::421::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Reloading vm.conf from the shared storage domain
MainThread::INFO::2016-09-01 11:36:10,961::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2016-09-01 11:36:16,122::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:c231f06b-0d4e-4d91-be55-2de903351fd3, volUUID:57e7e406-32f3-47ff-b879-9b608ba9cd42
MainThread::INFO::2016-09-01 11:36:16,193::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:d2696090-e25c-4195-8c06-dafd80cf0720, volUUID:20b39847-ce63-49bb-a623-15f2aed29515
MainThread::INFO::2016-09-01 11:36:16,383::ovf_store::111::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2016-09-01 11:36:16,384::ovf_store::118::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/blockSD/81f19871-4d91-4698-a97d-36452bfae281/images/d2696090-e25c-4195-8c06-dafd80cf0720/20b39847-ce63-49bb-a623-15f2aed29515 
MainThread::ERROR::2016-09-01 11:36:16,407::ovf_store::123::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Unable to extract HEVM OVF
MainThread::ERROR::2016-09-01 11:36:16,407::config::235::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf
MainThread::ERROR::2016-09-01 11:36:16,511::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: ''Configuration value not found: file=/var/run/ovirt-hosted-engine-ha/vm.conf, key=memSize'' - trying to restart agent
MainThread::WARNING::2016-09-01 11:36:21,516::agent::208::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '0'
MainThread::INFO::2016-09-01 11:36:21,542::hosted_engine::243::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: myhostname
MainThread::INFO::2016-09-01 11:36:21,543::hosted_engine::612::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2016-09-01 11:36:26,741::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2016-09-01 11:36:26,742::storage_server::218::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2016-09-01 11:36:31,871::storage_server::232::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2016-09-01 11:36:32,037::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2016-09-01 11:36:32,037::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2016-09-01 11:36:37,614::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Reloading vm.conf from the shared storage domain
MainThread::INFO::2016-09-01 11:36:37,615::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2016-09-01 11:36:42,742::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:c231f06b-0d4e-4d91-be55-2de903351fd3, volUUID:57e7e406-32f3-47ff-b879-9b608ba9cd42
MainThread::INFO::2016-09-01 11:36:42,813::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:d2696090-e25c-4195-8c06-dafd80cf0720, volUUID:20b39847-ce63-49bb-a623-15f2aed29515
MainThread::INFO::2016-09-01 11:36:43,002::ovf_store::111::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2016-09-01 11:36:43,003::ovf_store::118::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/blockSD/81f19871-4d91-4698-a97d-36452bfae281/images/d2696090-e25c-4195-8c06-dafd80cf0720/20b39847-ce63-49bb-a623-15f2aed29515 
MainThread::ERROR::2016-09-01 11:36:43,026::ovf_store::123::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Unable to extract HEVM OVF
MainThread::ERROR::2016-09-01 11:36:43,026::config::235::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf
MainThread::INFO::2016-09-01 11:36:43,057::hosted_engine::517::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2016-09-01 11:36:43,058::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '131.225.193.200'}
MainThread::INFO::2016-09-01 11:36:43,062::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140278732397840
MainThread::INFO::2016-09-01 11:36:43,062::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2016-09-01 11:36:43,067::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140278732394960
MainThread::INFO::2016-09-01 11:36:43,067::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2016-09-01 11:36:43,071::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140278732397456
MainThread::INFO::2016-09-01 11:36:43,071::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '823d3e5b-60c2-4e53-a9e8-313aedcaf808', 'address': '0'}
MainThread::INFO::2016-09-01 11:36:43,075::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 23539920
MainThread::INFO::2016-09-01 11:36:43,075::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '823d3e5b-60c2-4e53-a9e8-313aedcaf808', 'address': '0'}
MainThread::INFO::2016-09-01 11:36:43,079::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140278732548112
MainThread::INFO::2016-09-01 11:36:43,249::brokerlink::178::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140278732417744
MainThread::INFO::2016-09-01 11:36:43,249::hosted_engine::609::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2016-09-01 11:36:43,307::hosted_engine::708::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 4 is acquired (file: /var/run/vdsm/storage/81f19871-4d91-4698-a97d-36452bfae281/2c8c56f2-1711-4867-8ed0-3c502bb635ff/0593a3b8-1d75-4be3-b65b-ce9a164d0309)
MainThread::INFO::2016-09-01 11:36:48,448::upgrade::1001::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Host configuration is already up-to-date
MainThread::INFO::2016-09-01 11:36:48,448::hosted_engine::421::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Reloading vm.conf from the shared storage domain
MainThread::INFO::2016-09-01 11:36:48,448::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2016-09-01 11:36:53,630::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:c231f06b-0d4e-4d91-be55-2de903351fd3, volUUID:57e7e406-32f3-47ff-b879-9b608ba9cd42
MainThread::INFO::2016-09-01 11:36:53,784::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:d2696090-e25c-4195-8c06-dafd80cf0720, volUUID:20b39847-ce63-49bb-a623-15f2aed29515
MainThread::INFO::2016-09-01 11:36:53,978::ovf_store::111::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2016-09-01 11:36:53,979::ovf_store::118::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/blockSD/81f19871-4d91-4698-a97d-36452bfae281/images/d2696090-e25c-4195-8c06-dafd80cf0720/20b39847-ce63-49bb-a623-15f2aed29515 