[ovirt-users] ovirt-hosted-engine state transition messages

Wed Jul 19 14:44:47 UTC 2017

Hi,

the BadHealth status means we could not "ping" the engine. There is an
http endpoint we use to check the service is up and responding and it
is possible there was a timeout while the system was too loaded.

We actually use hosted-engine --check-liveliness that tries to talk to
http://{fqdn}/ovirt-engine/services/health

There is a 5 minute grace period and if the engine recovers (and it
usually does in this case) we move the status back to Up.

Best regards

--
Martin Sivak
oVirt

On Tue, Jul 18, 2017 at 6:22 PM, Darrell Budic <budic at onholyground.com> wrote:
> I had some of this going on recently under 4.1.2, started with one or two
> warning messages, then a flood of them. Did the upgrade to 4.1.3 and haven’t
> seen it yet, but it’s only been a few days so far. A java process was
> consuming much CPU, and the DataWarehouse appears to not be collecting data
> (evidenced by a blank dashboard). My DWH has since recovered as well.
>
> I forgot to check, but suspect I was low/out of memory on my engine VM, it’s
> an old one with only 6G allocated currently. Watching for this to happen
> again, and will confirm RAM utilization and bump up appropriately if it
> looks like it’s starved for RAM.
>
>
> On Jul 18, 2017, at 5:45 AM, Christophe TREFOIS <christophe.trefois at uni.lu>
> wrote:
>
> I have the same as you on 4.1.0
>
> EngineBadHealth-EngineUp 1 minute later. Sometimes 20 times per day, mostly
> on weekends.
>
> Cheers,
>
> --
>
> Dr Christophe Trefois, Dipl.-Ing.
> Technical Specialist / Post-Doc
>
> UNIVERSITÉ DU LUXEMBOURG
>
> LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
> Campus Belval | House of Biomedicine
> 6, avenue du Swing
> L-4367 Belvaux
> T: +352 46 66 44 6124
> F: +352 46 66 44 6949
> http://www.uni.lu/lcsb
>
>
>
>
> ----
> This message is confidential and may contain privileged information.
> It is intended for the named recipient only.
> If you receive it in error please notify me and permanently delete the
> original message and any copies.
> ----
>
>
>
> On 17 Jul 2017, at 17:35, Jim Kusznir <jim at palousetech.com> wrote:
>
> Ok, I've been ignoring this for a long time as the logs were so verbose and
> didn't show anything I could identify as usable debug info.  Recently one of
> my ovirt hosts (currently NOT running the main engine, but a candidate) was
> cycling as much as 40 times a day between "EngineUpBadHealth and EngineUp".
> Here's the log snippit.  I included some time before and after if that's
> helpful.  In this case, I got an email about bad health at 8:15 and a
> restore (engine up) at 8:16.  I see where the messages are sent, but I don't
> see any explanation as to why / what the problem is.
>
> BTW: 192.168.8.11 is this computer's physical IP; 192.168.8.12 is the
> computer currently running the engine.  Both are also hosting the gluster
> store (eg, I have 3 hosts, all are participating in the gluster replica
> 2+arbitrator).
>
> I'd appreciate it if someone could shed some light on why this keeps
> happening!
>
> --Jim
> ----------------
>
> MainThread::INFO::2017-07-17
> 08:12:06,230::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:12:06,230::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:12:08,877::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:12:09,432::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:12:09,925::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:12:10,324::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:12:10,696::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:12:10,704::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2017-07-17
> 08:12:10,705::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
> Engine vm running on localhost
> MainThread::INFO::2017-07-17
> 08:12:10,714::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
> Initializing VDSM
> MainThread::INFO::2017-07-17
> 08:12:14,426::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Connecting the storage
> MainThread::INFO::2017-07-17
> 08:12:14,470::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:12:19,648::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:12:19,900::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
> MainThread::INFO::2017-07-17
> 08:12:20,298::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:12:20,298::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:12:24,051::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Refreshing vm.conf
> MainThread::INFO::2017-07-17
> 08:12:24,051::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:12:24,052::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:12:26,895::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:12:27,429::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:12:27,976::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:12:28,240::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:12:28,564::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:12:28,572::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2017-07-17
> 08:12:34,574::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Current state EngineUp (score: 3400)
> MainThread::INFO::2017-07-17
> 08:12:34,575::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Best remote host ovirt2.nwfiber.com (id: 2, score: 3400)
> MainThread::INFO::2017-07-17
> 08:12:44,692::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:12:44,692::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:12:47,493::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:12:48,127::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:12:48,758::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:12:49,128::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:12:49,552::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:12:49,559::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2017-07-17
> 08:12:49,567::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
> Initializing VDSM
> MainThread::INFO::2017-07-17
> 08:12:52,152::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Connecting the storage
> MainThread::INFO::2017-07-17
> 08:12:52,192::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:12:58,227::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:12:58,406::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
> MainThread::INFO::2017-07-17
> 08:12:58,805::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:12:58,806::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:13:02,662::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Refreshing vm.conf
> MainThread::INFO::2017-07-17
> 08:13:02,662::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:13:02,663::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:13:05,462::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:13:06,057::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:13:06,646::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:13:07,100::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:13:07,595::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:13:07,602::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2017-07-17
> 08:13:13,643::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Current state EngineUp (score: 3400)
> MainThread::INFO::2017-07-17
> 08:13:13,644::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Best remote host ovirt2.nwfiber.com (id: 2, score: 3400)
> MainThread::INFO::2017-07-17
> 08:13:23,760::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:13:23,760::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:13:26,550::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:13:27,129::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:13:27,770::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:13:28,154::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:13:28,603::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:13:28,612::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2017-07-17
> 08:13:28,613::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
> Engine vm running on localhost
> MainThread::INFO::2017-07-17
> 08:13:28,622::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
> Initializing VDSM
> MainThread::INFO::2017-07-17
> 08:13:31,357::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Connecting the storage
> MainThread::INFO::2017-07-17
> 08:13:31,420::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:13:37,594::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:13:37,848::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
> MainThread::INFO::2017-07-17
> 08:13:38,300::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:13:38,301::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:13:41,859::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Refreshing vm.conf
> MainThread::INFO::2017-07-17
> 08:13:41,860::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:13:41,860::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:13:44,478::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:13:44,916::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:13:45,437::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:13:45,703::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:13:46,010::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:13:46,014::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2017-07-17
> 08:13:51,170::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Current state EngineUp (score: 3400)
> MainThread::INFO::2017-07-17
> 08:13:51,170::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Best remote host ovirt2.nwfiber.com (id: 2, score: 3400)
> MainThread::INFO::2017-07-17
> 08:14:01,282::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:14:01,282::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:14:04,906::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:14:05,394::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:14:05,989::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:14:06,277::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:14:06,627::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:14:06,632::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2017-07-17
> 08:14:06,637::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
> Initializing VDSM
> MainThread::INFO::2017-07-17
> 08:14:09,448::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Connecting the storage
> MainThread::INFO::2017-07-17
> 08:14:09,494::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:14:14,997::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:14:15,267::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
> MainThread::INFO::2017-07-17
> 08:14:15,670::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:14:15,671::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:14:20,547::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Refreshing vm.conf
> MainThread::INFO::2017-07-17
> 08:14:20,548::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:14:20,548::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:14:23,443::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:14:24,001::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:14:24,604::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:14:24,988::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:14:25,391::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:14:25,398::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2017-07-17
> 08:14:30,596::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Current state EngineUp (score: 3400)
> MainThread::INFO::2017-07-17
> 08:14:30,597::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Best remote host ovirt2.nwfiber.com (id: 2, score: 3400)
> MainThread::INFO::2017-07-17
> 08:14:40,749::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:14:40,749::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:14:43,642::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:14:44,310::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:14:44,868::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:14:45,253::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:14:45,670::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:14:45,676::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2017-07-17
> 08:14:45,677::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
> Engine vm running on localhost
> MainThread::INFO::2017-07-17
> 08:14:45,684::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
> Initializing VDSM
> MainThread::INFO::2017-07-17
> 08:14:49,208::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Connecting the storage
> MainThread::INFO::2017-07-17
> 08:14:49,246::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:14:54,226::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:14:54,453::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
> MainThread::INFO::2017-07-17
> 08:14:54,835::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:14:54,835::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:14:58,377::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Refreshing vm.conf
> MainThread::INFO::2017-07-17
> 08:14:58,377::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:14:58,377::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:15:01,136::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:15:01,714::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:15:02,386::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:15:02,803::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:15:03,292::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:15:03,299::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2017-07-17
> 08:15:09,285::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Current state EngineUp (score: 3400)
> MainThread::INFO::2017-07-17
> 08:15:09,286::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Best remote host ovirt2.nwfiber.com (id: 2, score: 3400)
> MainThread::INFO::2017-07-17
> 08:15:19,412::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:15:19,413::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:15:22,266::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:15:22,913::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:15:23,506::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:15:23,881::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:15:24,303::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:15:24,309::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2017-07-17
> 08:15:24,318::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
> Trying: notify time=1500304524.32 type=state_transition
> detail=EngineUp-EngineUpBadHealth hostname='ovirt1.nwfiber.com'
> MainThread::INFO::2017-07-17
> 08:15:24,450::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
> Success, was notification of state_transition (EngineUp-EngineUpBadHealth)
> sent? sent
> MainThread::INFO::2017-07-17
> 08:15:24,451::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
> Initializing VDSM
> MainThread::INFO::2017-07-17
> 08:15:27,085::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Connecting the storage
> MainThread::INFO::2017-07-17
> 08:15:27,126::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:15:33,220::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:15:33,514::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
> MainThread::INFO::2017-07-17
> 08:15:34,134::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:15:34,134::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:15:37,991::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Refreshing vm.conf
> MainThread::INFO::2017-07-17
> 08:15:37,991::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:15:37,991::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:15:40,745::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:15:41,372::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:15:42,054::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:15:42,563::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:15:43,062::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:15:43,069::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2017-07-17
> 08:15:49,373::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Current state EngineUpBadHealth (score: 3400)
> MainThread::INFO::2017-07-17
> 08:15:49,374::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Best remote host ovirt2.nwfiber.com (id: 2, score: 3400)
> MainThread::INFO::2017-07-17
> 08:15:59,492::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2017-07-17
> 08:15:59,492::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:16:02,358::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57,
> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac
> MainThread::INFO::2017-07-17
> 08:16:03,114::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4,
> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:16:03,801::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-07-17
> 08:16:04,206::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929
> MainThread::INFO::2017-07-17
> 08:16:04,680::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-07-17
> 08:16:04,688::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
> Got vm.conf from OVF_STORE
> MainThread::ERROR::2017-07-17
> 08:16:04,690::states::602::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
> Engine VM has bad health status, timeout in 300 seconds
> MainThread::INFO::2017-07-17
> 08:16:04,691::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
> Engine vm running on localhost
> MainThread::INFO::2017-07-17
> 08:16:04,700::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
> Timeout cleared while transitioning <class
> 'ovirt_hosted_engine_ha.agent.states.EngineUpBadHealth'> -> <class
> 'ovirt_hosted_engine_ha.agent.states.EngineUp'>
> MainThread::INFO::2017-07-17
> 08:16:04,710::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
> Trying: notify time=1500304564.71 type=state_transition
> detail=EngineUpBadHealth-EngineUp hostname='ovirt1.nwfiber.com'
> MainThread::INFO::2017-07-17
> 08:16:04,798::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
> Success, was notification of state_transition (EngineUpBadHealth-EngineUp)
> sent? sent
> MainThread::INFO::2017-07-17
> 08:16:04,799::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
> Initializing VDSM
> MainThread::INFO::2017-07-17
> 08:16:07,435::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Connecting the storage
> MainThread::INFO::2017-07-17
> 08:16:07,491::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:16:13,906::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2017-07-17
> 08:16:14,131::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
> MainThread::INFO::2017-07-17
> 08:16:14,437::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Preparing images
> MainThread::INFO::2017-07-17
> 08:16:14,438::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
> Preparing images
>
>
> On Thu, Mar 30, 2017 at 5:58 AM, Simone Tiraboschi <stirabos at redhat.com>
> wrote:
>>
>> Could you please check your /var/log/ovirt-hosted-engine-ha/agent.log ?
>>
>> On Thu, Mar 30, 2017 at 3:10 AM, Jim Kusznir <jim at palousetech.com> wrote:
>>>
>>> Hello:
>>>
>>> I find that I often get random-seeming messages.  A lot of them mention
>>> "ReintializeFSM", but I also get engine down, engine start, etc.  messages.
>>> All the time, nothing appears to be happening on the cluster, and I rarely
>>> can find anything wrong or any trigger/cause.  Is this normal?  What causes
>>> this (beyond obvious hardware issues / hosts rebooting)?  Most of the time
>>> when I get these, my cluster is going along smoothly, and nothing (not even
>>> administrative access) is interrupted.
>>>
>>> Could ISP issues cause these messages to be generated?
>>>
>>> Thanks!
>>> --Jim
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>