<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I had some of this going on recently under 4.1.2, started with one or two warning messages, then a flood of them. Did the upgrade to 4.1.3 and haven’t seen it yet, but it’s only been a few days so far. A java process was consuming much CPU, and the DataWarehouse appears to not be collecting data (evidenced by a blank dashboard). My DWH has since recovered as well.<div class=""><br class=""></div><div class="">I forgot to check, but suspect I was low/out of memory on my engine VM, it’s an old one with only 6G allocated currently. Watching for this to happen again, and will confirm RAM utilization and bump up appropriately if it looks like it’s starved for RAM.</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 18, 2017, at 5:45 AM, Christophe TREFOIS <<a href="mailto:christophe.trefois@uni.lu" class="">christophe.trefois@uni.lu</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
I have the same as you on 4.1.0
<div class=""><br class="">
</div>
<div class="">EngineBadHealth-EngineUp 1 minute later. Sometimes 20 times per day, mostly on weekends.<br class="">
<div class=""><br class="">
</div>
<div class="">Cheers,<br class="">
<div class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><p style="font-family: Arial, sans-serif; font-size: 10pt; line-height: 16px; color: rgb(33, 33, 33);" class="">
<span style="color: rgb(61, 59, 59); display: inline;" class="">-- </span></p><p style="font-family: Arial, sans-serif; font-size: 10pt; line-height: 16px; color: rgb(33, 33, 33);" class="">
<span style="font-weight: bold; color: rgb(61, 59, 59); display: inline;" class="">Dr Christophe Trefois, Dipl.-Ing.</span><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><span style="display: inline;" class=""></span><br class="">
<span style="color: rgb(61, 59, 59); display: inline; font-size: 7.5pt;" class="">Technical Specialist / Post-Doc</span></p><p style="font-family: Arial, sans-serif; font-size: 7.5pt; line-height: 16px;" class="">
<span style="font-weight: bold; color: rgb(61, 59, 59); display: inline;" class="">UNIVERSITÉ DU LUXEMBOURG</span><br class="">
<span style="display: inline;" class=""><br class="">
</span><span style="font-weight: bold; color: rgb(61, 59, 59); display: inline;" class="">LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE</span><br class="">
<span style="color: rgb(61, 59, 59); display: inline;" class="">Campus Belval | House of Biomedicine<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><br class="">
<span class="Apple-converted-space">6, avenue du Swing </span><br class="">
L-4367 Belvaux<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></span><br class="">
<span style="color: rgb(61, 59, 59); display: inline;" class="">T:<span class="Apple-converted-space"> </span></span><span style="color: rgb(61, 59, 59); display: inline;" class="">+352 46 66 44 6124</span><span class="Apple-converted-space"> </span><br class="">
<span style="color: rgb(61, 59, 59); display: inline;" class="">F:<span class="Apple-converted-space"> </span></span><span style="color: rgb(61, 59, 59); display: inline;" class="">+352 46 66 44 6949</span><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><span style="display: inline;" class=""><br class="">
</span><a href="http://www.uni.lu/lcsb" style="color: rgb(0, 109, 189); display: inline;" class="">http://www.uni.lu/lcsb</a></p><p style="font-family: Arial, sans-serif; font-size: 14px; line-height: 16px;" class="">
<a href="https://www.facebook.com/trefex" style="display: inline;" class=""><img width="24" height="24" data-filename="facebook.png" src="https://s3.amazonaws.com/htmlsig-assets/rounded/facebook.png" alt="Facebook" class=""></a><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><a href="https://twitter.com/Trefex" style="display: inline;" class=""><img width="24" height="24" data-filename="twitter.png" src="https://s3.amazonaws.com/htmlsig-assets/rounded/twitter.png" alt="Twitter" class=""></a><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><a href="https://plus.google.com/+ChristopheTrefois/" style="display: inline;" class=""><img width="24" height="24" data-filename="googleplus.png" src="https://s3.amazonaws.com/htmlsig-assets/rounded/googleplus.png" alt="Google Plus" class=""></a><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><a href="https://www.linkedin.com/in/trefoischristophe" style="display: inline;" class=""><img width="24" height="24" data-filename="linkedin.png" src="https://s3.amazonaws.com/htmlsig-assets/rounded/linkedin.png" alt="Linkedin" class=""></a><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><a href="http://skype:Trefex?call" style="display: inline;" class=""><img width="24" height="24" data-filename="skype.png" src="https://s3.amazonaws.com/htmlsig-assets/rounded/skype.png" alt="skype" class=""></a></p><div style="font-family: Arial, sans-serif; font-size: 14px; line-height: 16px;" class="">
<br class="webkit-block-placeholder"></div><p style="font-family: Arial, sans-serif; color: rgb(61, 59, 59); font-size: 9px; line-height: 16px;" class="">
----<br class="">
This message is confidential and may contain privileged information.<span class="Apple-converted-space"> </span><br class="">
It is intended for the named recipient only.<span class="Apple-converted-space"> </span><br class="">
If you receive it in error please notify me and permanently delete the original message and any copies.<span class="Apple-converted-space"> </span><br class="">
----<br class="">
</p>
<span class="Apple-converted-space"> </span></div>
</div>
</div>
</div>
<br class="">
<div style="" class="">
<blockquote type="cite" class="">
<div class="">On 17 Jul 2017, at 17:35, Jim Kusznir <<a href="mailto:jim@palousetech.com" class="">jim@palousetech.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Ok, I've been ignoring this for a long time as the logs were so verbose and didn't show anything I could identify as usable debug info. Recently one of my ovirt hosts (currently NOT running the main engine, but a candidate) was cycling
as much as 40 times a day between "EngineUpBadHealth and EngineUp". Here's the log snippit. I included some time before and after if that's helpful. In this case, I got an email about bad health at 8:15 and a restore (engine up) at 8:16. I see where the
messages are sent, but I don't see any explanation as to why / what the problem is.
<div class=""><br class="">
</div>
<div class="">BTW: 192.168.8.11 is this computer's physical IP; 192.168.8.12 is the computer currently running the engine. Both are also hosting the gluster store (eg, I have 3 hosts, all are participating in the gluster replica 2+arbitrator).</div>
<div class=""><br class="">
</div>
<div class="">I'd appreciate it if someone could shed some light on why this keeps happening!</div>
<div class=""><br class="">
</div>
<div class="">--Jim</div>
<div class="">----------------<br class="">
<div class=""><br class="">
</div>
<div class="">
<div class="">MainThread::INFO::2017-07-17 08:12:06,230::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:12:06,230::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:12:08,877::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:12:09,432::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:12:09,925::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:12:10,324::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:12:10,696::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:12:10,704::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:12:10,705::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost</div>
<div class="">MainThread::INFO::2017-07-17 08:12:10,714::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM</div>
<div class="">MainThread::INFO::2017-07-17 08:12:14,426::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage</div>
<div class="">MainThread::INFO::2017-07-17 08:12:14,470::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:12:19,648::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:12:19,900::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:12:20,298::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:12:20,298::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:12:24,051::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf</div>
<div class="">MainThread::INFO::2017-07-17 08:12:24,051::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:12:24,052::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:12:26,895::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:12:27,429::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:12:27,976::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:12:28,240::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:12:28,564::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:12:28,572::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:12:34,574::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 3400)</div>
<div class="">MainThread::INFO::2017-07-17 08:12:34,575::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host
<a href="http://ovirt2.nwfiber.com/" class="">ovirt2.nwfiber.com</a> (id: 2, score: 3400)</div>
<div class="">MainThread::INFO::2017-07-17 08:12:44,692::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:12:44,692::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:12:47,493::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:12:48,127::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:12:48,758::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:12:49,128::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:12:49,552::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:12:49,559::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:12:49,567::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM</div>
<div class="">MainThread::INFO::2017-07-17 08:12:52,152::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage</div>
<div class="">MainThread::INFO::2017-07-17 08:12:52,192::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:12:58,227::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:12:58,406::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:12:58,805::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:12:58,806::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:13:02,662::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf</div>
<div class="">MainThread::INFO::2017-07-17 08:13:02,662::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:13:02,663::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:13:05,462::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:13:06,057::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:13:06,646::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:13:07,100::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:13:07,595::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:13:07,602::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:13:13,643::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 3400)</div>
<div class="">MainThread::INFO::2017-07-17 08:13:13,644::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host
<a href="http://ovirt2.nwfiber.com/" class="">ovirt2.nwfiber.com</a> (id: 2, score: 3400)</div>
<div class="">MainThread::INFO::2017-07-17 08:13:23,760::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:13:23,760::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:13:26,550::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:13:27,129::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:13:27,770::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:13:28,154::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:13:28,603::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:13:28,612::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:13:28,613::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost</div>
<div class="">MainThread::INFO::2017-07-17 08:13:28,622::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM</div>
<div class="">MainThread::INFO::2017-07-17 08:13:31,357::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage</div>
<div class="">MainThread::INFO::2017-07-17 08:13:31,420::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:13:37,594::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:13:37,848::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:13:38,300::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:13:38,301::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:13:41,859::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf</div>
<div class="">MainThread::INFO::2017-07-17 08:13:41,860::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:13:41,860::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:13:44,478::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:13:44,916::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:13:45,437::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:13:45,703::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:13:46,010::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:13:46,014::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:13:51,170::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 3400)</div>
<div class="">MainThread::INFO::2017-07-17 08:13:51,170::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host
<a href="http://ovirt2.nwfiber.com/" class="">ovirt2.nwfiber.com</a> (id: 2, score: 3400)</div>
<div class="">MainThread::INFO::2017-07-17 08:14:01,282::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:14:01,282::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:14:04,906::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:14:05,394::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:14:05,989::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:14:06,277::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:14:06,627::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:14:06,632::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:14:06,637::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM</div>
<div class="">MainThread::INFO::2017-07-17 08:14:09,448::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage</div>
<div class="">MainThread::INFO::2017-07-17 08:14:09,494::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:14:14,997::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:14:15,267::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:14:15,670::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:14:15,671::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:14:20,547::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf</div>
<div class="">MainThread::INFO::2017-07-17 08:14:20,548::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:14:20,548::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:14:23,443::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:14:24,001::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:14:24,604::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:14:24,988::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:14:25,391::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:14:25,398::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:14:30,596::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 3400)</div>
<div class="">MainThread::INFO::2017-07-17 08:14:30,597::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host
<a href="http://ovirt2.nwfiber.com/" class="">ovirt2.nwfiber.com</a> (id: 2, score: 3400)</div>
<div class="">MainThread::INFO::2017-07-17 08:14:40,749::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:14:40,749::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:14:43,642::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:14:44,310::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:14:44,868::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:14:45,253::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:14:45,670::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:14:45,676::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:14:45,677::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost</div>
<div class="">MainThread::INFO::2017-07-17 08:14:45,684::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM</div>
<div class="">MainThread::INFO::2017-07-17 08:14:49,208::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage</div>
<div class="">MainThread::INFO::2017-07-17 08:14:49,246::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:14:54,226::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:14:54,453::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:14:54,835::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:14:54,835::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:14:58,377::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf</div>
<div class="">MainThread::INFO::2017-07-17 08:14:58,377::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:14:58,377::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:15:01,136::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:15:01,714::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:15:02,386::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:15:02,803::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:15:03,292::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:15:03,299::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:15:09,285::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 3400)</div>
<div class="">MainThread::INFO::2017-07-17 08:15:09,286::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host
<a href="http://ovirt2.nwfiber.com/" class="">ovirt2.nwfiber.com</a> (id: 2, score: 3400)</div>
<div class="">MainThread::INFO::2017-07-17 08:15:19,412::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:15:19,413::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:15:22,266::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:15:22,913::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:15:23,506::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:15:23,881::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:15:24,303::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:15:24,309::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:15:24,318::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1500304524.32 type=state_transition detail=EngineUp-EngineUpBadHealth hostname='<a href="http://ovirt1.nwfiber.com/" class="">ovirt1.nwfiber.com</a>'</div>
<div class="">MainThread::INFO::2017-07-17 08:15:24,450::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineUp-EngineUpBadHealth) sent? sent</div>
<div class="">MainThread::INFO::2017-07-17 08:15:24,451::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM</div>
<div class="">MainThread::INFO::2017-07-17 08:15:27,085::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage</div>
<div class="">MainThread::INFO::2017-07-17 08:15:27,126::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:15:33,220::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:15:33,514::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:15:34,134::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:15:34,134::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:15:37,991::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf</div>
<div class="">MainThread::INFO::2017-07-17 08:15:37,991::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:15:37,991::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:15:40,745::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:15:41,372::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:15:42,054::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:15:42,563::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:15:43,062::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:15:43,069::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:15:49,373::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUpBadHealth (score: 3400)</div>
<div class="">MainThread::INFO::2017-07-17 08:15:49,374::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host
<a href="http://ovirt2.nwfiber.com/" class="">ovirt2.nwfiber.com</a> (id: 2, score: 3400)</div>
<div class="">MainThread::INFO::2017-07-17 08:15:59,492::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:15:59,492::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:16:02,358::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div>
<div class="">MainThread::INFO::2017-07-17 08:16:03,114::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div>
<div class="">MainThread::INFO::2017-07-17 08:16:03,801::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE</div>
<div class="">MainThread::INFO::2017-07-17 08:16:04,206::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 </div>
<div class="">MainThread::INFO::2017-07-17 08:16:04,680::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert</div>
<div class="">MainThread::INFO::2017-07-17 08:16:04,688::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE</div>
<div class="">MainThread::ERROR::2017-07-17 08:16:04,690::states::602::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health status, timeout in 300 seconds</div>
<div class="">MainThread::INFO::2017-07-17 08:16:04,691::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost</div>
<div class="">MainThread::INFO::2017-07-17 08:16:04,700::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning <class 'ovirt_hosted_engine_ha.agent.states.EngineUpBadHealth'> -> <class 'ovirt_hosted_engine_ha.agent.states.EngineUp'></div>
<div class="">MainThread::INFO::2017-07-17 08:16:04,710::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1500304564.71 type=state_transition detail=EngineUpBadHealth-EngineUp hostname='<a href="http://ovirt1.nwfiber.com/" class="">ovirt1.nwfiber.com</a>'</div>
<div class="">MainThread::INFO::2017-07-17 08:16:04,798::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineUpBadHealth-EngineUp) sent? sent</div>
<div class="">MainThread::INFO::2017-07-17 08:16:04,799::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM</div>
<div class="">MainThread::INFO::2017-07-17 08:16:07,435::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage</div>
<div class="">MainThread::INFO::2017-07-17 08:16:07,491::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:16:13,906::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server</div>
<div class="">MainThread::INFO::2017-07-17 08:16:14,131::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain</div>
<div class="">MainThread::INFO::2017-07-17 08:16:14,437::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images</div>
<div class="">MainThread::INFO::2017-07-17 08:16:14,438::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images</div>
</div>
<div class=""><br class="">
</div>
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Thu, Mar 30, 2017 at 5:58 AM, Simone Tiraboschi <span dir="ltr" class="">
<<a href="mailto:stirabos@redhat.com" target="_blank" class="">stirabos@redhat.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">Could you please check your /var/log/ovirt-hosted-engine-<wbr class="">ha/agent.log ?</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">
<div class="">
<div class="h5">On Thu, Mar 30, 2017 at 3:10 AM, Jim Kusznir <span dir="ltr" class="">
<<a href="mailto:jim@palousetech.com" target="_blank" class="">jim@palousetech.com</a>></span> wrote:<br class="">
</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="">
<div class="h5">
<div dir="ltr" class="">Hello:
<div class=""><br class="">
</div>
<div class="">I find that I often get random-seeming messages. A lot of them mention "ReintializeFSM", but I also get engine down, engine start, etc. messages. All the time, nothing appears to be happening on the cluster, and I rarely can find anything wrong
or any trigger/cause. Is this normal? What causes this (beyond obvious hardware issues / hosts rebooting)? Most of the time when I get these, my cluster is going along smoothly, and nothing (not even administrative access) is interrupted.</div>
<div class=""><br class="">
</div>
<div class="">Could ISP issues cause these messages to be generated?</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<span class="m_8860371200978816520HOEnZb"><font color="#888888" class="">
<div class="">--Jim</div>
</font></span></div>
<br class="">
</div>
</div>
______________________________<wbr class="">_________________<br class="">
Users mailing list<br class="">
<a href="mailto:Users@ovirt.org" target="_blank" class="">Users@ovirt.org</a><br class="">
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank" class="">http://lists.ovirt.org/mailman<wbr class="">/listinfo/users</a><br class="">
<br class="">
</blockquote>
</div>
<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
_______________________________________________<br class="">
Users mailing list<br class="">
<a href="mailto:Users@ovirt.org" class="">Users@ovirt.org</a><br class="">
<a href="http://lists.ovirt.org/mailman/listinfo/users" class="">http://lists.ovirt.org/mailman/listinfo/users</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
_______________________________________________<br class="">Users mailing list<br class=""><a href="mailto:Users@ovirt.org" class="">Users@ovirt.org</a><br class="">http://lists.ovirt.org/mailman/listinfo/users<br class=""></div></blockquote></div><br class=""></div></body></html>