Re: [ovirt-users] ovirt-hosted-engine state transition messages

--Apple-Mail=_8C444FA1-8569-42DE-B126-04B407031384 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 This happened to me again, started last night so it was almost a week = from the last restart. System was not out of memory, a bit low, and it = may have been churning buffers or java GC, I=E2=80=99m on vacation and = didn=E2=80=99t dig into it very far. Restarted the engine and it=E2=80=99s= happy. DWH was still working, but web interface was a bit slow before = the restart. This was 4.1.3 now. Added some ram to the Hosted Engine, = but looks like I need to restart it and will probably wait until I=E2=80=99= m back for that. wrote: two warning messages, then a flood of them. Did the upgrade to 4.1.3 and = haven=E2=80=99t seen it yet, but it=E2=80=99s only been a few days so = far. A java process was consuming much CPU, and the DataWarehouse = appears to not be collecting data (evidenced by a blank dashboard). My = DWH has since recovered as well. VM, it=E2=80=99s an old one with only 6G allocated currently. Watching = for this to happen again, and will confirm RAM utilization and bump up = appropriately if it looks like it=E2=80=99s starved for RAM. <christophe.trefois(a)uni.lu <mailto:christophe.trefois@uni.lu>> wrote: mostly on weekends. <https://plus.google.com/+ChristopheTrefois/> = <https://www.linkedin.com/in/trefoischristophe> = <http://skype:Trefex?call> the original message and any copies.=20 <mailto:jim@palousetech.com>> wrote: verbose and didn't show anything I could identify as usable debug info. = Recently one of my ovirt hosts (currently NOT running the main engine, = but a candidate) was cycling as much as 40 times a day between = "EngineUpBadHealth and EngineUp". Here's the log snippit. I included = some time before and after if that's helpful. In this case, I got an = email about bad health at 8:15 and a restore (engine up) at 8:16. I see = where the messages are sent, but I don't see any explanation as to why / = what the problem is. the computer currently running the engine. Both are also hosting the = gluster store (eg, I have 3 hosts, all are participating in the gluster = replica 2+arbitrator). happening! 08:12:06,230::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:12:06,230::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:12:08,877::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:12:09,432::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:12:09,925::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:12:10,324::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:12:10,696::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:12:10,704::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:12:10,705::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine::(consume) Engine vm running on localhost 08:12:10,714::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM 08:12:14,426::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the storage 08:12:14,470::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:12:19,648::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:12:19,900::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage domain 08:12:20,298::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images 08:12:20,298::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images 08:12:24,051::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Refreshing vm.conf 08:12:24,051::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:12:24,052::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:12:26,895::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:12:27,429::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:12:27,976::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:12:28,240::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:12:28,564::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:12:28,572::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:12:34,574::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Current state EngineUp (score: 3400) 08:12:34,575::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Best remote host ovirt2.nwfiber.com = <http://ovirt2.nwfiber.com/> (id: 2, score: 3400) 08:12:44,692::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:12:44,692::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:12:47,493::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:12:48,127::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:12:48,758::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:12:49,128::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:12:49,552::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:12:49,559::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:12:49,567::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM 08:12:52,152::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the storage 08:12:52,192::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:12:58,227::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:12:58,406::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage domain 08:12:58,805::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images 08:12:58,806::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images 08:13:02,662::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Refreshing vm.conf 08:13:02,662::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:13:02,663::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:13:05,462::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:13:06,057::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:13:06,646::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:13:07,100::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:13:07,595::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:13:07,602::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:13:13,643::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Current state EngineUp (score: 3400) 08:13:13,644::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Best remote host ovirt2.nwfiber.com = <http://ovirt2.nwfiber.com/> (id: 2, score: 3400) 08:13:23,760::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:13:23,760::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:13:26,550::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:13:27,129::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:13:27,770::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:13:28,154::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:13:28,603::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:13:28,612::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:13:28,613::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine::(consume) Engine vm running on localhost 08:13:28,622::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM 08:13:31,357::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the storage 08:13:31,420::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:13:37,594::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:13:37,848::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage domain 08:13:38,300::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images 08:13:38,301::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images 08:13:41,859::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Refreshing vm.conf 08:13:41,860::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:13:41,860::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:13:44,478::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:13:44,916::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:13:45,437::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:13:45,703::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:13:46,010::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:13:46,014::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:13:51,170::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Current state EngineUp (score: 3400) 08:13:51,170::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Best remote host ovirt2.nwfiber.com = <http://ovirt2.nwfiber.com/> (id: 2, score: 3400) 08:14:01,282::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:14:01,282::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:14:04,906::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:14:05,394::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:14:05,989::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:14:06,277::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:14:06,627::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:14:06,632::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:14:06,637::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM 08:14:09,448::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the storage 08:14:09,494::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:14:14,997::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:14:15,267::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage domain 08:14:15,670::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images 08:14:15,671::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images 08:14:20,547::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Refreshing vm.conf 08:14:20,548::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:14:20,548::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:14:23,443::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:14:24,001::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:14:24,604::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:14:24,988::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:14:25,391::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:14:25,398::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:14:30,596::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Current state EngineUp (score: 3400) 08:14:30,597::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Best remote host ovirt2.nwfiber.com = <http://ovirt2.nwfiber.com/> (id: 2, score: 3400) 08:14:40,749::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:14:40,749::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:14:43,642::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:14:44,310::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:14:44,868::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:14:45,253::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:14:45,670::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:14:45,676::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:14:45,677::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine::(consume) Engine vm running on localhost 08:14:45,684::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM 08:14:49,208::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the storage 08:14:49,246::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:14:54,226::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:14:54,453::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage domain 08:14:54,835::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images 08:14:54,835::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images 08:14:58,377::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Refreshing vm.conf 08:14:58,377::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:14:58,377::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:15:01,136::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:15:01,714::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:15:02,386::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:15:02,803::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:15:03,292::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:15:03,299::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:15:09,285::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Current state EngineUp (score: 3400) 08:15:09,286::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Best remote host ovirt2.nwfiber.com = <http://ovirt2.nwfiber.com/> (id: 2, score: 3400) 08:15:19,412::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:15:19,413::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:15:22,266::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:15:22,913::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:15:23,506::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:15:23,881::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:15:24,303::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:15:24,309::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:15:24,318::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.Broke= rLink::(notify) Trying: notify time=3D1500304524.32 = type=3Dstate_transition detail=3DEngineUp-EngineUpBadHealth = hostname=3D'ovirt1.nwfiber.com <http://ovirt1.nwfiber.com/>' 08:15:24,450::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.Broke= rLink::(notify) Success, was notification of state_transition = (EngineUp-EngineUpBadHealth) sent? sent 08:15:24,451::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM 08:15:27,085::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the storage 08:15:27,126::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:15:33,220::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:15:33,514::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage domain 08:15:34,134::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images 08:15:34,134::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images 08:15:37,991::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Refreshing vm.conf 08:15:37,991::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:15:37,991::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:15:40,745::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:15:41,372::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:15:42,054::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:15:42,563::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:15:43,062::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:15:43,069::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:15:49,373::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Current state EngineUpBadHealth = (score: 3400) 08:15:49,374::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Best remote host ovirt2.nwfiber.com = <http://ovirt2.nwfiber.com/> (id: 2, score: 3400) 08:15:59,492::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain 08:15:59,492::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE 08:16:02,358::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac 08:16:03,114::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 08:16:03,801::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE 08:16:04,206::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929=20 08:16:04,680::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert 08:16:04,688::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE 08:16:04,690::states::602::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine::(consume) Engine VM has bad health status, timeout in 300 = seconds 08:16:04,691::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine::(consume) Engine vm running on localhost 08:16:04,700::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_en= gine.HostedEngine::(check) Timeout cleared while transitioning <class = 'ovirt_hosted_engine_ha.agent.states.EngineUpBadHealth'> -> <class = 'ovirt_hosted_engine_ha.agent.states.EngineUp'> 08:16:04,710::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.Broke= rLink::(notify) Trying: notify time=3D1500304564.71 = type=3Dstate_transition detail=3DEngineUpBadHealth-EngineUp = hostname=3D'ovirt1.nwfiber.com <http://ovirt1.nwfiber.com/>' 08:16:04,798::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.Broke= rLink::(notify) Success, was notification of state_transition = (EngineUpBadHealth-EngineUp) sent? sent 08:16:04,799::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM 08:16:07,435::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the storage 08:16:07,491::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:16:13,906::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage server 08:16:14,131::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage domain 08:16:14,437::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images 08:16:14,438::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images <stirabos(a)redhat.com <mailto:stirabos@redhat.com>> wrote: /var/log/ovirt-hosted-engine-ha/agent.log ? <mailto:jim@palousetech.com>> wrote: mention "ReintializeFSM", but I also get engine down, engine start, etc. = messages. All the time, nothing appears to be happening on the = cluster, and I rarely can find anything wrong or any trigger/cause. Is = this normal? What causes this (beyond obvious hardware issues / hosts = rebooting)? Most of the time when I get these, my cluster is going = along smoothly, and nothing (not even administrative access) is = interrupted. <http://lists.ovirt.org/mailman/listinfo/users> <http://lists.ovirt.org/mailman/listinfo/users> --Apple-Mail=_8C444FA1-8569-42DE-B126-04B407031384 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">This happened to me again, started last night so it was = almost a week from the last restart. System was not out of memory, a bit = low, and it may have been churning buffers or java GC, I=E2=80=99m on = vacation and didn=E2=80=99t dig into it very far. Restarted the engine = and it=E2=80=99s happy. DWH was still working, but web interface was a = bit slow before the restart. This was 4.1.3 now. Added some ram to the = Hosted Engine, but looks like I need to restart it and will probably = wait until I=E2=80=99m back for that.<div class=3D""><br = class=3D""></div><div class=3D""><br class=3D""><div><blockquote = type=3D"cite" class=3D""><div class=3D"">On Jul 18, 2017, at 9:22 AM, = Darrell Budic &lt;<a href=3D"mailto:budic@onholyground.com" = class=3D&quot;&quot;&gt;budic(a)onholyground.com&lt;/a&gt;&amp;gt; wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><meta = http-equiv=3D"Content-Type" content=3D"text/html charset=3Dutf-8" = class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: = space; -webkit-line-break: after-white-space;" class=3D"">I had some of = this going on recently under 4.1.2, started with one or two warning = messages, then a flood of them. Did the upgrade to 4.1.3 and haven=E2=80=99= t seen it yet, but it=E2=80=99s only been a few days so far. A java = process was consuming much CPU, and the DataWarehouse appears to not be = collecting data (evidenced by a blank dashboard). My DWH has since = recovered as well.<div class=3D""><br class=3D""></div><div class=3D"">I = forgot to check, but suspect I was low/out of memory on my engine VM, = it=E2=80=99s an old one with only 6G allocated currently. Watching for = this to happen again, and will confirm RAM utilization and bump up = appropriately if it looks like it=E2=80=99s starved for RAM.</div><div = class=3D""><br class=3D""></div><div class=3D""><br class=3D""><div = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On Jul = 18, 2017, at 5:45 AM, Christophe TREFOIS &lt;<a = href=3D"mailto:christophe.trefois@uni.lu" = class=3D&quot;&quot;&gt;christophe.trefois(a)uni.lu&lt;/a&gt;&amp;gt; wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8" = class=3D""> <div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""> I have the same as you on 4.1.0 <div class=3D""><br class=3D""> </div> <div class=3D"">EngineBadHealth-EngineUp 1 minute later. Sometimes 20 = times per day, mostly on weekends.<br class=3D""> <div class=3D""><br class=3D""> </div> <div class=3D"">Cheers,<br class=3D""> <div class=3D""> <div style=3D"letter-spacing: normal; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""> <div style=3D"letter-spacing: normal; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""> <div style=3D"letter-spacing: normal; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><p style=3D"font-family: Arial, sans-serif; font-size: 10pt; = line-height: 16px; color: rgb(33, 33, 33);" class=3D""> <span style=3D"color: rgb(61, 59, 59); display: inline;" = class=3D"">--&nbsp;</span></p><p style=3D"font-family: Arial, = sans-serif; font-size: 10pt; line-height: 16px; color: rgb(33, 33, 33);" = class=3D""> <span style=3D"font-weight: bold; color: rgb(61, 59, 59); display: = inline;" class=3D"">Dr Christophe Trefois, Dipl.-Ing.</span><span = class=3D"Apple-converted-space">&nbsp;</span><span = class=3D"Apple-converted-space">&nbsp;</span><span style=3D"display: = inline;" class=3D""></span><br class=3D""> <span style=3D"color: rgb(61, 59, 59); display: inline; font-size: = 7.5pt;" class=3D"">Technical Specialist / Post-Doc</span></p><p = style=3D"font-family: Arial, sans-serif; font-size: 7.5pt; line-height: = 16px;" class=3D""> <span style=3D"font-weight: bold; color: rgb(61, 59, 59); display: = inline;" class=3D"">UNIVERSIT=C3=89 DU LUXEMBOURG</span><br class=3D""> <span style=3D"display: inline;" class=3D""><br class=3D""> </span><span style=3D"font-weight: bold; color: rgb(61, 59, 59); = display: inline;" class=3D"">LUXEMBOURG CENTRE FOR SYSTEMS = BIOMEDICINE</span><br class=3D""> <span style=3D"color: rgb(61, 59, 59); display: inline;" class=3D"">Campus= Belval | House of Biomedicine<span = class=3D"Apple-converted-space">&nbsp;</span><span = class=3D"Apple-converted-space">&nbsp;</span><br class=3D""> <span class=3D"Apple-converted-space">6, avenue du Swing&nbsp;</span><br = class=3D""> L-4367 Belvaux<span class=3D"Apple-converted-space">&nbsp;</span><span = class=3D"Apple-converted-space">&nbsp;</span></span><br class=3D""> <span style=3D"color: rgb(61, 59, 59); display: inline;" = class=3D"">T:<span = class=3D"Apple-converted-space">&nbsp;</span></span><span style=3D"color: = rgb(61, 59, 59); display: inline;" class=3D"">+352 46 66 44 = 6124</span><span class=3D"Apple-converted-space">&nbsp;</span><br = class=3D""> <span style=3D"color: rgb(61, 59, 59); display: inline;" = class=3D"">F:<span = class=3D"Apple-converted-space">&nbsp;</span></span><span style=3D"color: = rgb(61, 59, 59); display: inline;" class=3D"">+352 46 66 44 = 6949</span><span class=3D"Apple-converted-space">&nbsp;</span><span = class=3D"Apple-converted-space">&nbsp;</span><span style=3D"display: = inline;" class=3D""><br class=3D""> </span><a href=3D"http://www.uni.lu/lcsb" style=3D"color: rgb(0, 109, = 189); display: inline;" class=3D"">http://www.uni.lu/lcsb</a></p><p = style=3D"font-family: Arial, sans-serif; font-size: 14px; line-height: = 16px;" class=3D""> <a href=3D"https://www.facebook.com/trefex" style=3D"display: inline;" = class=3D""><img width=3D"24" height=3D"24" data-filename=3D"facebook.png" = src=3D"https://s3.amazonaws.com/htmlsig-assets/rounded/facebook.png&... = alt=3D"Facebook" class=3D""></a><span = class=3D"Apple-converted-space">&nbsp;</span><span = class=3D"Apple-converted-space">&nbsp;</span><a = href=3D"https://twitter.com/Trefex" style=3D"display: inline;" = class=3D""><img width=3D"24" height=3D"24" data-filename=3D"twitter.png" = src=3D"https://s3.amazonaws.com/htmlsig-assets/rounded/twitter.png&q... = alt=3D"Twitter" class=3D""></a><span = class=3D"Apple-converted-space">&nbsp;</span><span = class=3D"Apple-converted-space">&nbsp;</span><a = href=3D"https://plus.google.com/+ChristopheTrefois/" style=3D"display: = inline;" class=3D""><img width=3D"24" height=3D"24" = data-filename=3D"googleplus.png" = src=3D"https://s3.amazonaws.com/htmlsig-assets/rounded/googleplus.pn... = alt=3D"Google Plus" class=3D""></a><span = class=3D"Apple-converted-space">&nbsp;</span><span = class=3D"Apple-converted-space">&nbsp;</span><a = href=3D"https://www.linkedin.com/in/trefoischristophe" style=3D"display: = inline;" class=3D""><img width=3D"24" height=3D"24" = data-filename=3D"linkedin.png" = src=3D"https://s3.amazonaws.com/htmlsig-assets/rounded/linkedin.png&... = alt=3D"Linkedin" class=3D""></a><span = class=3D"Apple-converted-space">&nbsp;</span><span = class=3D"Apple-converted-space">&nbsp;</span><a = href=3D"http://skype:Trefex?call" style=3D"display: inline;" = class=3D""><img width=3D"24" height=3D"24" data-filename=3D"skype.png" = src=3D"https://s3.amazonaws.com/htmlsig-assets/rounded/skype.png" = alt=3D"skype" class=3D""></a></p><div style=3D"font-family: Arial, = sans-serif; font-size: 14px; line-height: 16px;" class=3D""> <br class=3D"webkit-block-placeholder"></div><p style=3D"font-family: = Arial, sans-serif; color: rgb(61, 59, 59); font-size: 9px; line-height: = 16px;" class=3D""> ----<br class=3D""> This message is confidential and may contain privileged = information.<span class=3D"Apple-converted-space">&nbsp;</span><br = class=3D""> It is intended for the named recipient only.<span = class=3D"Apple-converted-space">&nbsp;</span><br class=3D""> If you receive it in error please notify me and permanently delete the = original message and any copies.<span = class=3D"Apple-converted-space">&nbsp;</span><br class=3D""> ----<br class=3D""> </p> &nbsp;<span class=3D"Apple-converted-space">&nbsp;</span></div> </div> </div> </div> <br class=3D""> <div style=3D"" class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"">On 17 Jul 2017, at 17:35, Jim Kusznir &lt;<a = href=3D"mailto:jim@palousetech.com" class=3D&quot;&quot;&gt;jim(a)palousetech.com&lt;/a&gt;&amp;gt;= wrote:</div> <br class=3D"Apple-interchange-newline"> <div class=3D""> <div dir=3D"ltr" class=3D"">Ok, I've been ignoring this for a long time = as the logs were so verbose and didn't show anything I could identify as = usable debug info.&nbsp; Recently one of my ovirt hosts (currently NOT = running the main engine, but a candidate) was cycling as much as 40 times a day between "EngineUpBadHealth and = EngineUp".&nbsp; Here's the log snippit.&nbsp; I included some time = before and after if that's helpful.&nbsp; In this case, I got an email = about bad health at 8:15 and a restore (engine up) at 8:16.&nbsp; I see = where the messages are sent, but I don't see any explanation as to why / what the = problem is. <div class=3D""><br class=3D""> </div> <div class=3D"">BTW: 192.168.8.11 is this computer's physical IP; = 192.168.8.12 is the computer currently running the engine.&nbsp; Both = are also hosting the gluster store (eg, I have 3 hosts, all are = participating in the gluster replica 2+arbitrator).</div> <div class=3D""><br class=3D""> </div> <div class=3D"">I'd appreciate it if someone could shed some light on = why this keeps happening!</div> <div class=3D""><br class=3D""> </div> <div class=3D"">--Jim</div> <div class=3D"">----------------<br class=3D""> <div class=3D""><br class=3D""> </div> <div class=3D""> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:06,230::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:06,230::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:08,877::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:09,432::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:09,925::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:10,324::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:10,696::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:10,704::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:10,705::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine::(consume) Engine vm running on localhost</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:10,714::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:14,426::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the = storage</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:14,470::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:19,648::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:19,900::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage = domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:20,298::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:20,298::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:24,051::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Refreshing vm.conf</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:24,051::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:24,052::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:26,895::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:27,429::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:27,976::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:28,240::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:28,564::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:28,572::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:34,574::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Current state EngineUp (score: = 3400)</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:34,575::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Best remote host <a href=3D"http://ovirt2.nwfiber.com/" class=3D"">ovirt2.nwfiber.com</a> = (id: 2, score: 3400)</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:44,692::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:44,692::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:47,493::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:48,127::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:48,758::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:49,128::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:49,552::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:49,559::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:49,567::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:52,152::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the = storage</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:52,192::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:58,227::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:58,406::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage = domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:58,805::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:12:58,806::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:02,662::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Refreshing vm.conf</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:02,662::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:02,663::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:05,462::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:06,057::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:06,646::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:07,100::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:07,595::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:07,602::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:13,643::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Current state EngineUp (score: = 3400)</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:13,644::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Best remote host <a href=3D"http://ovirt2.nwfiber.com/" class=3D"">ovirt2.nwfiber.com</a> = (id: 2, score: 3400)</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:23,760::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:23,760::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:26,550::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:27,129::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:27,770::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:28,154::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:28,603::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:28,612::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:28,613::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine::(consume) Engine vm running on localhost</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:28,622::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:31,357::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the = storage</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:31,420::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:37,594::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:37,848::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage = domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:38,300::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:38,301::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:41,859::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Refreshing vm.conf</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:41,860::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:41,860::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:44,478::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:44,916::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:45,437::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:45,703::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:46,010::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:46,014::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:51,170::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Current state EngineUp (score: = 3400)</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:13:51,170::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Best remote host <a href=3D"http://ovirt2.nwfiber.com/" class=3D"">ovirt2.nwfiber.com</a> = (id: 2, score: 3400)</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:01,282::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:01,282::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:04,906::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:05,394::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:05,989::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:06,277::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:06,627::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:06,632::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:06,637::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:09,448::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the = storage</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:09,494::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:14,997::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:15,267::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage = domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:15,670::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:15,671::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:20,547::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Refreshing vm.conf</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:20,548::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:20,548::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:23,443::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:24,001::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:24,604::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:24,988::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:25,391::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:25,398::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:30,596::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Current state EngineUp (score: = 3400)</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:30,597::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Best remote host <a href=3D"http://ovirt2.nwfiber.com/" class=3D"">ovirt2.nwfiber.com</a> = (id: 2, score: 3400)</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:40,749::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:40,749::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:43,642::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:44,310::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:44,868::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:45,253::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:45,670::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:45,676::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:45,677::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine::(consume) Engine vm running on localhost</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:45,684::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:49,208::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the = storage</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:49,246::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:54,226::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:54,453::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage = domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:54,835::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:54,835::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:58,377::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Refreshing vm.conf</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:58,377::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:14:58,377::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:01,136::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:01,714::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:02,386::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:02,803::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:03,292::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:03,299::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:09,285::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Current state EngineUp (score: = 3400)</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:09,286::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Best remote host <a href=3D"http://ovirt2.nwfiber.com/" class=3D"">ovirt2.nwfiber.com</a> = (id: 2, score: 3400)</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:19,412::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:19,413::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:22,266::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:22,913::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:23,506::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:23,881::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:24,303::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:24,309::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:24,318::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.Broke= rLink::(notify) Trying: notify time=3D1500304524.32 = type=3Dstate_transition detail=3DEngineUp-EngineUpBadHealth hostname=3D'<a= href=3D"http://ovirt1.nwfiber.com/" = class=3D"">ovirt1.nwfiber.com</a>'</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:24,450::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.Broke= rLink::(notify) Success, was notification of state_transition = (EngineUp-EngineUpBadHealth) sent? sent</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:24,451::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:27,085::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the = storage</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:27,126::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:33,220::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:33,514::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage = domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:34,134::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:34,134::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:37,991::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Refreshing vm.conf</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:37,991::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:37,991::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:40,745::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:41,372::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:42,054::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:42,563::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:43,062::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:43,069::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:49,373::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Current state EngineUpBadHealth = (score: 3400)</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:49,374::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring) Best remote host <a href=3D"http://ovirt2.nwfiber.com/" class=3D"">ovirt2.nwfiber.com</a> = (id: 2, score: 3400)</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:59,492::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared = storage domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:15:59,492::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a = fresher copy of vm configuration from the OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:02,358::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, = volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:03,114::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(scan) Found OVF_STORE: = imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, = volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:03,801::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) Extracting Engine VM OVF from the = OVF_STORE</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:04,206::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVF= Store::(getEngineVMOVF) OVF_STORE volume path: = /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d= 76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-= 412a-a1e8-49e73e6ba929&nbsp;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:04,680::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for = HE VM, trying to convert</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:04,688::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from = OVF_STORE</div> <div class=3D"">MainThread::ERROR::2017-07-17 = 08:16:04,690::states::602::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine::(consume) Engine VM has bad health status, timeout in 300 = seconds</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:04,691::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine::(consume) Engine vm running on localhost</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:04,700::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_en= gine.HostedEngine::(check) Timeout cleared while transitioning &lt;class = 'ovirt_hosted_engine_ha.agent.states.EngineUpBadHealth'&gt; -&gt; = &lt;class 'ovirt_hosted_engine_ha.agent.states.EngineUp'&gt;</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:04,710::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.Broke= rLink::(notify) Trying: notify time=3D1500304564.71 = type=3Dstate_transition detail=3DEngineUpBadHealth-EngineUp hostname=3D'<a= href=3D"http://ovirt1.nwfiber.com/" = class=3D"">ovirt1.nwfiber.com</a>'</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:04,798::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.Broke= rLink::(notify) Success, was notification of state_transition = (EngineUpBadHealth-EngineUp) sent? sent</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:04,799::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_vdsm) Initializing VDSM</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:07,435::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Connecting the = storage</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:07,491::storage_server::219::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:13,906::storage_server::226::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Connecting storage = server</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:14,131::storage_server::233::ovirt_hosted_engine_ha.lib.storage_serv= er.StorageServer::(connect_storage_server) Refreshing the storage = domain</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:14,437::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(_initialize_storage_images) Preparing images</div> <div class=3D"">MainThread::INFO::2017-07-17 = 08:16:14,438::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare= _images) Preparing images</div> </div> <div class=3D""><br class=3D""> </div> </div> </div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote">On Thu, Mar 30, 2017 at 5:58 AM, Simone = Tiraboschi <span dir=3D"ltr" class=3D""> &lt;<a href=3D"mailto:stirabos@redhat.com" target=3D"_blank" = class=3D&quot;&quot;&gt;stirabos(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt; wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr" class=3D"">Could you please check your = /var/log/ovirt-hosted-engine-<wbr class=3D"">ha/agent.log ?</div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote"> <div class=3D""> <div class=3D"h5">On Thu, Mar 30, 2017 at 3:10 AM, Jim Kusznir <span = dir=3D"ltr" class=3D""> &lt;<a href=3D"mailto:jim@palousetech.com" target=3D"_blank" = class=3D&quot;&quot;&gt;jim(a)palousetech.com&lt;/a&gt;&amp;gt;&lt;/span&gt; wrote:<br class=3D""> </div> </div> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div class=3D""> <div class=3D"h5"> <div dir=3D"ltr" class=3D"">Hello: <div class=3D""><br class=3D""> </div> <div class=3D"">I find that I often get random-seeming messages.&nbsp; A = lot of them mention "ReintializeFSM", but I also get engine down, engine = start, etc. &nbsp;messages.&nbsp; All the time, nothing appears to be = happening on the cluster, and I rarely can find anything wrong or any trigger/cause.&nbsp; Is this normal?&nbsp; What causes this = (beyond obvious hardware issues / hosts rebooting)?&nbsp; Most of the = time when I get these, my cluster is going along smoothly, and nothing = (not even administrative access) is interrupted.</div> <div class=3D""><br class=3D""> </div> <div class=3D"">Could ISP issues cause these messages to be = generated?</div> <div class=3D""><br class=3D""> </div> <div class=3D"">Thanks!</div> <span class=3D"m_8860371200978816520HOEnZb"><font color=3D"#888888" = class=3D""> <div class=3D"">--Jim</div> </font></span></div> <br class=3D""> </div> </div> ______________________________<wbr class=3D"">_________________<br = class=3D""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" target=3D"_blank" = class=3D&quot;&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/users</a><br class=3D""> <br class=3D""> </blockquote> </div> <br class=3D""> </div> </blockquote> </div> <br class=3D""> </div> _______________________________________________<br class=3D""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" class=3D&quot;&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br = class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<... = class=3D""> </div> </blockquote> </div> <br class=3D""> </div> </div> </div> _______________________________________________<br class=3D"">Users = mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org" = class=3D&quot;&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/users" = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<... = class=3D""></div></blockquote></div><br = class=3D""></div></div></div></blockquote></div><br = class=3D""></div></body></html>= --Apple-Mail=_8C444FA1-8569-42DE-B126-04B407031384--