Hosted engine errors after 4.1 upgrade.
Seeing issues with my hosted engine, it seems it's unable to extract vm.conf from storage. My ovirt-hosted-engine-ha/agent.log is full of this repeating over and over. This is happening on all 3 of my hosts. My storage is glusterfs on the hosts themselves. Hopefully this is enough info to get started. Thanks! MainThread::INFO::2017-02-07 19:27:33,063::hosted_engine::612::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-02-07 19:27:35,455::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-02-07 19:27:35,456::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-02-07 19:27:40,169::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-02-07 19:27:40,202::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-02-07 19:27:40,418::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-02-07 19:27:40,419::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-02-07 19:27:43,370::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-02-07 19:27:43,371::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-02-07 19:27:45,968::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:3e14c1b5-5ade-4827-aad4-66c59824acd2, volUUID:3cbeeb3b-f755-4d42-a654-8dab34213792 MainThread::INFO::2017-02-07 19:27:46,257::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:9b49968b-5a62-4ab2-a2c5-b94bc0b2d3ce, volUUID:8f4d69c5-73a7-4e8c-a58f-909b55efec7d MainThread::INFO::2017-02-07 19:27:46,355::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-02-07 19:27:46,366::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt01-gluster.doonga.org: _engine/536cd721-4396-4029-b1ea-8ce84738137e/images/9b49968b-5a62-4ab2-a2c5-b94bc0b2d3ce/8f4d69c5-73a7-4e8c-a58f-909b55efec7d MainThread::ERROR::2017-02-07 19:27:46,389::ovf_store::124::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Unable to extract HEVM OVF MainThread::ERROR::2017-02-07 19:27:46,390::config::235::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf
On Wed, Feb 8, 2017 at 2:31 AM, Todd Punderson <punderstorm93@gmail.com> wrote:
Seeing issues with my hosted engine, it seems it's unable to extract vm.conf from storage. My ovirt-hosted-engine-ha/agent.log is full of this repeating over and over. This is happening on all 3 of my hosts. My storage is glusterfs on the hosts themselves.
Hopefully this is enough info to get started.
Thanks!
MainThread::INFO::2017-02-07 19:27:33,063::hosted_engine::612::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-02-07 19:27:35,455::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-02-07 19:27:35,456::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-02-07 19:27:40,169::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-02-07 19:27:40,202::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-02-07 19:27:40,418::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-02-07 19:27:40,419::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-02-07 19:27:43,370::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-02-07 19:27:43,371::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-02-07 19:27:45,968::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:3e14c1b5-5ade-4827-aad4-66c59824acd2, volUUID:3cbeeb3b-f755-4d42-a654-8dab34213792 MainThread::INFO::2017-02-07 19:27:46,257::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:9b49968b-5a62-4ab2-a2c5-b94bc0b2d3ce, volUUID:8f4d69c5-73a7-4e8c-a58f-909b55efec7d MainThread::INFO::2017-02-07 19:27:46,355::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-02-07 19:27:46,366::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt01-gluster.doonga.org:_engine/536cd721-4396-4029-b1ea-8ce84738137e/images/9b49968b-5a62-4ab2-a2c5-b94bc0b2d3ce/8f4d69c5-73a7-4e8c-a58f-909b55efec7d MainThread::ERROR::2017-02-07 19:27:46,389::ovf_store::124::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Unable to extract HEVM OVF MainThread::ERROR::2017-02-07 19:27:46,390::config::235::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf
Can you please attach the output of: sudo -u vdsm dd if=/rhev/data-center/mnt/glusterSD/ovirt01-gluster.doonga.org:_engine/536cd721-4396-4029-b1ea-8ce84738137e/images/9b49968b-5a62-4ab2-a2c5-b94bc0b2d3ce/8f4d69c5-73a7-4e8c-a58f-909b55efec7d | tar -tvf - Thanks. Did everything work well in 4.0? How did you upgrade? Best, -- Didi
On Wed, Feb 8, 2017 at 8:59 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Feb 8, 2017 at 2:31 AM, Todd Punderson <punderstorm93@gmail.com> wrote:
Seeing issues with my hosted engine, it seems it's unable to extract vm.conf from storage. My ovirt-hosted-engine-ha/agent.log is full of this repeating over and over. This is happening on all 3 of my hosts. My storage is glusterfs on the hosts themselves.
Hopefully this is enough info to get started.
Another step is editing /etc/ovirt-hosted-engine-ha/agent-log.conf changing from [logger_root] level=INFO handlers=syslog,logfile propagate=0 to [logger_root] level=DEBUG handlers=syslog,logfile propagate=0 and restart ovirt-ha-agent to get more detailed info about the issue.
Thanks!
MainThread::INFO::2017-02-07 19:27:33,063::hosted_engine::612::ovirt_hosted_engine_ha.
Initializing VDSM MainThread::INFO::2017-02-07 19:27:35,455::hosted_engine::639::ovirt_hosted_engine_ha. agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-02-07 19:27:35,456::storage_server::219::ovirt_hosted_engine_ha.
Connecting storage server MainThread::INFO::2017-02-07 19:27:40,169::storage_server::226::ovirt_hosted_engine_ha.
Connecting storage server MainThread::INFO::2017-02-07 19:27:40,202::storage_server::233::ovirt_hosted_engine_ha.
agent.hosted_engine.HostedEngine::(_initialize_vdsm) lib.storage_server.StorageServer::(connect_storage_server) lib.storage_server.StorageServer::(connect_storage_server) lib.storage_server.StorageServer::(connect_storage_server)
Refreshing the storage domain MainThread::INFO::2017-02-07 19:27:40,418::hosted_engine::666::ovirt_hosted_engine_ha. agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-02-07 19:27:40,419::image::126::ovirt_hosted_engine_ha.lib. image.Image::(prepare_images) Preparing images MainThread::INFO::2017-02-07 19:27:43,370::hosted_engine::669::ovirt_hosted_engine_ha. agent.hosted_engine.HostedEngine::(_initialize_storage_images) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-02-07 19:27:43,371::config::206::ovirt_hosted_engine_ha.agent. hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-02-07 19:27:45,968::ovf_store::103::ovirt_hosted_engine_ha.lib. ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:3e14c1b5-5ade-4827-aad4-66c59824acd2, volUUID:3cbeeb3b-f755-4d42-a654-8dab34213792 MainThread::INFO::2017-02-07 19:27:46,257::ovf_store::103::ovirt_hosted_engine_ha.lib. ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:9b49968b-5a62-4ab2-a2c5-b94bc0b2d3ce, volUUID:8f4d69c5-73a7-4e8c-a58f-909b55efec7d MainThread::INFO::2017-02-07 19:27:46,355::ovf_store::112::ovirt_hosted_engine_ha.lib. ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-02-07 19:27:46,366::ovf_store::119::ovirt_hosted_engine_ha.lib. ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt01-gluster.doonga.org: _engine/536cd721-4396-4029-b1ea-8ce84738137e/images/9b49968b-5a62-4ab2- a2c5-b94bc0b2d3ce/8f4d69c5-73a7-4e8c-a58f-909b55efec7d MainThread::ERROR::2017-02-07 19:27:46,389::ovf_store::124::ovirt_hosted_engine_ha.lib. ovf.ovf_store.OVFStore::(getEngineVMOVF) Unable to extract HEVM OVF MainThread::ERROR::2017-02-07 19:27:46,390::config::235::ovirt_hosted_engine_ha.agent. hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf
Can you please attach the output of:
sudo -u vdsm dd if=/rhev/data-center/mnt/glusterSD/ovirt01-gluster.doonga.org: _engine/536cd721-4396-4029-b1ea-8ce84738137e/images/9b49968b-5a62-4ab2- a2c5-b94bc0b2d3ce/8f4d69c5-73a7-4e8c-a58f-909b55efec7d | tar -tvf -
Thanks.
Did everything work well in 4.0? How did you upgrade?
Best, -- Didi _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi, I was able to resolve the issue by reinstalling the hosts via "Install/Reinstall" in the engine. I set the hosted engine to deploy, I did that on all three hosts and it seems to be working fine now. Thanks On Wed, Feb 8, 2017 at 2:59 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Feb 8, 2017 at 2:31 AM, Todd Punderson <punderstorm93@gmail.com> wrote:
Seeing issues with my hosted engine, it seems it's unable to extract vm.conf from storage. My ovirt-hosted-engine-ha/agent.log is full of this repeating over and over. This is happening on all 3 of my hosts. My storage is glusterfs on the hosts themselves.
Hopefully this is enough info to get started.
Thanks!
MainThread::INFO::2017-02-07
19:27:33,063::hosted_engine::612::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
Initializing VDSM MainThread::INFO::2017-02-07
19:27:35,455::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Connecting the storage MainThread::INFO::2017-02-07
19:27:35,456::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server MainThread::INFO::2017-02-07
19:27:40,169::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server MainThread::INFO::2017-02-07
19:27:40,202::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Refreshing the storage domain MainThread::INFO::2017-02-07
19:27:40,418::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Preparing images MainThread::INFO::2017-02-07
19:27:40,419::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
Preparing images MainThread::INFO::2017-02-07
19:27:43,370::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Reloading vm.conf from the shared storage domain MainThread::INFO::2017-02-07
19:27:43,371::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-02-07
19:27:45,968::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
Found OVF_STORE: imgUUID:3e14c1b5-5ade-4827-aad4-66c59824acd2, volUUID:3cbeeb3b-f755-4d42-a654-8dab34213792 MainThread::INFO::2017-02-07
19:27:46,257::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
Found OVF_STORE: imgUUID:9b49968b-5a62-4ab2-a2c5-b94bc0b2d3ce, volUUID:8f4d69c5-73a7-4e8c-a58f-909b55efec7d MainThread::INFO::2017-02-07
19:27:46,355::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-02-07
19:27:46,366::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt01-gluster.doonga.org: _engine/536cd721-4396-4029-b1ea-8ce84738137e/images/9b49968b-5a62-4ab2-a2c5-b94bc0b2d3ce/8f4d69c5-73a7-4e8c-a58f-909b55efec7d MainThread::ERROR::2017-02-07
19:27:46,389::ovf_store::124::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
Unable to extract HEVM OVF MainThread::ERROR::2017-02-07
19:27:46,390::config::235::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf
Can you please attach the output of:
sudo -u vdsm dd if=/rhev/data-center/mnt/glusterSD/ovirt01-gluster.doonga.org: _engine/536cd721-4396-4029-b1ea-8ce84738137e/images/9b49968b-5a62-4ab2-a2c5-b94bc0b2d3ce/8f4d69c5-73a7-4e8c-a58f-909b55efec7d | tar -tvf -
Thanks.
Did everything work well in 4.0? How did you upgrade?
Best, -- Didi
participants (3)
-
Simone Tiraboschi -
Todd Punderson -
Yedidyah Bar David