Re: Hosted engine not starting after 4.3 Upgrade - cannot find OVF_STORE
It sure if this is the same bug I hit but check ownership of the cam images. There’s a bug in 4.3 upgrade that changes ownership to root and causes vms to not start until you change back to vdsm On Wed, Mar 6, 2019 at 4:57 AM Shawn Southern <shawn.southern@entegrus.com> wrote:
After running 'hosted-engine --vm-start', the status of the hosted engine VM is:
conf_on_shared_storage : True Status up-to-date : True Hostname : ovirtnode-01 Host ID : 3 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 7e3db850 local_conf_timestamp : 3509 Host timestamp : 3508 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=3508 (Tue Mar 5 16:03:30 2019) host-id=3 score=0 vm_conf_refresh_time=3509 (Tue Mar 5 16:03:31 2019) conf_on_shared_storage=True maintenance=False state=EngineUnexpectedlyDown stopped=False timeout=Wed Dec 31 20:05:37 1969
The /var/log/libvirt/qemu/HostedEngine.log has no entries since the hosted engine VM was rebooted.
/var/log/ovirt-hosted-engine-ha/agent.log: MainThread::ERROR::2019-03-05 16:07:31,916::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID': u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID': u'48ee766b-185d-4928-a046-b048d65af2a6'} failed: (code=201, message=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)) MainThread::ERROR::2019-03-05 16:07:31,916::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs MainThread::INFO::2019-03-05 16:07:31,971::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineUnexpectedlyDown (score: 0) MainThread::ERROR::2019-03-05 16:07:42,304::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID': u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID': u'48ee766b-185d-4928-a046-b048d65af2a6'} failed: (code=201, message=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)) MainThread::ERROR::2019-03-05 16:07:42,305::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs MainThread::INFO::2019-03-05 16:07:42,365::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineUnexpectedlyDown (score: 0) MainThread::ERROR::2019-03-05 16:07:51,791::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID': u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID': u'48ee766b-185d-4928-a046-b048d65af2a6'} failed: (code=201, message=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)) MainThread::ERROR::2019-03-05 16:07:51,792::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs MainThread::INFO::2019-03-05 16:07:51,850::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineUnexpectedlyDown (score: 0) MainThread::INFO::2019-03-05 16:08:01,868::states::684::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down, local host does not have best score MainThread::ERROR::2019-03-05 16:08:02,196::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID': u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID': u'48ee766b-185d-4928-a046-b048d65af2a6'} failed: (code=201, message=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)) MainThread::ERROR::2019-03-05 16:08:02,197::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs MainThread::INFO::2019-03-05 16:08:02,286::states::699::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) Score is 0 due to unexpected vm shutdown at Tue Mar 5 16:01:19 2019 MainThread::INFO::2019-03-05 16:08:02,286::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineUnexpectedlyDown (score: 0) MainThread::ERROR::2019-03-05 16:08:11,697::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID': u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID': u'48ee766b-185d-4928-a046-b048d65af2a6'} failed: (code=201, message=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)) MainThread::ERROR::2019-03-05 16:08:11,697::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs MainThread::INFO::2019-03-05 16:08:11,752::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineUnexpectedlyDown (score: 0)
I expect the issue is this: MainThread::ERROR::2019-03-05 16:08:11,697::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID': u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID': u'48ee766b-185d-4928-a046-b048d65af2a6'} failed: (code=201, message=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',))
Where do I go from here? _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RSPFVWDYFDMYNF...
Thank you! The ownership of the volume file had changed to root:root. I changed it back to vdsm:kvm and the hosted engine started. For anyone else who runs in to this, the file was in: /rhev/data-center/mnt/glusterSD/ovirtnode-02:_vmstore/79376c46-b80c-4c44-bbb1-80c0714a4b52/images/48ee766b-185d-4928-a046-b048d65af2a6 The errors in vdsm.log that pointed to this: 2019-03-06 08:16:24,470-0500 INFO (jsonrpc/4) [vdsm.api] START getVolumeInfo(sdUUID=u'79376c46-b80c-4c44-bbb1-80c0714a4b52', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'48ee766b-185d-4928-a046-b048d65af2a6', volUUID=u'687e9c0d-e988-4f76-89ff-931685acdf76', options=None) from=::1,37228, task_id=8170eec2-b3f7-488c-adda-3f1d9b1d0c57 (api:48) 2019-03-06 08:16:24,472-0500 INFO (jsonrpc/4) [vdsm.api] FINISH getVolumeInfo error=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',) from=::1,37228, task_id=8170eec2-b3f7-488c-adda-3f1d9b1d0c57 (api:52) 2019-03-06 08:16:24,472-0500 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='8170eec2-b3f7-488c-adda-3f1d9b1d0c57') aborting: Task is aborted: "Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)" - code 201 (task:1181) 2019-03-06 08:16:24,472-0500 INFO (jsonrpc/4) [storage.Dispatcher] FINISH getVolumeInfo error=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',) (dispatcher:81) From: Jayme <jaymef@gmail.com> Sent: March 6, 2019 6:51 AM To: Shawn Southern <shawn.southern@entegrus.com> Cc: users@ovirt.org Subject: Re: [ovirt-users] Hosted engine not starting after 4.3 Upgrade - cannot find OVF_STORE It sure if this is the same bug I hit but check ownership of the cam images. There’s a bug in 4.3 upgrade that changes ownership to root and causes vms to not start until you change back to vdsm On Wed, Mar 6, 2019 at 4:57 AM Shawn Southern <shawn.southern@entegrus.com<mailto:shawn.southern@entegrus.com>> wrote: After running 'hosted-engine --vm-start', the status of the hosted engine VM is: conf_on_shared_storage : True Status up-to-date : True Hostname : ovirtnode-01 Host ID : 3 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 7e3db850 local_conf_timestamp : 3509 Host timestamp : 3508 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=3508 (Tue Mar 5 16:03:30 2019) host-id=3 score=0 vm_conf_refresh_time=3509 (Tue Mar 5 16:03:31 2019) conf_on_shared_storage=True maintenance=False state=EngineUnexpectedlyDown stopped=False timeout=Wed Dec 31 20:05:37 1969 The /var/log/libvirt/qemu/HostedEngine.log has no entries since the hosted engine VM was rebooted. /var/log/ovirt-hosted-engine-ha/agent.log: MainThread::ERROR::2019-03-05 16:07:31,916::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID': u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID': u'48ee766b-185d-4928-a046-b048d65af2a6'} failed: (code=201, message=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)) MainThread::ERROR::2019-03-05 16:07:31,916::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs MainThread::INFO::2019-03-05 16:07:31,971::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineUnexpectedlyDown (score: 0) MainThread::ERROR::2019-03-05 16:07:42,304::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID': u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID': u'48ee766b-185d-4928-a046-b048d65af2a6'} failed: (code=201, message=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)) MainThread::ERROR::2019-03-05 16:07:42,305::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs MainThread::INFO::2019-03-05 16:07:42,365::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineUnexpectedlyDown (score: 0) MainThread::ERROR::2019-03-05 16:07:51,791::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID': u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID': u'48ee766b-185d-4928-a046-b048d65af2a6'} failed: (code=201, message=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)) MainThread::ERROR::2019-03-05 16:07:51,792::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs MainThread::INFO::2019-03-05 16:07:51,850::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineUnexpectedlyDown (score: 0) MainThread::INFO::2019-03-05 16:08:01,868::states::684::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down, local host does not have best score MainThread::ERROR::2019-03-05 16:08:02,196::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID': u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID': u'48ee766b-185d-4928-a046-b048d65af2a6'} failed: (code=201, message=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)) MainThread::ERROR::2019-03-05 16:08:02,197::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs MainThread::INFO::2019-03-05 16:08:02,286::states::699::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) Score is 0 due to unexpected vm shutdown at Tue Mar 5 16:01:19 2019 MainThread::INFO::2019-03-05 16:08:02,286::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineUnexpectedlyDown (score: 0) MainThread::ERROR::2019-03-05 16:08:11,697::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID': u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID': u'48ee766b-185d-4928-a046-b048d65af2a6'} failed: (code=201, message=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)) MainThread::ERROR::2019-03-05 16:08:11,697::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs MainThread::INFO::2019-03-05 16:08:11,752::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineUnexpectedlyDown (score: 0) I expect the issue is this: MainThread::ERROR::2019-03-05 16:08:11,697::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID': u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID': u'48ee766b-185d-4928-a046-b048d65af2a6'} failed: (code=201, message=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)) Where do I go from here? _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RSPFVWDYFDMYNF...
participants (2)
-
Jayme -
Shawn Southern