Re: [ovirt-users] HostedEngine VM not visible, but running

Hi Jenny/Martin, Any idea what I can do here? The hosted engine VM has no log on any host in /var/log/libvirt/qemu, and I fear that if I need to put the host into maintenance, e.g., to upgrade it that I created it on (which I think is hosting it), or if it fails for any reason, it won't get migrated to another host, and I will not be able to manage the cluster. It seems to be a very dangerous position to be in. Thanks, Cam On Wed, Jun 21, 2017 at 11:48 AM, cmc <iucounu(a)gmail.com&gt; wrote: > Thanks Martin. The hosts are all part of the same cluster. > > I get these errors in the engine.log on the engine: > > 2017-06-19 03:28:05,030Z WARN > [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] > (org.ovirt.thread.pool-6-thread-23) [] Validation of action 'ImportVm' > failed for user SYST > EM. Reasons: VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_ ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS > 2017-06-19 03:28:05,030Z INFO > [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] > (org.ovirt.thread.pool-6-thread-23) [] Lock freed to object > 'EngineLock:{exclusiveLocks='[a > 79e6b0e-fff4-4cba-a02c-4c00be151300=<VM, > ACTION_TYPE_FAILED_VM_IS_BEING_IMPORTED$VmName HostedEngine>, > HostedEngine=<VM_NAME, ACTION_TYPE_FAILED_NAME_ALREADY_USED>]', > sharedLocks= > '[a79e6b0e-fff4-4cba-a02c-4c00be151300=<REMOTE_VM, > ACTION_TYPE_FAILED_VM_IS_BEING_IMPORTED$VmName HostedEngine>]'}' > 2017-06-19 03:28:05,030Z ERROR > [org.ovirt.engine.core.bll.HostedEngineImporter] > (org.ovirt.thread.pool-6-thread-23) [] Failed importing the Hosted > Engine VM > > The sanlock.log reports conflicts on that same host, and a different > error on the other hosts, not sure if they are related. > > And this in the /var/log/ovirt-hosted-engine-ha/agent log on the host > which I deployed the hosted engine VM on: > > MainThread::ERROR::2017-06-19 > 13:09:49,743::ovf_store::124::ovirt_hosted_engine_ha.lib. ovf.ovf_store.OVFStore::(getEngineVMOVF) > Unable to extract HEVM OVF > MainThread::ERROR::2017-06-19 > 13:09:49,743::config::445::ovirt_hosted_engine_ha.agent. hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) > Failed extracting VM OVF from the OVF_STORE volume, falling back to > initial vm.conf > > I've seen some of these issues reported in bugzilla, but they were for > older versions of oVirt (and appear to be resolved). > > I will install that package on the other two hosts, for which I will > put them in maintenance as vdsm is installed as an upgrade. I guess > restarting vdsm is a good idea after that? > > Thanks, > > Campbell > > On Wed, Jun 21, 2017 at 10:51 AM, Martin Sivak <msivak(a)redhat.com&gt; wrote: >> Hi, >> >> you do not have to install it on all hosts. But you should have more >> than one and ideally all hosted engine enabled nodes should belong to >> the same engine cluster. >> >> Best regards >> >> Martin Sivak >> >> On Wed, Jun 21, 2017 at 11:29 AM, cmc <iucounu(a)gmail.com&gt; wrote: >>> Hi Jenny, >>> >>> Does ovirt-hosted-engine-ha need to be installed across all hosts? >>> Could that be the reason it is failing to see it properly? >>> >>> Thanks, >>> >>> Cam >>> >>> On Mon, Jun 19, 2017 at 1:27 PM, cmc <iucounu(a)gmail.com&gt; wrote: >>>> Hi Jenny, >>>> >>>> Logs are attached. I can see errors in there, but am unsure how they arose. >>>> >>>> Thanks, >>>> >>>> Campbell >>>> >>>> On Mon, Jun 19, 2017 at 12:29 PM, Evgenia Tokar <etokar(a)redhat.com&gt; wrote: >>>>> From the output it looks like the agent is down, try starting it by running: >>>>> systemctl start ovirt-ha-agent. >>>>> >>>>> The engine is supposed to see the hosted engine storage domain and import it >>>>> to the system, then it should import the hosted engine vm. >>>>> >>>>> Can you attach the agent log from the host >>>>> (/var/log/ovirt-hosted-engine-ha/agent.log) >>>>> and the engine log from the engine vm (/var/log/ovirt-engine/engine. log)? >>>>> >>>>> Thanks, >>>>> Jenny >>>>> >>>>> >>>>> On Mon, Jun 19, 2017 at 12:41 PM, cmc <iucounu(a)gmail.com&gt; wrote: >>>>>> >>>>>> Hi Jenny, >>>>>> >>>>>> > What version are you running? >>>>>> >>>>>> 4.1.2.2-1.el7.centos >>>>>> >>>>>> > For the hosted engine vm to be imported and displayed in the engine, you >>>>>> > must first create a master storage domain. >>>>>> >>>>>> To provide a bit more detail: this was a migration of a bare-metal >>>>>> engine in an existing cluster to a hosted engine VM for that cluster. >>>>>> As part of this migration, I built an entirely new host and ran >>>>>> 'hosted-engine --deploy' (followed these instructions: >>>>>> >>>>>> http://www.ovirt.org/documentation/self-hosted/ chap-Migrating_from_Bare_Metal_to_an_EL-Based_Self-Hosted_Environment/). >>>>>> I restored the backup from the engine and it completed without any >>>>>> errors. I didn't see any instructions regarding a master storage >>>>>> domain in the page above. The cluster has two existing master storage >>>>>> domains, one is fibre channel, which is up, and one ISO domain, which >>>>>> is currently offline. >>>>>> >>>>>> > What do you mean the hosted engine commands are failing? What happens >>>>>> > when >>>>>> > you run hosted-engine --vm-status now? >>>>>> >>>>>> Interestingly, whereas when I ran it before, it exited with no output >>>>>> and a return code of '1', it now reports: >>>>>> >>>>>> --== Host 1 status ==-- >>>>>> >>>>>> conf_on_shared_storage : True >>>>>> Status up-to-date : False >>>>>> Hostname : kvm-ldn-03.ldn.fscfc.co.uk >>>>>> Host ID : 1 >>>>>> Engine status : unknown stale-data >>>>>> Score : 0 >>>>>> stopped : True >>>>>> Local maintenance : False >>>>>> crc32 : 0217f07b >>>>>> local_conf_timestamp : 2911 >>>>>> Host timestamp : 2897 >>>>>> Extra metadata (valid at timestamp): >>>>>> metadata_parse_version=1 >>>>>> metadata_feature_version=1 >>>>>> timestamp=2897 (Thu Jun 15 16:22:54 2017) >>>>>> host-id=1 >>>>>> score=0 >>>>>> vm_conf_refresh_time=2911 (Thu Jun 15 16:23:08 2017) >>>>>> conf_on_shared_storage=True >>>>>> maintenance=False >>>>>> state=AgentStopped >>>>>> stopped=True >>>>>> >>>>>> Yet I can login to the web GUI fine. I guess it is not HA due to being >>>>>> in an unknown state currently? Does the hosted-engine-ha rpm need to >>>>>> be installed across all nodes in the cluster, btw? >>>>>> >>>>>> Thanks for the help, >>>>>> >>>>>> Cam >>>>>> >>>>>> > >>>>>> > Jenny Tokar >>>>>> > >>>>>> > >>>>>> > On Thu, Jun 15, 2017 at 6:32 PM, cmc <iucounu(a)gmail.com&gt; wrote: >>>>>> >> >>>>>> >> Hi, >>>>>> >> >>>>>> >> I've migrated from a bare-metal engine to a hosted engine. There were >>>>>> >> no errors during the install, however, the hosted engine did not get >>>>>> >> started. I tried running: >>>>>> >> >>>>>> >> hosted-engine --status >>>>>> >> >>>>>> >> on the host I deployed it on, and it returns nothing (exit code is 1 >>>>>> >> however). I could not ping it either. So I tried starting it via >>>>>> >> 'hosted-engine --vm-start' and it returned: >>>>>> >> >>>>>> >> Virtual machine does not exist >>>>>> >> >>>>>> >> But it then became available. I logged into it successfully. It is not >>>>>> >> in the list of VMs however. >>>>>> >> >>>>>> >> Any ideas why the hosted-engine commands fail, and why it is not in >>>>>> >> the list of virtual machines? >>>>>> >> >>>>>> >> Thanks for any help, >>>>>> >> >>>>>> >> Cam >>>>>> >> _______________________________________________ >>>>>> >> Users mailing list >>>>>> >> Users(a)ovirt.org >>>>>> >> http://lists.ovirt.org/mailman/listinfo/users >>>>>> > >>>>>> > >>>>> >>>>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users