Re: [ovirt-users] HostedEngine VM not visible, but running

Hi Yanir, Thanks for the reply. > First of all, maybe a chain reaction of : > WARN [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] > (org.ovirt.thread.pool-6-thread-23) [] Validation of action 'ImportVm' > failed for user SYSTEM. Reasons: VAR__ACTION__IMPORT > ,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS > is causing the hosted engine vm not to be set up correctly and further > actions were made when the hosted engine vm wasnt in a stable state. > > As for now, are you trying to revert back to a previous/initial state ? I'm not trying to revert it to a previous state for now. This was a migration from a bare metal engine, and it didn't report any error during the migration. I'd had some problems on my first attempts at this migration, whereby it never completed (due to a proxy issue) but I managed to resolve this. Do you know of a way to get the Hosted Engine VM into a stable state, without rebuilding the entire cluster from scratch (since I have a lot of VMs on it)? Thanks for any help. Regards, Cam > Regards, > Yanir > > On Wed, Jun 21, 2017 at 4:32 PM, cmc <iucounu(a)gmail.com&gt; wrote: >> >> Hi Jenny/Martin, >> >> Any idea what I can do here? The hosted engine VM has no log on any >> host in /var/log/libvirt/qemu, and I fear that if I need to put the >> host into maintenance, e.g., to upgrade it that I created it on (which >> I think is hosting it), or if it fails for any reason, it won't get >> migrated to another host, and I will not be able to manage the >> cluster. It seems to be a very dangerous position to be in. >> >> Thanks, >> >> Cam >> >> On Wed, Jun 21, 2017 at 11:48 AM, cmc <iucounu(a)gmail.com&gt; wrote: >> > Thanks Martin. The hosts are all part of the same cluster. >> > >> > I get these errors in the engine.log on the engine: >> > >> > 2017-06-19 03:28:05,030Z WARN >> > [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >> > (org.ovirt.thread.pool-6-thread-23) [] Validation of action 'ImportVm' >> > failed for user SYST >> > EM. Reasons: >> > VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS >> > 2017-06-19 03:28:05,030Z INFO >> > [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >> > (org.ovirt.thread.pool-6-thread-23) [] Lock freed to object >> > 'EngineLock:{exclusiveLocks='[a >> > 79e6b0e-fff4-4cba-a02c-4c00be151300=<VM, >> > ACTION_TYPE_FAILED_VM_IS_BEING_IMPORTED$VmName HostedEngine>, >> > HostedEngine=<VM_NAME, ACTION_TYPE_FAILED_NAME_ALREADY_USED>]', >> > sharedLocks= >> > '[a79e6b0e-fff4-4cba-a02c-4c00be151300=<REMOTE_VM, >> > ACTION_TYPE_FAILED_VM_IS_BEING_IMPORTED$VmName HostedEngine>]'}' >> > 2017-06-19 03:28:05,030Z ERROR >> > [org.ovirt.engine.core.bll.HostedEngineImporter] >> > (org.ovirt.thread.pool-6-thread-23) [] Failed importing the Hosted >> > Engine VM >> > >> > The sanlock.log reports conflicts on that same host, and a different >> > error on the other hosts, not sure if they are related. >> > >> > And this in the /var/log/ovirt-hosted-engine-ha/agent log on the host >> > which I deployed the hosted engine VM on: >> > >> > MainThread::ERROR::2017-06-19 >> > >> > 13:09:49,743::ovf_store::124::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >> > Unable to extract HEVM OVF >> > MainThread::ERROR::2017-06-19 >> > >> > 13:09:49,743::config::445::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >> > Failed extracting VM OVF from the OVF_STORE volume, falling back to >> > initial vm.conf >> > >> > I've seen some of these issues reported in bugzilla, but they were for >> > older versions of oVirt (and appear to be resolved). >> > >> > I will install that package on the other two hosts, for which I will >> > put them in maintenance as vdsm is installed as an upgrade. I guess >> > restarting vdsm is a good idea after that? >> > >> > Thanks, >> > >> > Campbell >> > >> > On Wed, Jun 21, 2017 at 10:51 AM, Martin Sivak <msivak(a)redhat.com&gt; >> > wrote: >> >> Hi, >> >> >> >> you do not have to install it on all hosts. But you should have more >> >> than one and ideally all hosted engine enabled nodes should belong to >> >> the same engine cluster. >> >> >> >> Best regards >> >> >> >> Martin Sivak >> >> >> >> On Wed, Jun 21, 2017 at 11:29 AM, cmc <iucounu(a)gmail.com&gt; wrote: >> >>> Hi Jenny, >> >>> >> >>> Does ovirt-hosted-engine-ha need to be installed across all hosts? >> >>> Could that be the reason it is failing to see it properly? >> >>> >> >>> Thanks, >> >>> >> >>> Cam >> >>> >> >>> On Mon, Jun 19, 2017 at 1:27 PM, cmc <iucounu(a)gmail.com&gt; wrote: >> >>>> Hi Jenny, >> >>>> >> >>>> Logs are attached. I can see errors in there, but am unsure how they >> >>>> arose. >> >>>> >> >>>> Thanks, >> >>>> >> >>>> Campbell >> >>>> >> >>>> On Mon, Jun 19, 2017 at 12:29 PM, Evgenia Tokar <etokar(a)redhat.com&gt; >> >>>> wrote: >> >>>>> From the output it looks like the agent is down, try starting it by >> >>>>> running: >> >>>>> systemctl start ovirt-ha-agent. >> >>>>> >> >>>>> The engine is supposed to see the hosted engine storage domain and >> >>>>> import it >> >>>>> to the system, then it should import the hosted engine vm. >> >>>>> >> >>>>> Can you attach the agent log from the host >> >>>>> (/var/log/ovirt-hosted-engine-ha/agent.log) >> >>>>> and the engine log from the engine vm >> >>>>> (/var/log/ovirt-engine/engine.log)? >> >>>>> >> >>>>> Thanks, >> >>>>> Jenny >> >>>>> >> >>>>> >> >>>>> On Mon, Jun 19, 2017 at 12:41 PM, cmc <iucounu(a)gmail.com&gt; wrote: >> >>>>>> >> >>>>>> Hi Jenny, >> >>>>>> >> >>>>>> > What version are you running? >> >>>>>> >> >>>>>> 4.1.2.2-1.el7.centos >> >>>>>> >> >>>>>> > For the hosted engine vm to be imported and displayed in the >> >>>>>> > engine, you >> >>>>>> > must first create a master storage domain. >> >>>>>> >> >>>>>> To provide a bit more detail: this was a migration of a bare-metal >> >>>>>> engine in an existing cluster to a hosted engine VM for that >> >>>>>> cluster. >> >>>>>> As part of this migration, I built an entirely new host and ran >> >>>>>> 'hosted-engine --deploy' (followed these instructions: >> >>>>>> >> >>>>>> >> >>>>>> http://www.ovirt.org/documentation/self-hosted/chap-Migrating_from_Bare_M...). >> >>>>>> I restored the backup from the engine and it completed without any >> >>>>>> errors. I didn't see any instructions regarding a master storage >> >>>>>> domain in the page above. The cluster has two existing master >> >>>>>> storage >> >>>>>> domains, one is fibre channel, which is up, and one ISO domain, >> >>>>>> which >> >>>>>> is currently offline. >> >>>>>> >> >>>>>> > What do you mean the hosted engine commands are failing? What >> >>>>>> > happens >> >>>>>> > when >> >>>>>> > you run hosted-engine --vm-status now? >> >>>>>> >> >>>>>> Interestingly, whereas when I ran it before, it exited with no >> >>>>>> output >> >>>>>> and a return code of '1', it now reports: >> >>>>>> >> >>>>>> --== Host 1 status ==-- >> >>>>>> >> >>>>>> conf_on_shared_storage : True >> >>>>>> Status up-to-date : False >> >>>>>> Hostname : kvm-ldn-03.ldn.fscfc.co.uk >> >>>>>> Host ID : 1 >> >>>>>> Engine status : unknown stale-data >> >>>>>> Score : 0 >> >>>>>> stopped : True >> >>>>>> Local maintenance : False >> >>>>>> crc32 : 0217f07b >> >>>>>> local_conf_timestamp : 2911 >> >>>>>> Host timestamp : 2897 >> >>>>>> Extra metadata (valid at timestamp): >> >>>>>> metadata_parse_version=1 >> >>>>>> metadata_feature_version=1 >> >>>>>> timestamp=2897 (Thu Jun 15 16:22:54 2017) >> >>>>>> host-id=1 >> >>>>>> score=0 >> >>>>>> vm_conf_refresh_time=2911 (Thu Jun 15 16:23:08 2017) >> >>>>>> conf_on_shared_storage=True >> >>>>>> maintenance=False >> >>>>>> state=AgentStopped >> >>>>>> stopped=True >> >>>>>> >> >>>>>> Yet I can login to the web GUI fine. I guess it is not HA due to >> >>>>>> being >> >>>>>> in an unknown state currently? Does the hosted-engine-ha rpm need >> >>>>>> to >> >>>>>> be installed across all nodes in the cluster, btw? >> >>>>>> >> >>>>>> Thanks for the help, >> >>>>>> >> >>>>>> Cam >> >>>>>> >> >>>>>> > >> >>>>>> > Jenny Tokar >> >>>>>> > >> >>>>>> > >> >>>>>> > On Thu, Jun 15, 2017 at 6:32 PM, cmc <iucounu(a)gmail.com&gt; wrote: >> >>>>>> >> >> >>>>>> >> Hi, >> >>>>>> >> >> >>>>>> >> I've migrated from a bare-metal engine to a hosted engine. There >> >>>>>> >> were >> >>>>>> >> no errors during the install, however, the hosted engine did not >> >>>>>> >> get >> >>>>>> >> started. I tried running: >> >>>>>> >> >> >>>>>> >> hosted-engine --status >> >>>>>> >> >> >>>>>> >> on the host I deployed it on, and it returns nothing (exit code >> >>>>>> >> is 1 >> >>>>>> >> however). I could not ping it either. So I tried starting it via >> >>>>>> >> 'hosted-engine --vm-start' and it returned: >> >>>>>> >> >> >>>>>> >> Virtual machine does not exist >> >>>>>> >> >> >>>>>> >> But it then became available. I logged into it successfully. It >> >>>>>> >> is not >> >>>>>> >> in the list of VMs however. >> >>>>>> >> >> >>>>>> >> Any ideas why the hosted-engine commands fail, and why it is not >> >>>>>> >> in >> >>>>>> >> the list of virtual machines? >> >>>>>> >> >> >>>>>> >> Thanks for any help, >> >>>>>> >> >> >>>>>> >> Cam >> >>>>>> >> _______________________________________________ >> >>>>>> >> Users mailing list >> >>>>>> >> Users(a)ovirt.org >> >>>>>> >> http://lists.ovirt.org/mailman/listinfo/users >> >>>>>> > >> >>>>>> > >> >>>>> >> >>>>> >> >>> _______________________________________________ >> >>> Users mailing list >> >>> Users(a)ovirt.org >> >>> http://lists.ovirt.org/mailman/listinfo/users >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > >