Re: [ovirt-users] HostedEngine VM not visible, but running

Hi Tomas, So in my /usr/share/ovirt-engine/conf/osinfo-defaults.properties on my engine VM, I have: os.other.devices.display.protocols.value = spice/qxl,vnc/vga,vnc/qxl,vnc/cirrus os.other.devices.display.protocols.value.3.6 = spice/qxl,vnc/cirrus,vnc/qxl That seems to match - I assume since this is 4.1, the 3.6 should not apply Is there somewhere else I should be looking? Thanks, Cam On Thu, Jun 22, 2017 at 11:40 AM, Tomas Jelinek <tjelinek(a)redhat.com&gt; wrote: > > > On Thu, Jun 22, 2017 at 12:38 PM, Michal Skrivanek > <michal.skrivanek(a)redhat.com&gt; wrote: >> >> >> > On 22 Jun 2017, at 12:31, Martin Sivak <msivak(a)redhat.com&gt; wrote: >> > >> > Tomas, what fields are needed in a VM to pass the check that causes >> > the following error? >> > >> >>>>> WARN [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >> >>>>> (org.ovirt.thread.pool-6-thread-23) [] Validation of action >> >>>>> 'ImportVm' >> >>>>> failed for user SYSTEM. Reasons: VAR__ACTION__IMPORT >> >>>>> >> >>>>> ,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS >> >> to match the OS and VM Display type;-) >> Configuration is in osinfo….e.g. if that is import from older releases on >> Linux this is typically caused by the cahgen of cirrus to vga for non-SPICE >> VMs > > > yep, the default supported combinations for 4.0+ is this: > os.other.devices.display.protocols.value = > spice/qxl,vnc/vga,vnc/qxl,vnc/cirrus > >> >> >> > >> > Thanks. >> > >> > On Thu, Jun 22, 2017 at 12:19 PM, cmc <iucounu(a)gmail.com&gt; wrote: >> >> Hi Martin, >> >> >> >>> >> >>> just as a random comment, do you still have the database backup from >> >>> the bare metal -> VM attempt? It might be possible to just try again >> >>> using it. Or in the worst case.. update the offending value there >> >>> before restoring it to the new engine instance. >> >> >> >> I still have the backup. I'd rather do the latter, as re-running the >> >> HE deployment is quite lengthy and involved (I have to re-initialise >> >> the FC storage each time). Do you know what the offending value(s) >> >> would be? Would it be in the Postgres DB or in a config file >> >> somewhere? >> >> >> >> Cheers, >> >> >> >> Cam >> >> >> >>> Regards >> >>> >> >>> Martin Sivak >> >>> >> >>> On Thu, Jun 22, 2017 at 11:39 AM, cmc <iucounu(a)gmail.com&gt; wrote: >> >>>> Hi Yanir, >> >>>> >> >>>> Thanks for the reply. >> >>>> >> >>>>> First of all, maybe a chain reaction of : >> >>>>> WARN [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >> >>>>> (org.ovirt.thread.pool-6-thread-23) [] Validation of action >> >>>>> 'ImportVm' >> >>>>> failed for user SYSTEM. Reasons: VAR__ACTION__IMPORT >> >>>>> >> >>>>> ,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS >> >>>>> is causing the hosted engine vm not to be set up correctly and >> >>>>> further >> >>>>> actions were made when the hosted engine vm wasnt in a stable state. >> >>>>> >> >>>>> As for now, are you trying to revert back to a previous/initial >> >>>>> state ? >> >>>> >> >>>> I'm not trying to revert it to a previous state for now. This was a >> >>>> migration from a bare metal engine, and it didn't report any error >> >>>> during the migration. I'd had some problems on my first attempts at >> >>>> this migration, whereby it never completed (due to a proxy issue) but >> >>>> I managed to resolve this. Do you know of a way to get the Hosted >> >>>> Engine VM into a stable state, without rebuilding the entire cluster >> >>>> from scratch (since I have a lot of VMs on it)? >> >>>> >> >>>> Thanks for any help. >> >>>> >> >>>> Regards, >> >>>> >> >>>> Cam >> >>>> >> >>>>> Regards, >> >>>>> Yanir >> >>>>> >> >>>>> On Wed, Jun 21, 2017 at 4:32 PM, cmc <iucounu(a)gmail.com&gt; wrote: >> >>>>>> >> >>>>>> Hi Jenny/Martin, >> >>>>>> >> >>>>>> Any idea what I can do here? The hosted engine VM has no log on any >> >>>>>> host in /var/log/libvirt/qemu, and I fear that if I need to put the >> >>>>>> host into maintenance, e.g., to upgrade it that I created it on >> >>>>>> (which >> >>>>>> I think is hosting it), or if it fails for any reason, it won't get >> >>>>>> migrated to another host, and I will not be able to manage the >> >>>>>> cluster. It seems to be a very dangerous position to be in. >> >>>>>> >> >>>>>> Thanks, >> >>>>>> >> >>>>>> Cam >> >>>>>> >> >>>>>> On Wed, Jun 21, 2017 at 11:48 AM, cmc <iucounu(a)gmail.com&gt; wrote: >> >>>>>>> Thanks Martin. The hosts are all part of the same cluster. >> >>>>>>> >> >>>>>>> I get these errors in the engine.log on the engine: >> >>>>>>> >> >>>>>>> 2017-06-19 03:28:05,030Z WARN >> >>>>>>> [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >> >>>>>>> (org.ovirt.thread.pool-6-thread-23) [] Validation of action >> >>>>>>> 'ImportVm' >> >>>>>>> failed for user SYST >> >>>>>>> EM. Reasons: >> >>>>>>> >> >>>>>>> VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS >> >>>>>>> 2017-06-19 03:28:05,030Z INFO >> >>>>>>> [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >> >>>>>>> (org.ovirt.thread.pool-6-thread-23) [] Lock freed to object >> >>>>>>> 'EngineLock:{exclusiveLocks='[a >> >>>>>>> 79e6b0e-fff4-4cba-a02c-4c00be151300=<VM, >> >>>>>>> ACTION_TYPE_FAILED_VM_IS_BEING_IMPORTED$VmName HostedEngine>, >> >>>>>>> HostedEngine=<VM_NAME, ACTION_TYPE_FAILED_NAME_ALREADY_USED>]', >> >>>>>>> sharedLocks= >> >>>>>>> '[a79e6b0e-fff4-4cba-a02c-4c00be151300=<REMOTE_VM, >> >>>>>>> ACTION_TYPE_FAILED_VM_IS_BEING_IMPORTED$VmName HostedEngine>]'}' >> >>>>>>> 2017-06-19 03:28:05,030Z ERROR >> >>>>>>> [org.ovirt.engine.core.bll.HostedEngineImporter] >> >>>>>>> (org.ovirt.thread.pool-6-thread-23) [] Failed importing the Hosted >> >>>>>>> Engine VM >> >>>>>>> >> >>>>>>> The sanlock.log reports conflicts on that same host, and a >> >>>>>>> different >> >>>>>>> error on the other hosts, not sure if they are related. >> >>>>>>> >> >>>>>>> And this in the /var/log/ovirt-hosted-engine-ha/agent log on the >> >>>>>>> host >> >>>>>>> which I deployed the hosted engine VM on: >> >>>>>>> >> >>>>>>> MainThread::ERROR::2017-06-19 >> >>>>>>> >> >>>>>>> >> >>>>>>> 13:09:49,743::ovf_store::124::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >> >>>>>>> Unable to extract HEVM OVF >> >>>>>>> MainThread::ERROR::2017-06-19 >> >>>>>>> >> >>>>>>> >> >>>>>>> 13:09:49,743::config::445::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >> >>>>>>> Failed extracting VM OVF from the OVF_STORE volume, falling back >> >>>>>>> to >> >>>>>>> initial vm.conf >> >>>>>>> >> >>>>>>> I've seen some of these issues reported in bugzilla, but they were >> >>>>>>> for >> >>>>>>> older versions of oVirt (and appear to be resolved). >> >>>>>>> >> >>>>>>> I will install that package on the other two hosts, for which I >> >>>>>>> will >> >>>>>>> put them in maintenance as vdsm is installed as an upgrade. I >> >>>>>>> guess >> >>>>>>> restarting vdsm is a good idea after that? >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> >> >>>>>>> Campbell >> >>>>>>> >> >>>>>>> On Wed, Jun 21, 2017 at 10:51 AM, Martin Sivak <msivak(a)redhat.com&gt; >> >>>>>>> wrote: >> >>>>>>>> Hi, >> >>>>>>>> >> >>>>>>>> you do not have to install it on all hosts. But you should have >> >>>>>>>> more >> >>>>>>>> than one and ideally all hosted engine enabled nodes should >> >>>>>>>> belong to >> >>>>>>>> the same engine cluster. >> >>>>>>>> >> >>>>>>>> Best regards >> >>>>>>>> >> >>>>>>>> Martin Sivak >> >>>>>>>> >> >>>>>>>> On Wed, Jun 21, 2017 at 11:29 AM, cmc <iucounu(a)gmail.com&gt; wrote: >> >>>>>>>>> Hi Jenny, >> >>>>>>>>> >> >>>>>>>>> Does ovirt-hosted-engine-ha need to be installed across all >> >>>>>>>>> hosts? >> >>>>>>>>> Could that be the reason it is failing to see it properly? >> >>>>>>>>> >> >>>>>>>>> Thanks, >> >>>>>>>>> >> >>>>>>>>> Cam >> >>>>>>>>> >> >>>>>>>>> On Mon, Jun 19, 2017 at 1:27 PM, cmc <iucounu(a)gmail.com&gt; wrote: >> >>>>>>>>>> Hi Jenny, >> >>>>>>>>>> >> >>>>>>>>>> Logs are attached. I can see errors in there, but am unsure how >> >>>>>>>>>> they >> >>>>>>>>>> arose. >> >>>>>>>>>> >> >>>>>>>>>> Thanks, >> >>>>>>>>>> >> >>>>>>>>>> Campbell >> >>>>>>>>>> >> >>>>>>>>>> On Mon, Jun 19, 2017 at 12:29 PM, Evgenia Tokar >> >>>>>>>>>> <etokar(a)redhat.com&gt; >> >>>>>>>>>> wrote: >> >>>>>>>>>>> From the output it looks like the agent is down, try starting >> >>>>>>>>>>> it by >> >>>>>>>>>>> running: >> >>>>>>>>>>> systemctl start ovirt-ha-agent. >> >>>>>>>>>>> >> >>>>>>>>>>> The engine is supposed to see the hosted engine storage domain >> >>>>>>>>>>> and >> >>>>>>>>>>> import it >> >>>>>>>>>>> to the system, then it should import the hosted engine vm. >> >>>>>>>>>>> >> >>>>>>>>>>> Can you attach the agent log from the host >> >>>>>>>>>>> (/var/log/ovirt-hosted-engine-ha/agent.log) >> >>>>>>>>>>> and the engine log from the engine vm >> >>>>>>>>>>> (/var/log/ovirt-engine/engine.log)? >> >>>>>>>>>>> >> >>>>>>>>>>> Thanks, >> >>>>>>>>>>> Jenny >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> On Mon, Jun 19, 2017 at 12:41 PM, cmc <iucounu(a)gmail.com&gt; >> >>>>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>> Hi Jenny, >> >>>>>>>>>>>> >> >>>>>>>>>>>>> What version are you running? >> >>>>>>>>>>>> >> >>>>>>>>>>>> 4.1.2.2-1.el7.centos >> >>>>>>>>>>>> >> >>>>>>>>>>>>> For the hosted engine vm to be imported and displayed in the >> >>>>>>>>>>>>> engine, you >> >>>>>>>>>>>>> must first create a master storage domain. >> >>>>>>>>>>>> >> >>>>>>>>>>>> To provide a bit more detail: this was a migration of a >> >>>>>>>>>>>> bare-metal >> >>>>>>>>>>>> engine in an existing cluster to a hosted engine VM for that >> >>>>>>>>>>>> cluster. >> >>>>>>>>>>>> As part of this migration, I built an entirely new host and >> >>>>>>>>>>>> ran >> >>>>>>>>>>>> 'hosted-engine --deploy' (followed these instructions: >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> http://www.ovirt.org/documentation/self-hosted/chap-Migrating_from_Bare_M...). >> >>>>>>>>>>>> I restored the backup from the engine and it completed >> >>>>>>>>>>>> without any >> >>>>>>>>>>>> errors. I didn't see any instructions regarding a master >> >>>>>>>>>>>> storage >> >>>>>>>>>>>> domain in the page above. The cluster has two existing master >> >>>>>>>>>>>> storage >> >>>>>>>>>>>> domains, one is fibre channel, which is up, and one ISO >> >>>>>>>>>>>> domain, >> >>>>>>>>>>>> which >> >>>>>>>>>>>> is currently offline. >> >>>>>>>>>>>> >> >>>>>>>>>>>>> What do you mean the hosted engine commands are failing? >> >>>>>>>>>>>>> What >> >>>>>>>>>>>>> happens >> >>>>>>>>>>>>> when >> >>>>>>>>>>>>> you run hosted-engine --vm-status now? >> >>>>>>>>>>>> >> >>>>>>>>>>>> Interestingly, whereas when I ran it before, it exited with >> >>>>>>>>>>>> no >> >>>>>>>>>>>> output >> >>>>>>>>>>>> and a return code of '1', it now reports: >> >>>>>>>>>>>> >> >>>>>>>>>>>> --== Host 1 status ==-- >> >>>>>>>>>>>> >> >>>>>>>>>>>> conf_on_shared_storage : True >> >>>>>>>>>>>> Status up-to-date : False >> >>>>>>>>>>>> Hostname : >> >>>>>>>>>>>> kvm-ldn-03.ldn.fscfc.co.uk >> >>>>>>>>>>>> Host ID : 1 >> >>>>>>>>>>>> Engine status : unknown stale-data >> >>>>>>>>>>>> Score : 0 >> >>>>>>>>>>>> stopped : True >> >>>>>>>>>>>> Local maintenance : False >> >>>>>>>>>>>> crc32 : 0217f07b >> >>>>>>>>>>>> local_conf_timestamp : 2911 >> >>>>>>>>>>>> Host timestamp : 2897 >> >>>>>>>>>>>> Extra metadata (valid at timestamp): >> >>>>>>>>>>>> metadata_parse_version=1 >> >>>>>>>>>>>> metadata_feature_version=1 >> >>>>>>>>>>>> timestamp=2897 (Thu Jun 15 16:22:54 2017) >> >>>>>>>>>>>> host-id=1 >> >>>>>>>>>>>> score=0 >> >>>>>>>>>>>> vm_conf_refresh_time=2911 (Thu Jun 15 16:23:08 2017) >> >>>>>>>>>>>> conf_on_shared_storage=True >> >>>>>>>>>>>> maintenance=False >> >>>>>>>>>>>> state=AgentStopped >> >>>>>>>>>>>> stopped=True >> >>>>>>>>>>>> >> >>>>>>>>>>>> Yet I can login to the web GUI fine. I guess it is not HA due >> >>>>>>>>>>>> to >> >>>>>>>>>>>> being >> >>>>>>>>>>>> in an unknown state currently? Does the hosted-engine-ha rpm >> >>>>>>>>>>>> need >> >>>>>>>>>>>> to >> >>>>>>>>>>>> be installed across all nodes in the cluster, btw? >> >>>>>>>>>>>> >> >>>>>>>>>>>> Thanks for the help, >> >>>>>>>>>>>> >> >>>>>>>>>>>> Cam >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Jenny Tokar >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On Thu, Jun 15, 2017 at 6:32 PM, cmc <iucounu(a)gmail.com&gt; >> >>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Hi, >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> I've migrated from a bare-metal engine to a hosted engine. >> >>>>>>>>>>>>>> There >> >>>>>>>>>>>>>> were >> >>>>>>>>>>>>>> no errors during the install, however, the hosted engine >> >>>>>>>>>>>>>> did not >> >>>>>>>>>>>>>> get >> >>>>>>>>>>>>>> started. I tried running: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> hosted-engine --status >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> on the host I deployed it on, and it returns nothing (exit >> >>>>>>>>>>>>>> code >> >>>>>>>>>>>>>> is 1 >> >>>>>>>>>>>>>> however). I could not ping it either. So I tried starting >> >>>>>>>>>>>>>> it via >> >>>>>>>>>>>>>> 'hosted-engine --vm-start' and it returned: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Virtual machine does not exist >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> But it then became available. I logged into it >> >>>>>>>>>>>>>> successfully. It >> >>>>>>>>>>>>>> is not >> >>>>>>>>>>>>>> in the list of VMs however. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Any ideas why the hosted-engine commands fail, and why it >> >>>>>>>>>>>>>> is not >> >>>>>>>>>>>>>> in >> >>>>>>>>>>>>>> the list of virtual machines? >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Thanks for any help, >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Cam >> >>>>>>>>>>>>>> _______________________________________________ >> >>>>>>>>>>>>>> Users mailing list >> >>>>>>>>>>>>>> Users(a)ovirt.org >> >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> _______________________________________________ >> >>>>>>>>> Users mailing list >> >>>>>>>>> Users(a)ovirt.org >> >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >> >>>>>> _______________________________________________ >> >>>>>> Users mailing list >> >>>>>> Users(a)ovirt.org >> >>>>>> http://lists.ovirt.org/mailman/listinfo/users >> >>>>> >> >>>>> >> > _______________________________________________ >> > Users mailing list >> > Users(a)ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/users >> > >> > >> >