Re: [ovirt-users] HostedEngine VM not visible, but running

On Thu, Jun 22, 2017 at 12:38 PM, Michal Skrivanek <michal.skrivanek(a)redhat.com&gt; wrote: > > > > On 22 Jun 2017, at 12:31, Martin Sivak <msivak(a)redhat.com&gt; wrote: > > > > Tomas, what fields are needed in a VM to pass the check that causes > > the following error? > > > >>>>> WARN [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] > >>>>> (org.ovirt.thread.pool-6-thread-23) [] Validation of action > >>>>> 'ImportVm' > >>>>> failed for user SYSTEM. Reasons: VAR__ACTION__IMPORT > >>>>> > >>>>> ,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS > > to match the OS and VM Display type;-) > Configuration is in osinfo….e.g. if that is import from older releases on > Linux this is typically caused by the cahgen of cirrus to vga for non-SPICE > VMs yep, the default supported combinations for 4.0+ is this: os.other.devices.display.protocols.value = spice/qxl,vnc/vga,vnc/qxl,vnc/cirrus > > > > > > Thanks. > > > > On Thu, Jun 22, 2017 at 12:19 PM, cmc <iucounu(a)gmail.com&gt; wrote: > >> Hi Martin, > >> > >>> > >>> just as a random comment, do you still have the database backup from > >>> the bare metal -> VM attempt? It might be possible to just try again > >>> using it. Or in the worst case.. update the offending value there > >>> before restoring it to the new engine instance. > >> > >> I still have the backup. I'd rather do the latter, as re-running the > >> HE deployment is quite lengthy and involved (I have to re-initialise > >> the FC storage each time). Do you know what the offending value(s) > >> would be? Would it be in the Postgres DB or in a config file > >> somewhere? > >> > >> Cheers, > >> > >> Cam > >> > >>> Regards > >>> > >>> Martin Sivak > >>> > >>> On Thu, Jun 22, 2017 at 11:39 AM, cmc <iucounu(a)gmail.com&gt; wrote: > >>>> Hi Yanir, > >>>> > >>>> Thanks for the reply. > >>>> > >>>>> First of all, maybe a chain reaction of : > >>>>> WARN [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] > >>>>> (org.ovirt.thread.pool-6-thread-23) [] Validation of action > >>>>> 'ImportVm' > >>>>> failed for user SYSTEM. Reasons: VAR__ACTION__IMPORT > >>>>> > >>>>> ,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS > >>>>> is causing the hosted engine vm not to be set up correctly and > >>>>> further > >>>>> actions were made when the hosted engine vm wasnt in a stable state. > >>>>> > >>>>> As for now, are you trying to revert back to a previous/initial > >>>>> state ? > >>>> > >>>> I'm not trying to revert it to a previous state for now. This was a > >>>> migration from a bare metal engine, and it didn't report any error > >>>> during the migration. I'd had some problems on my first attempts at > >>>> this migration, whereby it never completed (due to a proxy issue) but > >>>> I managed to resolve this. Do you know of a way to get the Hosted > >>>> Engine VM into a stable state, without rebuilding the entire cluster > >>>> from scratch (since I have a lot of VMs on it)? > >>>> > >>>> Thanks for any help. > >>>> > >>>> Regards, > >>>> > >>>> Cam > >>>> > >>>>> Regards, > >>>>> Yanir > >>>>> > >>>>> On Wed, Jun 21, 2017 at 4:32 PM, cmc <iucounu(a)gmail.com&gt; wrote: > >>>>>> > >>>>>> Hi Jenny/Martin, > >>>>>> > >>>>>> Any idea what I can do here? The hosted engine VM has no log on any > >>>>>> host in /var/log/libvirt/qemu, and I fear that if I need to put the > >>>>>> host into maintenance, e.g., to upgrade it that I created it on > >>>>>> (which > >>>>>> I think is hosting it), or if it fails for any reason, it won't get > >>>>>> migrated to another host, and I will not be able to manage the > >>>>>> cluster. It seems to be a very dangerous position to be in. > >>>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> Cam > >>>>>> > >>>>>> On Wed, Jun 21, 2017 at 11:48 AM, cmc <iucounu(a)gmail.com&gt; wrote: > >>>>>>> Thanks Martin. The hosts are all part of the same cluster. > >>>>>>> > >>>>>>> I get these errors in the engine.log on the engine: > >>>>>>> > >>>>>>> 2017-06-19 03:28:05,030Z WARN > >>>>>>> [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] > >>>>>>> (org.ovirt.thread.pool-6-thread-23) [] Validation of action > >>>>>>> 'ImportVm' > >>>>>>> failed for user SYST > >>>>>>> EM. Reasons: > >>>>>>> > >>>>>>> VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS > >>>>>>> 2017-06-19 03:28:05,030Z INFO > >>>>>>> [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] > >>>>>>> (org.ovirt.thread.pool-6-thread-23) [] Lock freed to object > >>>>>>> 'EngineLock:{exclusiveLocks='[a > >>>>>>> 79e6b0e-fff4-4cba-a02c-4c00be151300=<VM, > >>>>>>> ACTION_TYPE_FAILED_VM_IS_BEING_IMPORTED$VmName HostedEngine>, > >>>>>>> HostedEngine=<VM_NAME, ACTION_TYPE_FAILED_NAME_ALREADY_USED>]', > >>>>>>> sharedLocks= > >>>>>>> '[a79e6b0e-fff4-4cba-a02c-4c00be151300=<REMOTE_VM, > >>>>>>> ACTION_TYPE_FAILED_VM_IS_BEING_IMPORTED$VmName HostedEngine>]'}' > >>>>>>> 2017-06-19 03:28:05,030Z ERROR > >>>>>>> [org.ovirt.engine.core.bll.HostedEngineImporter] > >>>>>>> (org.ovirt.thread.pool-6-thread-23) [] Failed importing the Hosted > >>>>>>> Engine VM > >>>>>>> > >>>>>>> The sanlock.log reports conflicts on that same host, and a > >>>>>>> different > >>>>>>> error on the other hosts, not sure if they are related. > >>>>>>> > >>>>>>> And this in the /var/log/ovirt-hosted-engine-ha/agent log on the > >>>>>>> host > >>>>>>> which I deployed the hosted engine VM on: > >>>>>>> > >>>>>>> MainThread::ERROR::2017-06-19 > >>>>>>> > >>>>>>> > >>>>>>> 13:09:49,743::ovf_store::124::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) > >>>>>>> Unable to extract HEVM OVF > >>>>>>> MainThread::ERROR::2017-06-19 > >>>>>>> > >>>>>>> > >>>>>>> 13:09:49,743::config::445::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) > >>>>>>> Failed extracting VM OVF from the OVF_STORE volume, falling back > >>>>>>> to > >>>>>>> initial vm.conf > >>>>>>> > >>>>>>> I've seen some of these issues reported in bugzilla, but they were > >>>>>>> for > >>>>>>> older versions of oVirt (and appear to be resolved). > >>>>>>> > >>>>>>> I will install that package on the other two hosts, for which I > >>>>>>> will > >>>>>>> put them in maintenance as vdsm is installed as an upgrade. I > >>>>>>> guess > >>>>>>> restarting vdsm is a good idea after that? > >>>>>>> > >>>>>>> Thanks, > >>>>>>> > >>>>>>> Campbell > >>>>>>> > >>>>>>> On Wed, Jun 21, 2017 at 10:51 AM, Martin Sivak <msivak(a)redhat.com&gt; > >>>>>>> wrote: > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> you do not have to install it on all hosts. But you should have > >>>>>>>> more > >>>>>>>> than one and ideally all hosted engine enabled nodes should > >>>>>>>> belong to > >>>>>>>> the same engine cluster. > >>>>>>>> > >>>>>>>> Best regards > >>>>>>>> > >>>>>>>> Martin Sivak > >>>>>>>> > >>>>>>>> On Wed, Jun 21, 2017 at 11:29 AM, cmc <iucounu(a)gmail.com&gt; wrote: > >>>>>>>>> Hi Jenny, > >>>>>>>>> > >>>>>>>>> Does ovirt-hosted-engine-ha need to be installed across all > >>>>>>>>> hosts? > >>>>>>>>> Could that be the reason it is failing to see it properly? > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> > >>>>>>>>> Cam > >>>>>>>>> > >>>>>>>>> On Mon, Jun 19, 2017 at 1:27 PM, cmc <iucounu(a)gmail.com&gt; wrote: > >>>>>>>>>> Hi Jenny, > >>>>>>>>>> > >>>>>>>>>> Logs are attached. I can see errors in there, but am unsure how > >>>>>>>>>> they > >>>>>>>>>> arose. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> > >>>>>>>>>> Campbell > >>>>>>>>>> > >>>>>>>>>> On Mon, Jun 19, 2017 at 12:29 PM, Evgenia Tokar > >>>>>>>>>> <etokar(a)redhat.com&gt; > >>>>>>>>>> wrote: > >>>>>>>>>>> From the output it looks like the agent is down, try starting > >>>>>>>>>>> it by > >>>>>>>>>>> running: > >>>>>>>>>>> systemctl start ovirt-ha-agent. > >>>>>>>>>>> > >>>>>>>>>>> The engine is supposed to see the hosted engine storage domain > >>>>>>>>>>> and > >>>>>>>>>>> import it > >>>>>>>>>>> to the system, then it should import the hosted engine vm. > >>>>>>>>>>> > >>>>>>>>>>> Can you attach the agent log from the host > >>>>>>>>>>> (/var/log/ovirt-hosted-engine-ha/agent.log) > >>>>>>>>>>> and the engine log from the engine vm > >>>>>>>>>>> (/var/log/ovirt-engine/engine.log)? > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Jenny > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Mon, Jun 19, 2017 at 12:41 PM, cmc <iucounu(a)gmail.com&gt; > >>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Hi Jenny, > >>>>>>>>>>>> > >>>>>>>>>>>>> What version are you running? > >>>>>>>>>>>> > >>>>>>>>>>>> 4.1.2.2-1.el7.centos > >>>>>>>>>>>> > >>>>>>>>>>>>> For the hosted engine vm to be imported and displayed in the > >>>>>>>>>>>>> engine, you > >>>>>>>>>>>>> must first create a master storage domain. > >>>>>>>>>>>> > >>>>>>>>>>>> To provide a bit more detail: this was a migration of a > >>>>>>>>>>>> bare-metal > >>>>>>>>>>>> engine in an existing cluster to a hosted engine VM for that > >>>>>>>>>>>> cluster. > >>>>>>>>>>>> As part of this migration, I built an entirely new host and > >>>>>>>>>>>> ran > >>>>>>>>>>>> 'hosted-engine --deploy' (followed these instructions: > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> http://www.ovirt.org/documentation/self-hosted/chap-Migrating_from_Bare_M...). > >>>>>>>>>>>> I restored the backup from the engine and it completed > >>>>>>>>>>>> without any > >>>>>>>>>>>> errors. I didn't see any instructions regarding a master > >>>>>>>>>>>> storage > >>>>>>>>>>>> domain in the page above. The cluster has two existing master > >>>>>>>>>>>> storage > >>>>>>>>>>>> domains, one is fibre channel, which is up, and one ISO > >>>>>>>>>>>> domain, > >>>>>>>>>>>> which > >>>>>>>>>>>> is currently offline. > >>>>>>>>>>>> > >>>>>>>>>>>>> What do you mean the hosted engine commands are failing? > >>>>>>>>>>>>> What > >>>>>>>>>>>>> happens > >>>>>>>>>>>>> when > >>>>>>>>>>>>> you run hosted-engine --vm-status now? > >>>>>>>>>>>> > >>>>>>>>>>>> Interestingly, whereas when I ran it before, it exited with > >>>>>>>>>>>> no > >>>>>>>>>>>> output > >>>>>>>>>>>> and a return code of '1', it now reports: > >>>>>>>>>>>> > >>>>>>>>>>>> --== Host 1 status ==-- > >>>>>>>>>>>> > >>>>>>>>>>>> conf_on_shared_storage : True > >>>>>>>>>>>> Status up-to-date : False > >>>>>>>>>>>> Hostname : > >>>>>>>>>>>> kvm-ldn-03.ldn.fscfc.co.uk > >>>>>>>>>>>> Host ID : 1 > >>>>>>>>>>>> Engine status : unknown stale-data > >>>>>>>>>>>> Score : 0 > >>>>>>>>>>>> stopped : True > >>>>>>>>>>>> Local maintenance : False > >>>>>>>>>>>> crc32 : 0217f07b > >>>>>>>>>>>> local_conf_timestamp : 2911 > >>>>>>>>>>>> Host timestamp : 2897 > >>>>>>>>>>>> Extra metadata (valid at timestamp): > >>>>>>>>>>>> metadata_parse_version=1 > >>>>>>>>>>>> metadata_feature_version=1 > >>>>>>>>>>>> timestamp=2897 (Thu Jun 15 16:22:54 2017) > >>>>>>>>>>>> host-id=1 > >>>>>>>>>>>> score=0 > >>>>>>>>>>>> vm_conf_refresh_time=2911 (Thu Jun 15 16:23:08 2017) > >>>>>>>>>>>> conf_on_shared_storage=True > >>>>>>>>>>>> maintenance=False > >>>>>>>>>>>> state=AgentStopped > >>>>>>>>>>>> stopped=True > >>>>>>>>>>>> > >>>>>>>>>>>> Yet I can login to the web GUI fine. I guess it is not HA due > >>>>>>>>>>>> to > >>>>>>>>>>>> being > >>>>>>>>>>>> in an unknown state currently? Does the hosted-engine-ha rpm > >>>>>>>>>>>> need > >>>>>>>>>>>> to > >>>>>>>>>>>> be installed across all nodes in the cluster, btw? > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks for the help, > >>>>>>>>>>>> > >>>>>>>>>>>> Cam > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Jenny Tokar > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, Jun 15, 2017 at 6:32 PM, cmc <iucounu(a)gmail.com&gt; > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I've migrated from a bare-metal engine to a hosted engine. > >>>>>>>>>>>>>> There > >>>>>>>>>>>>>> were > >>>>>>>>>>>>>> no errors during the install, however, the hosted engine > >>>>>>>>>>>>>> did not > >>>>>>>>>>>>>> get > >>>>>>>>>>>>>> started. I tried running: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> hosted-engine --status > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> on the host I deployed it on, and it returns nothing (exit > >>>>>>>>>>>>>> code > >>>>>>>>>>>>>> is 1 > >>>>>>>>>>>>>> however). I could not ping it either. So I tried starting > >>>>>>>>>>>>>> it via > >>>>>>>>>>>>>> 'hosted-engine --vm-start' and it returned: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Virtual machine does not exist > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> But it then became available. I logged into it > >>>>>>>>>>>>>> successfully. It > >>>>>>>>>>>>>> is not > >>>>>>>>>>>>>> in the list of VMs however. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Any ideas why the hosted-engine commands fail, and why it > >>>>>>>>>>>>>> is not > >>>>>>>>>>>>>> in > >>>>>>>>>>>>>> the list of virtual machines? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks for any help, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Cam > >>>>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>>>> Users mailing list > >>>>>>>>>>>>>> Users(a)ovirt.org > >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> Users mailing list > >>>>>>>>> Users(a)ovirt.org > >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>>> _______________________________________________ > >>>>>> Users mailing list > >>>>>> Users(a)ovirt.org > >>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>> > >>>>> > > _______________________________________________ > > Users mailing list > > Users(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > > > > >