Re: [ovirt-users] HostedEngine VM not visible, but running

Hi, yep, you have to restart the ovirt-ha-agent and ovirt-ha-broker services. The scheduling message just means that the host has score 0 or is not reporting score at all. Martin On Thu, Jun 29, 2017 at 1:33 PM, cmc <iucounu(a)gmail.com&gt; wrote: > Thanks Martin, do I have to restart anything? When I try to use the > 'migrate' operation, it complains that the other two hosts 'did not > satisfy internal filter HA because it is not a Hosted Engine host..' > (even though I reinstalled both these hosts with the 'deploy hosted > engine' option, which suggests that something needs restarting. Should > I worry about the sanlock errors, or will that be resolved by the > change in host_id? > > Kind regards, > > Cam > > On Thu, Jun 29, 2017 at 12:22 PM, Martin Sivak <msivak(a)redhat.com&gt; wrote: >> Change the ids so they are distinct. I need to check if there is a way >> to read the SPM ids from the engine as using the same numbers would be >> the best. >> >> Martin >> >> >> >> On Thu, Jun 29, 2017 at 12:46 PM, cmc <iucounu(a)gmail.com&gt; wrote: >>> Is there any way of recovering from this situation? I'd prefer to fix >>> the issue rather than re-deploy, but if there is no recovery path, I >>> could perhaps try re-deploying the hosted engine. In which case, would >>> the best option be to take a backup of the Hosted Engine, and then >>> shut it down, re-initialise the SAN partition (or use another >>> partition) and retry the deployment? Would it be better to use the >>> older backup from the bare metal engine that I originally used, or use >>> a backup from the Hosted Engine? I'm not sure if any VMs have been >>> added since switching to Hosted Engine. >>> >>> Unfortunately I have very little time left to get this working before >>> I have to hand it over for eval (by end of Friday). >>> >>> Here are some log snippets from the cluster that are current >>> >>> In /var/log/vdsm/vdsm.log on the host that has the Hosted Engine: >>> >>> 2017-06-29 10:50:15,071+0100 INFO (monitor/207221b) [storage.SANLock] >>> Acquiring host id for domain 207221b2-959b-426b-b945-18e1adfed62f (id: >>> 3) (clusterlock:282) >>> 2017-06-29 10:50:15,072+0100 ERROR (monitor/207221b) [storage.Monitor] >>> Error acquiring host id 3 for domain >>> 207221b2-959b-426b-b945-18e1adfed62f (monitor:558) >>> Traceback (most recent call last): >>> File "/usr/share/vdsm/storage/monitor.py", line 555, in _acquireHostId >>> self.domain.acquireHostId(self.hostId, async=True) >>> File "/usr/share/vdsm/storage/sd.py", line 790, in acquireHostId >>> self._manifest.acquireHostId(hostId, async) >>> File "/usr/share/vdsm/storage/sd.py", line 449, in acquireHostId >>> self._domainLock.acquireHostId(hostId, async) >>> File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", >>> line 297, in acquireHostId >>> raise se.AcquireHostIdFailure(self._sdUUID, e) >>> AcquireHostIdFailure: Cannot acquire host id: >>> ('207221b2-959b-426b-b945-18e1adfed62f', SanlockException(22, 'Sanlock >>> lockspace add failure', 'Invalid argument')) >>> >>> From /var/log/ovirt-hosted-engine-ha/agent.log on the same host: >>> >>> MainThread::ERROR::2017-06-19 >>> 13:30:50,592::hosted_engine::822::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_domain_monitor) >>> Failed to start monitoring domain >>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): timeout >>> during domain acquisition >>> MainThread::WARNING::2017-06-19 >>> 13:30:50,593::hosted_engine::469::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Error while monitoring engine: Failed to start monitoring domain >>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): timeout >>> during domain acquisition >>> MainThread::WARNING::2017-06-19 >>> 13:30:50,593::hosted_engine::472::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Unexpected error >>> Traceback (most recent call last): >>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >>> line 443, in start_monitoring >>> self._initialize_domain_monitor() >>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >>> line 823, in _initialize_domain_monitor >>> raise Exception(msg) >>> Exception: Failed to start monitoring domain >>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): timeout >>> during domain acquisition >>> MainThread::ERROR::2017-06-19 >>> 13:30:50,593::hosted_engine::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Shutting down the agent because of 3 failures in a row! >>> >>> From sanlock.log: >>> >>> 2017-06-29 11:17:06+0100 1194149 [2530]: add_lockspace >>> 207221b2-959b-426b-b945-18e1adfed62f:3:/dev/207221b2-959b-426b-b945-18e1adfed62f/ids:0 >>> conflicts with name of list1 s5 >>> 207221b2-959b-426b-b945-18e1adfed62f:1:/dev/207221b2-959b-426b-b945-18e1adfed62f/ids:0 >>> >>> From the two other hosts: >>> >>> host 2: >>> >>> vdsm.log >>> >>> 2017-06-29 10:53:47,755+0100 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] >>> Internal server error (__init__:570) >>> Traceback (most recent call last): >>> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line >>> 565, in _handle_request >>> res = method(**params) >>> File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line >>> 202, in _dynamicMethod >>> result = fn(*methodArgs) >>> File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolicies >>> io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() >>> File "/usr/share/vdsm/clientIF.py", line 448, in getAllVmIoTunePolicies >>> 'current_values': v.getIoTune()} >>> File "/usr/share/vdsm/virt/vm.py", line 2803, in getIoTune >>> result = self.getIoTuneResponse() >>> File "/usr/share/vdsm/virt/vm.py", line 2816, in getIoTuneResponse >>> res = self._dom.blockIoTune( >>> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line >>> 47, in __getattr__ >>> % self.vmid) >>> NotConnectedError: VM u'a79e6b0e-fff4-4cba-a02c-4c00be151300' was not >>> started yet or was shut down >>> >>> /var/log/ovirt-hosted-engine-ha/agent.log >>> >>> MainThread::INFO::2017-06-29 >>> 10:56:33,636::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:222610db-7880-4f4f-8559-a3635fd73555, >>> volUUID:c6e0d29b-eabf-4a09-a330-df54cfdd73f1 >>> MainThread::INFO::2017-06-29 >>> 10:56:33,926::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-06-29 >>> 10:56:33,938::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/blockSD/207221b2-959b-426b-b945-18e1adfed62f/images/222610db-7880-4f4f-8559-a3635fd73555/c6e0d29b-eabf-4a09-a330-df54cfdd73f1 >>> MainThread::INFO::2017-06-29 >>> 10:56:33,967::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-06-29 >>> 10:56:33,971::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-06-29 >>> 10:56:36,736::states::678::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) >>> Score is 0 due to unexpected vm shutdown at Thu Jun 29 10:53:59 2017 >>> MainThread::INFO::2017-06-29 >>> 10:56:36,736::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Current state EngineUnexpectedlyDown (score: 0) >>> MainThread::INFO::2017-06-29 >>> 10:56:46,772::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> >>> /var/log/messages: >>> >>> Jun 29 10:53:46 kvm-ldn-02 kernel: dd: sending ioctl 80306d02 to a partition! >>> >>> >>> host 1: >>> >>> /var/log/messages also in sanlock.log >>> >>> Jun 29 11:01:02 kvm-ldn-01 sanlock[2400]: 2017-06-29 11:01:02+0100 >>> 678325 [9132]: s4531 delta_acquire host_id 1 busy1 1 2 1193177 >>> 3d4ec963-8486-43a2-a7d9-afa82508f89f.kvm-ldn-03 >>> Jun 29 11:01:03 kvm-ldn-01 sanlock[2400]: 2017-06-29 11:01:03+0100 >>> 678326 [24159]: s4531 add_lockspace fail result -262 >>> >>> /var/log/ovirt-hosted-engine-ha/agent.log: >>> >>> MainThread::ERROR::2017-06-27 >>> 15:21:01,143::hosted_engine::822::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_domain_monitor) >>> Failed to start monitoring domain >>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): timeout >>> during domain acquisition >>> MainThread::WARNING::2017-06-27 >>> 15:21:01,144::hosted_engine::469::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Error while monitoring engine: Failed to start monitoring domain >>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): timeout >>> during domain acquisition >>> MainThread::WARNING::2017-06-27 >>> 15:21:01,144::hosted_engine::472::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Unexpected error >>> Traceback (most recent call last): >>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >>> line 443, in start_monitoring >>> self._initialize_domain_monitor() >>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >>> line 823, in _initialize_domain_monitor >>> raise Exception(msg) >>> Exception: Failed to start monitoring domain >>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): timeout >>> during domain acquisition >>> MainThread::ERROR::2017-06-27 >>> 15:21:01,144::hosted_engine::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Shutting down the agent because of 3 failures in a row! >>> MainThread::INFO::2017-06-27 >>> 15:21:06,717::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) >>> VDSM domain monitor status: PENDING >>> MainThread::INFO::2017-06-27 >>> 15:21:09,335::hosted_engine::776::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor) >>> Failed to stop monitoring domain >>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f): Storage domain is >>> member of pool: u'domain=207221b2-959b-426b-b945-18e1adfed62f' >>> MainThread::INFO::2017-06-27 >>> 15:21:09,339::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(run) >>> Agent shutting down >>> >>> >>> Thanks for any help, >>> >>> >>> Cam >>> >>> >>> On Wed, Jun 28, 2017 at 11:25 AM, cmc <iucounu(a)gmail.com&gt; wrote: >>>> Hi Martin, >>>> >>>> yes, on two of the machines they have the same host_id. The other has >>>> a different host_id. >>>> >>>> To update since yesterday: I reinstalled and deployed Hosted Engine on >>>> the other host (so all three hosts in the cluster now have it >>>> installed). The second one I deployed said it was able to host the >>>> engine (unlike the first I reinstalled), so I tried putting the host >>>> with the Hosted Engine on it into maintenance to see if it would >>>> migrate over. It managed to move all hosts but the Hosted Engine. And >>>> now the host that said it was able to host the engine says >>>> 'unavailable due to HA score'. The host that it was trying to move >>>> from is now in 'preparing for maintenance' for the last 12 hours. >>>> >>>> The summary is: >>>> >>>> kvm-ldn-01 - one of the original, pre-Hosted Engine hosts, reinstalled >>>> with 'Deploy Hosted Engine'. No icon saying it can host the Hosted >>>> Hngine, host_id of '2' in /etc/ovirt-hosted-engine/hosted-engine.conf. >>>> 'add_lockspace' fails in sanlock.log >>>> >>>> kvm-ldn-02 - the other host that was pre-existing before Hosted Engine >>>> was created. Reinstalled with 'Deploy Hosted Engine'. Had an icon >>>> saying that it was able to host the Hosted Engine, but after migration >>>> was attempted when putting kvm-ldn-03 into maintenance, it reports: >>>> 'unavailable due to HA score'. It has a host_id of '1' in >>>> /etc/ovirt-hosted-engine/hosted-engine.conf. No errors in sanlock.log >>>> >>>> kvm-ldn-03 - this was the host I deployed Hosted Engine on, which was >>>> not part of the original cluster. I restored the bare-metal engine >>>> backup in the Hosted Engine on this host when deploying it, without >>>> error. It currently has the Hosted Engine on it (as the only VM after >>>> I put that host into maintenance to test the HA of Hosted Engine). >>>> Sanlock log shows conflicts >>>> >>>> I will look through all the logs for any other errors. Please let me >>>> know if you need any logs or other clarification/information. >>>> >>>> Thanks, >>>> >>>> Campbell >>>> >>>> On Wed, Jun 28, 2017 at 9:25 AM, Martin Sivak <msivak(a)redhat.com&gt; wrote: >>>>> Hi, >>>>> >>>>> can you please check the contents of >>>>> /etc/ovirt-hosted-engine/hosted-engine.conf or >>>>> /etc/ovirt-hosted-engine-ha/agent.conf (I am not sure which one it is >>>>> right now) and search for host-id? >>>>> >>>>> Make sure the IDs are different. If they are not, then there is a bug somewhere. >>>>> >>>>> Martin >>>>> >>>>> On Tue, Jun 27, 2017 at 6:26 PM, cmc <iucounu(a)gmail.com&gt; wrote: >>>>>> I see this on the host it is trying to migrate in /var/log/sanlock: >>>>>> >>>>>> 2017-06-27 17:10:40+0100 527703 [2407]: s3528 lockspace >>>>>> 207221b2-959b-426b-b945-18e1adfed62f:1:/dev/207221b2-959b-426b-b945-18e1adfed62f/ids:0 >>>>>> 2017-06-27 17:13:00+0100 527843 [27446]: s3528 delta_acquire host_id 1 >>>>>> busy1 1 2 1042692 3d4ec963-8486-43a2-a7d9-afa82508f89f.kvm-ldn-03 >>>>>> 2017-06-27 17:13:01+0100 527844 [2407]: s3528 add_lockspace fail result -262 >>>>>> >>>>>> The sanlock service is running. Why would this occur? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> C >>>>>> >>>>>> On Tue, Jun 27, 2017 at 5:21 PM, cmc <iucounu(a)gmail.com&gt; wrote: >>>>>>> Hi Martin, >>>>>>> >>>>>>> Thanks for the reply. I have done this, and the deployment completed >>>>>>> without error. However, it still will not allow the Hosted Engine >>>>>>> migrate to another host. The >>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf got created ok on the host >>>>>>> I re-installed, but the ovirt-ha-broker.service, though it starts, >>>>>>> reports: >>>>>>> >>>>>>> --------------------8<------------------- >>>>>>> >>>>>>> Jun 27 14:58:26 kvm-ldn-01 systemd[1]: Starting oVirt Hosted Engine >>>>>>> High Availability Communications Broker... >>>>>>> Jun 27 14:58:27 kvm-ldn-01 ovirt-ha-broker[6101]: ovirt-ha-broker >>>>>>> ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR >>>>>>> Failed to read metadata from >>>>>>> /rhev/data-center/mnt/blockSD/207221b2-959b-426b-b945-18e1adfed62f/ha_agent/hosted-engine.metadata >>>>>>> Traceback (most >>>>>>> recent call last): >>>>>>> File >>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", >>>>>>> line 129, in get_raw_stats_for_service_type >>>>>>> f = >>>>>>> os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC) >>>>>>> OSError: [Errno 2] >>>>>>> No such file or directory: >>>>>>> '/rhev/data-center/mnt/blockSD/207221b2-959b-426b-b945-18e1adfed62f/ha_agent/hosted-engine.metadata' >>>>>>> >>>>>>> --------------------8<------------------- >>>>>>> >>>>>>> I checked the path, and it exists. I can run 'less -f' on it fine. The >>>>>>> perms are slightly different on the host that is running the VM vs the >>>>>>> one that is reporting errors (600 vs 660), ownership is vdsm:qemu. Is >>>>>>> this a san locking issue? >>>>>>> >>>>>>> Thanks for any help, >>>>>>> >>>>>>> Cam >>>>>>> >>>>>>> On Tue, Jun 27, 2017 at 1:41 PM, Martin Sivak <msivak(a)redhat.com&gt; wrote: >>>>>>>>> Should it be? It was not in the instructions for the migration from >>>>>>>>> bare-metal to Hosted VM >>>>>>>> >>>>>>>> The hosted engine will only migrate to hosts that have the services >>>>>>>> running. Please put one other host to maintenance and select Hosted >>>>>>>> engine action: DEPLOY in the reinstall dialog. >>>>>>>> >>>>>>>> Best regards >>>>>>>> >>>>>>>> Martin Sivak >>>>>>>> >>>>>>>> On Tue, Jun 27, 2017 at 1:23 PM, cmc <iucounu(a)gmail.com&gt; wrote: >>>>>>>>> I changed the 'os.other.devices.display.protocols.value.3.6 = >>>>>>>>> spice/qxl,vnc/cirrus,vnc/qxl' line to have the same display protocols >>>>>>>>> as 4 and the hosted engine now appears in the list of VMs. I am >>>>>>>>> guessing the compatibility version was causing it to use the 3.6 >>>>>>>>> version. However, I am still unable to migrate the engine VM to >>>>>>>>> another host. When I try putting the host it is currently on into >>>>>>>>> maintenance, it reports: >>>>>>>>> >>>>>>>>> Error while executing action: Cannot switch the Host(s) to Maintenance mode. >>>>>>>>> There are no available hosts capable of running the engine VM. >>>>>>>>> >>>>>>>>> Running 'hosted-engine --vm-status' still shows 'Engine status: >>>>>>>>> unknown stale-data'. >>>>>>>>> >>>>>>>>> The ovirt-ha-broker service is only running on one host. It was set to >>>>>>>>> 'disabled' in systemd. It won't start as there is no >>>>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf on the other two hosts. >>>>>>>>> Should it be? It was not in the instructions for the migration from >>>>>>>>> bare-metal to Hosted VM >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Cam >>>>>>>>> >>>>>>>>> On Thu, Jun 22, 2017 at 1:07 PM, cmc <iucounu(a)gmail.com&gt; wrote: >>>>>>>>>> Hi Tomas, >>>>>>>>>> >>>>>>>>>> So in my /usr/share/ovirt-engine/conf/osinfo-defaults.properties on my >>>>>>>>>> engine VM, I have: >>>>>>>>>> >>>>>>>>>> os.other.devices.display.protocols.value = spice/qxl,vnc/vga,vnc/qxl,vnc/cirrus >>>>>>>>>> os.other.devices.display.protocols.value.3.6 = spice/qxl,vnc/cirrus,vnc/qxl >>>>>>>>>> >>>>>>>>>> That seems to match - I assume since this is 4.1, the 3.6 should not apply >>>>>>>>>> >>>>>>>>>> Is there somewhere else I should be looking? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Cam >>>>>>>>>> >>>>>>>>>> On Thu, Jun 22, 2017 at 11:40 AM, Tomas Jelinek <tjelinek(a)redhat.com&gt; wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Jun 22, 2017 at 12:38 PM, Michal Skrivanek >>>>>>>>>>> <michal.skrivanek(a)redhat.com&gt; wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > On 22 Jun 2017, at 12:31, Martin Sivak <msivak(a)redhat.com&gt; wrote: >>>>>>>>>>>> > >>>>>>>>>>>> > Tomas, what fields are needed in a VM to pass the check that causes >>>>>>>>>>>> > the following error? >>>>>>>>>>>> > >>>>>>>>>>>> >>>>> WARN [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >>>>>>>>>>>> >>>>> (org.ovirt.thread.pool-6-thread-23) [] Validation of action >>>>>>>>>>>> >>>>> 'ImportVm' >>>>>>>>>>>> >>>>> failed for user SYSTEM. Reasons: VAR__ACTION__IMPORT >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> ,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS >>>>>>>>>>>> >>>>>>>>>>>> to match the OS and VM Display type;-) >>>>>>>>>>>> Configuration is in osinfo….e.g. if that is import from older releases on >>>>>>>>>>>> Linux this is typically caused by the cahgen of cirrus to vga for non-SPICE >>>>>>>>>>>> VMs >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> yep, the default supported combinations for 4.0+ is this: >>>>>>>>>>> os.other.devices.display.protocols.value = >>>>>>>>>>> spice/qxl,vnc/vga,vnc/qxl,vnc/cirrus >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > Thanks. >>>>>>>>>>>> > >>>>>>>>>>>> > On Thu, Jun 22, 2017 at 12:19 PM, cmc <iucounu(a)gmail.com&gt; wrote: >>>>>>>>>>>> >> Hi Martin, >>>>>>>>>>>> >> >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> just as a random comment, do you still have the database backup from >>>>>>>>>>>> >>> the bare metal -> VM attempt? It might be possible to just try again >>>>>>>>>>>> >>> using it. Or in the worst case.. update the offending value there >>>>>>>>>>>> >>> before restoring it to the new engine instance. >>>>>>>>>>>> >> >>>>>>>>>>>> >> I still have the backup. I'd rather do the latter, as re-running the >>>>>>>>>>>> >> HE deployment is quite lengthy and involved (I have to re-initialise >>>>>>>>>>>> >> the FC storage each time). Do you know what the offending value(s) >>>>>>>>>>>> >> would be? Would it be in the Postgres DB or in a config file >>>>>>>>>>>> >> somewhere? >>>>>>>>>>>> >> >>>>>>>>>>>> >> Cheers, >>>>>>>>>>>> >> >>>>>>>>>>>> >> Cam >>>>>>>>>>>> >> >>>>>>>>>>>> >>> Regards >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> Martin Sivak >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> On Thu, Jun 22, 2017 at 11:39 AM, cmc <iucounu(a)gmail.com&gt; wrote: >>>>>>>>>>>> >>>> Hi Yanir, >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> Thanks for the reply. >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>>> First of all, maybe a chain reaction of : >>>>>>>>>>>> >>>>> WARN [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >>>>>>>>>>>> >>>>> (org.ovirt.thread.pool-6-thread-23) [] Validation of action >>>>>>>>>>>> >>>>> 'ImportVm' >>>>>>>>>>>> >>>>> failed for user SYSTEM. Reasons: VAR__ACTION__IMPORT >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> ,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS >>>>>>>>>>>> >>>>> is causing the hosted engine vm not to be set up correctly and >>>>>>>>>>>> >>>>> further >>>>>>>>>>>> >>>>> actions were made when the hosted engine vm wasnt in a stable state. >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> As for now, are you trying to revert back to a previous/initial >>>>>>>>>>>> >>>>> state ? >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> I'm not trying to revert it to a previous state for now. This was a >>>>>>>>>>>> >>>> migration from a bare metal engine, and it didn't report any error >>>>>>>>>>>> >>>> during the migration. I'd had some problems on my first attempts at >>>>>>>>>>>> >>>> this migration, whereby it never completed (due to a proxy issue) but >>>>>>>>>>>> >>>> I managed to resolve this. Do you know of a way to get the Hosted >>>>>>>>>>>> >>>> Engine VM into a stable state, without rebuilding the entire cluster >>>>>>>>>>>> >>>> from scratch (since I have a lot of VMs on it)? >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> Thanks for any help. >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> Regards, >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> Cam >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>>> Regards, >>>>>>>>>>>> >>>>> Yanir >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> On Wed, Jun 21, 2017 at 4:32 PM, cmc <iucounu(a)gmail.com&gt; wrote: >>>>>>>>>>>> >>>>>> >>>>>>>>>>>> >>>>>> Hi Jenny/Martin, >>>>>>>>>>>> >>>>>> >>>>>>>>>>>> >>>>>> Any idea what I can do here? The hosted engine VM has no log on any >>>>>>>>>>>> >>>>>> host in /var/log/libvirt/qemu, and I fear that if I need to put the >>>>>>>>>>>> >>>>>> host into maintenance, e.g., to upgrade it that I created it on >>>>>>>>>>>> >>>>>> (which >>>>>>>>>>>> >>>>>> I think is hosting it), or if it fails for any reason, it won't get >>>>>>>>>>>> >>>>>> migrated to another host, and I will not be able to manage the >>>>>>>>>>>> >>>>>> cluster. It seems to be a very dangerous position to be in. >>>>>>>>>>>> >>>>>> >>>>>>>>>>>> >>>>>> Thanks, >>>>>>>>>>>> >>>>>> >>>>>>>>>>>> >>>>>> Cam >>>>>>>>>>>> >>>>>> >>>>>>>>>>>> >>>>>> On Wed, Jun 21, 2017 at 11:48 AM, cmc <iucounu(a)gmail.com&gt; wrote: >>>>>>>>>>>> >>>>>>> Thanks Martin. The hosts are all part of the same cluster. >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> I get these errors in the engine.log on the engine: >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> 2017-06-19 03:28:05,030Z WARN >>>>>>>>>>>> >>>>>>> [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >>>>>>>>>>>> >>>>>>> (org.ovirt.thread.pool-6-thread-23) [] Validation of action >>>>>>>>>>>> >>>>>>> 'ImportVm' >>>>>>>>>>>> >>>>>>> failed for user SYST >>>>>>>>>>>> >>>>>>> EM. Reasons: >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS >>>>>>>>>>>> >>>>>>> 2017-06-19 03:28:05,030Z INFO >>>>>>>>>>>> >>>>>>> [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >>>>>>>>>>>> >>>>>>> (org.ovirt.thread.pool-6-thread-23) [] Lock freed to object >>>>>>>>>>>> >>>>>>> 'EngineLock:{exclusiveLocks='[a >>>>>>>>>>>> >>>>>>> 79e6b0e-fff4-4cba-a02c-4c00be151300=<VM, >>>>>>>>>>>> >>>>>>> ACTION_TYPE_FAILED_VM_IS_BEING_IMPORTED$VmName HostedEngine>, >>>>>>>>>>>> >>>>>>> HostedEngine=<VM_NAME, ACTION_TYPE_FAILED_NAME_ALREADY_USED>]', >>>>>>>>>>>> >>>>>>> sharedLocks= >>>>>>>>>>>> >>>>>>> '[a79e6b0e-fff4-4cba-a02c-4c00be151300=<REMOTE_VM, >>>>>>>>>>>> >>>>>>> ACTION_TYPE_FAILED_VM_IS_BEING_IMPORTED$VmName HostedEngine>]'}' >>>>>>>>>>>> >>>>>>> 2017-06-19 03:28:05,030Z ERROR >>>>>>>>>>>> >>>>>>> [org.ovirt.engine.core.bll.HostedEngineImporter] >>>>>>>>>>>> >>>>>>> (org.ovirt.thread.pool-6-thread-23) [] Failed importing the Hosted >>>>>>>>>>>> >>>>>>> Engine VM >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> The sanlock.log reports conflicts on that same host, and a >>>>>>>>>>>> >>>>>>> different >>>>>>>>>>>> >>>>>>> error on the other hosts, not sure if they are related. >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> And this in the /var/log/ovirt-hosted-engine-ha/agent log on the >>>>>>>>>>>> >>>>>>> host >>>>>>>>>>>> >>>>>>> which I deployed the hosted engine VM on: >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> MainThread::ERROR::2017-06-19 >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> 13:09:49,743::ovf_store::124::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>>>>>>>>>>> >>>>>>> Unable to extract HEVM OVF >>>>>>>>>>>> >>>>>>> MainThread::ERROR::2017-06-19 >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> 13:09:49,743::config::445::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>>>>>>>>>>> >>>>>>> Failed extracting VM OVF from the OVF_STORE volume, falling back >>>>>>>>>>>> >>>>>>> to >>>>>>>>>>>> >>>>>>> initial vm.conf >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> I've seen some of these issues reported in bugzilla, but they were >>>>>>>>>>>> >>>>>>> for >>>>>>>>>>>> >>>>>>> older versions of oVirt (and appear to be resolved). >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> I will install that package on the other two hosts, for which I >>>>>>>>>>>> >>>>>>> will >>>>>>>>>>>> >>>>>>> put them in maintenance as vdsm is installed as an upgrade. I >>>>>>>>>>>> >>>>>>> guess >>>>>>>>>>>> >>>>>>> restarting vdsm is a good idea after that? >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> Thanks, >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> Campbell >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>> On Wed, Jun 21, 2017 at 10:51 AM, Martin Sivak <msivak(a)redhat.com&gt; >>>>>>>>>>>> >>>>>>> wrote: >>>>>>>>>>>> >>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> >>>>>>>> you do not have to install it on all hosts. But you should have >>>>>>>>>>>> >>>>>>>> more >>>>>>>>>>>> >>>>>>>> than one and ideally all hosted engine enabled nodes should >>>>>>>>>>>> >>>>>>>> belong to >>>>>>>>>>>> >>>>>>>> the same engine cluster. >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> >>>>>>>> Best regards >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> >>>>>>>> Martin Sivak >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> >>>>>>>> On Wed, Jun 21, 2017 at 11:29 AM, cmc <iucounu(a)gmail.com&gt; wrote: >>>>>>>>>>>> >>>>>>>>> Hi Jenny, >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> Does ovirt-hosted-engine-ha need to be installed across all >>>>>>>>>>>> >>>>>>>>> hosts? >>>>>>>>>>>> >>>>>>>>> Could that be the reason it is failing to see it properly? >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> Cam >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> On Mon, Jun 19, 2017 at 1:27 PM, cmc <iucounu(a)gmail.com&gt; wrote: >>>>>>>>>>>> >>>>>>>>>> Hi Jenny, >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> Logs are attached. I can see errors in there, but am unsure how >>>>>>>>>>>> >>>>>>>>>> they >>>>>>>>>>>> >>>>>>>>>> arose. >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> Campbell >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> On Mon, Jun 19, 2017 at 12:29 PM, Evgenia Tokar >>>>>>>>>>>> >>>>>>>>>> <etokar(a)redhat.com&gt; >>>>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>> From the output it looks like the agent is down, try starting >>>>>>>>>>>> >>>>>>>>>>> it by >>>>>>>>>>>> >>>>>>>>>>> running: >>>>>>>>>>>> >>>>>>>>>>> systemctl start ovirt-ha-agent. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> The engine is supposed to see the hosted engine storage domain >>>>>>>>>>>> >>>>>>>>>>> and >>>>>>>>>>>> >>>>>>>>>>> import it >>>>>>>>>>>> >>>>>>>>>>> to the system, then it should import the hosted engine vm. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> Can you attach the agent log from the host >>>>>>>>>>>> >>>>>>>>>>> (/var/log/ovirt-hosted-engine-ha/agent.log) >>>>>>>>>>>> >>>>>>>>>>> and the engine log from the engine vm >>>>>>>>>>>> >>>>>>>>>>> (/var/log/ovirt-engine/engine.log)? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>> Jenny >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> On Mon, Jun 19, 2017 at 12:41 PM, cmc <iucounu(a)gmail.com&gt; >>>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Jenny, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> What version are you running? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 4.1.2.2-1.el7.centos >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> For the hosted engine vm to be imported and displayed in the >>>>>>>>>>>> >>>>>>>>>>>>> engine, you >>>>>>>>>>>> >>>>>>>>>>>>> must first create a master storage domain. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> To provide a bit more detail: this was a migration of a >>>>>>>>>>>> >>>>>>>>>>>> bare-metal >>>>>>>>>>>> >>>>>>>>>>>> engine in an existing cluster to a hosted engine VM for that >>>>>>>>>>>> >>>>>>>>>>>> cluster. >>>>>>>>>>>> >>>>>>>>>>>> As part of this migration, I built an entirely new host and >>>>>>>>>>>> >>>>>>>>>>>> ran >>>>>>>>>>>> >>>>>>>>>>>> 'hosted-engine --deploy' (followed these instructions: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> http://www.ovirt.org/documentation/self-hosted/chap-Migrating_from_Bare_M...). >>>>>>>>>>>> >>>>>>>>>>>> I restored the backup from the engine and it completed >>>>>>>>>>>> >>>>>>>>>>>> without any >>>>>>>>>>>> >>>>>>>>>>>> errors. I didn't see any instructions regarding a master >>>>>>>>>>>> >>>>>>>>>>>> storage >>>>>>>>>>>> >>>>>>>>>>>> domain in the page above. The cluster has two existing master >>>>>>>>>>>> >>>>>>>>>>>> storage >>>>>>>>>>>> >>>>>>>>>>>> domains, one is fibre channel, which is up, and one ISO >>>>>>>>>>>> >>>>>>>>>>>> domain, >>>>>>>>>>>> >>>>>>>>>>>> which >>>>>>>>>>>> >>>>>>>>>>>> is currently offline. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> What do you mean the hosted engine commands are failing? >>>>>>>>>>>> >>>>>>>>>>>>> What >>>>>>>>>>>> >>>>>>>>>>>>> happens >>>>>>>>>>>> >>>>>>>>>>>>> when >>>>>>>>>>>> >>>>>>>>>>>>> you run hosted-engine --vm-status now? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Interestingly, whereas when I ran it before, it exited with >>>>>>>>>>>> >>>>>>>>>>>> no >>>>>>>>>>>> >>>>>>>>>>>> output >>>>>>>>>>>> >>>>>>>>>>>> and a return code of '1', it now reports: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> --== Host 1 status ==-- >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> conf_on_shared_storage : True >>>>>>>>>>>> >>>>>>>>>>>> Status up-to-date : False >>>>>>>>>>>> >>>>>>>>>>>> Hostname : >>>>>>>>>>>> >>>>>>>>>>>> kvm-ldn-03.ldn.fscfc.co.uk >>>>>>>>>>>> >>>>>>>>>>>> Host ID : 1 >>>>>>>>>>>> >>>>>>>>>>>> Engine status : unknown stale-data >>>>>>>>>>>> >>>>>>>>>>>> Score : 0 >>>>>>>>>>>> >>>>>>>>>>>> stopped : True >>>>>>>>>>>> >>>>>>>>>>>> Local maintenance : False >>>>>>>>>>>> >>>>>>>>>>>> crc32 : 0217f07b >>>>>>>>>>>> >>>>>>>>>>>> local_conf_timestamp : 2911 >>>>>>>>>>>> >>>>>>>>>>>> Host timestamp : 2897 >>>>>>>>>>>> >>>>>>>>>>>> Extra metadata (valid at timestamp): >>>>>>>>>>>> >>>>>>>>>>>> metadata_parse_version=1 >>>>>>>>>>>> >>>>>>>>>>>> metadata_feature_version=1 >>>>>>>>>>>> >>>>>>>>>>>> timestamp=2897 (Thu Jun 15 16:22:54 2017) >>>>>>>>>>>> >>>>>>>>>>>> host-id=1 >>>>>>>>>>>> >>>>>>>>>>>> score=0 >>>>>>>>>>>> >>>>>>>>>>>> vm_conf_refresh_time=2911 (Thu Jun 15 16:23:08 2017) >>>>>>>>>>>> >>>>>>>>>>>> conf_on_shared_storage=True >>>>>>>>>>>> >>>>>>>>>>>> maintenance=False >>>>>>>>>>>> >>>>>>>>>>>> state=AgentStopped >>>>>>>>>>>> >>>>>>>>>>>> stopped=True >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Yet I can login to the web GUI fine. I guess it is not HA due >>>>>>>>>>>> >>>>>>>>>>>> to >>>>>>>>>>>> >>>>>>>>>>>> being >>>>>>>>>>>> >>>>>>>>>>>> in an unknown state currently? Does the hosted-engine-ha rpm >>>>>>>>>>>> >>>>>>>>>>>> need >>>>>>>>>>>> >>>>>>>>>>>> to >>>>>>>>>>>> >>>>>>>>>>>> be installed across all nodes in the cluster, btw? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks for the help, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Cam >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Jenny Tokar >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jun 15, 2017 at 6:32 PM, cmc <iucounu(a)gmail.com&gt; >>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> I've migrated from a bare-metal engine to a hosted engine. >>>>>>>>>>>> >>>>>>>>>>>>>> There >>>>>>>>>>>> >>>>>>>>>>>>>> were >>>>>>>>>>>> >>>>>>>>>>>>>> no errors during the install, however, the hosted engine >>>>>>>>>>>> >>>>>>>>>>>>>> did not >>>>>>>>>>>> >>>>>>>>>>>>>> get >>>>>>>>>>>> >>>>>>>>>>>>>> started. I tried running: >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> hosted-engine --status >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> on the host I deployed it on, and it returns nothing (exit >>>>>>>>>>>> >>>>>>>>>>>>>> code >>>>>>>>>>>> >>>>>>>>>>>>>> is 1 >>>>>>>>>>>> >>>>>>>>>>>>>> however). I could not ping it either. So I tried starting >>>>>>>>>>>> >>>>>>>>>>>>>> it via >>>>>>>>>>>> >>>>>>>>>>>>>> 'hosted-engine --vm-start' and it returned: >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> Virtual machine does not exist >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> But it then became available. I logged into it >>>>>>>>>>>> >>>>>>>>>>>>>> successfully. It >>>>>>>>>>>> >>>>>>>>>>>>>> is not >>>>>>>>>>>> >>>>>>>>>>>>>> in the list of VMs however. >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> Any ideas why the hosted-engine commands fail, and why it >>>>>>>>>>>> >>>>>>>>>>>>>> is not >>>>>>>>>>>> >>>>>>>>>>>>>> in >>>>>>>>>>>> >>>>>>>>>>>>>> the list of virtual machines? >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for any help, >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> Cam >>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> >>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>> >>>>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>> >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>>>>> >>>>>>>>> Users mailing list >>>>>>>>>>>> >>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>> >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>> >>>>>> _______________________________________________ >>>>>>>>>>>> >>>>>> Users mailing list >>>>>>>>>>>> >>>>>> Users(a)ovirt.org >>>>>>>>>>>> >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> >>>>>>>>>>>> > _______________________________________________ >>>>>>>>>>>> > Users mailing list >>>>>>>>>>>> > Users(a)ovirt.org >>>>>>>>>>>> > http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>