July 2019 - Users - oVirt List Archives

Ovirt-engine-ha cannot to see live status of Hosted Engine
by asm＠pioner.kz 01 Feb '20

01 Feb '20

Good day for all. I have some issues with Ovirt 4.2.6. But now the main this of it: I have two Centos 7 Nodes with same config and last Ovirt 4.2.6 with Hostedengine with disk on NFS storage. Also some of virtual machines working good. But, when HostedEngine running on one node (srv02.local) everything is fine. After migrating to another node (srv00.local), i see that agent cannot to check livelinness of HostedEngine. After few minutes HostedEngine going to reboot and after some time i see some situation. After migration to another node (srv00.local) all looks OK. hosted-engine --vm-status commang when HosterEngine on srv00 node: --== Host 1 status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : srv02.local Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down_unexpected", "detail": "unknown"} Score : 0 stopped : False Local maintenance : False crc32 : ecc7ad2d local_conf_timestamp : 78328 Host timestamp : 78328 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=78328 (Tue Sep 18 12:44:18 2018) host-id=1 score=0 vm_conf_refresh_time=78328 (Tue Sep 18 12:44:18 2018) conf_on_shared_storage=True maintenance=False state=EngineUnexpectedlyDown stopped=False timeout=Fri Jan 2 03:49:58 1970 --== Host 2 status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : srv00.local Host ID : 2 Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"} Score : 3400 stopped : False Local maintenance : False crc32 : 1d62b106 local_conf_timestamp : 326288 Host timestamp : 326288 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=326288 (Tue Sep 18 12:44:21 2018) host-id=2 score=3400 vm_conf_refresh_time=326288 (Tue Sep 18 12:44:21 2018) conf_on_shared_storage=True maintenance=False state=EngineStarting stopped=False Log agent.log from srv00.local: MainThread::INFO::2018-09-18 12:40:51,749::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE ngine::(consume) VM is powering up.. MainThread::INFO::2018-09-18 12:40:52,052::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine. HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400) MainThread::INFO::2018-09-18 12:41:01,066::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE ngine::(consume) VM is powering up.. MainThread::INFO::2018-09-18 12:41:01,374::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine. HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400) MainThread::INFO::2018-09-18 12:41:11,393::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine. HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2018-09-18 12:41:11,393::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine. HostedEngine::(refresh) Host srv02.local.pioner.kz (id 1): {'conf_on_shared_storage': True, 'extra': 'meta data_parse_version=1\nmetadata_feature_version=1\ntimestamp=78128 (Tue Sep 18 12:40:58 2018)\nhost-id=1\ns core=0\nvm_conf_refresh_time=78128 (Tue Sep 18 12:40:58 2018)\nconf_on_shared_storage=True\nmaintenance=Fa lse\nstate=EngineUnexpectedlyDown\nstopped=False\ntimeout=Fri Jan 2 03:49:58 1970\n', 'hostname': 'srv02. local.pioner.kz', 'alive': True, 'host-id': 1, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down_unexpected', 'detail': 'unknown'}, 'score': 0, 'stopped': False, 'maintenance ': False, 'crc32': 'e18e3f22', 'local_conf_timestamp': 78128, 'host-ts': 78128} MainThread::INFO::2018-09-18 12:41:11,393::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine. HostedEngine::(refresh) Local (id 2): {'engine-health': {'reason': 'failed liveliness check', 'health': 'b ad', 'vm': 'up', 'detail': 'Up'}, 'bridge': True, 'mem-free': 12763.0, 'maintenance': False, 'cpu-load': 0 .0364, 'gateway': 1.0, 'storage-domain': True} MainThread::INFO::2018-09-18 12:41:11,393::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE ngine::(consume) VM is powering up.. MainThread::INFO::2018-09-18 12:41:11,703::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine. HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400) MainThread::INFO::2018-09-18 12:41:21,716::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE ngine::(consume) VM is powering up.. MainThread::INFO::2018-09-18 12:41:22,020::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine. HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400) MainThread::INFO::2018-09-18 12:41:31,033::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE ngine::(consume) VM is powering up.. MainThread::INFO::2018-09-18 12:41:31,344::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine. HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400) As we can see, agent thinking that HostedEngine just in powering up mode. I cannot to do anythink with it. I allready reinstalled many times srv00 node without success. One time i even has to uninstall ovirt* and vdsm* software. Also here one interesting point, after installing just "yum install http://resources.ovirt.org/pub/yum-repo/ovirt-release42.rpm" on this node i try to install this node from engine web interface with "Deploy" action. But, installation was unsuccesfull, before i didnt install ovirt-hosted-engine-ha on this node. I dont see in documentation that its need bofore installation of new hosts. But this is for information and checking. After installing ovirt-hosted-engine-ha node was installed with HostedEngine support. But the main issue not changed. Thanks in advance for help. BR, Alexandr

3 5

Hyperconverged setup - storage architecture - scaling
by Leo David 10 Jan '20

10 Jan '20

Hello Everyone, Reading through the document: "Red Hat Hyperconverged Infrastructure for Virtualization 1.5 Automating RHHI for Virtualization deployment" Regarding storage scaling, i see the following statements: *2.7. SCALINGRed Hat Hyperconverged Infrastructure for Virtualization is supported for one node, and for clusters of 3, 6, 9, and 12 nodes.The initial deployment is either 1 or 3 nodes.There are two supported methods of horizontally scaling Red Hat Hyperconverged Infrastructure for Virtualization:* *1 Add new hyperconverged nodes to the cluster, in sets of three, up to the maximum of 12 hyperconverged nodes.* *2 Create new Gluster volumes using new disks on existing hyperconverged nodes.You cannot create a volume that spans more than 3 nodes, or expand an existing volume so that it spans across more than 3 nodes at a time* *2.9.1. Prerequisites for geo-replicationBe aware of the following requirements and limitations when configuring geo-replication:One geo-replicated volume onlyRed Hat Hyperconverged Infrastructure for Virtualization (RHHI for Virtualization) supports only one geo-replicated volume. Red Hat recommends backing up the volume that stores the data of your virtual machines, as this is usually contains the most valuable data.* ------ Also in oVirtEngine UI, when I add a brick to an existing volume i get the following warning: *"Expanding gluster volume in a hyper-converged setup is not recommended as it could lead to degraded performance. To expand storage for cluster, it is advised to add additional gluster volumes." * Those things are raising a couple of questions that maybe for some for you guys are easy to answer, but for me it creates a bit of confusion... I am also referring to RedHat product documentation, because I treat oVirt as production-ready as RHHI is. *1*. Is there any reason for not going to distributed-replicated volumes ( ie: spread one volume across 6,9, or 12 nodes ) ? - ie: is recomanded that in a 9 nodes scenario I should have 3 separated volumes, but how should I deal with the folowing question *2.* If only one geo-replicated volume can be configured, how should I deal with 2nd and 3rd volume replication for disaster recovery *3.* If the limit of hosts per datacenter is 250, then (in theory ) the recomended way in reaching this treshold would be to create 20 separated oVirt logical clusters with 12 nodes per each ( and datacenter managed from one ha-engine ) ? *4.* In present, I have the folowing one 9 nodes cluster , all hosts contributing with 2 disks each to a single replica 3 distributed replicated volume. They where added to the volume in the following order: node1 - disk1 node2 - disk1 ...... node9 - disk1 node1 - disk2 node2 - disk2 ...... node9 - disk2 At the moment, the volume is arbitrated, but I intend to go for full distributed replica 3. Is this a bad setup ? Why ? It oviously brakes the redhat recommended rules... Is there anyone so kind to discuss on these things ? Thank you very much ! Leo -- Best regards, Leo David -- Best regards, Leo David

3 5

disk locked after export as OVA
by adam_xu＠adagene.com.cn 15 Oct '19

15 Oct '19

Hello, everyone. I tried to export a vm as OVA. I got a error in webui: VDSM ovirt1.ntbaobei.com command HSMGetAllTasksStatusesVDS failed: Volume Group not big enough: (u'Not enough free extents for extending LV d9f2378f-92f0-4bc1-96a8-20a0f2c575cb/c515a3f9-0590-4ebe-81c5-9d0993f1fec9 (free=848, needed=872)',) seems no enough space in the storage domain. I deleted the vm which i wanted to export as OVA before, but I saw a disk with id 8140d2e7-6908-438e-a646-23e58a33913e left in the storage and the status is locked. here's some log in the engine.log: 2018-09-17 14:27:03,467+08 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (EE-ManagedThreadFactory-engine-Thread-107849) [a407f6a9-445d-4ccc-b998-f9dfc6dc67ec] START, CopyImageVDSCommand( CopyImageVDSCommandParameters:{storagePoolId='79432500-ad45-11e8-98f3-00163e188641', ignoreFailoverLimit='false', storageDomainId='d9f2378f-92f0-4bc1-96a8-20a0f2c575cb', imageGroupId='ee55bbe7-8001-4c82-b1a6-0cac7710704b', imageId='701a7472-8c00-4b9f-aa87-4837eb128695', dstImageGroupId='8140d2e7-6908-438e-a646-23e58a33913e', vmId='b3aa623e-3072-430e-87d6-b81a3b07f466', dstImageId='c515a3f9-0590-4ebe-81c5-9d0993f1fec9', imageDescription='', dstStorageDomainId='d9f2378f-92f0-4bc1-96a8-20a0f2c575cb', copyVolumeType='LeafVol', volumeFormat='COW', preallocate='Sparse', postZero='false', discard='false', force='true'}), log id: 164e3046 2018-09-17 14:27:03,467+08 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (EE-ManagedThreadFactory-engine-Thread-107849) [a407f6a9-445d-4ccc-b998-f9dfc6dc67ec] ++ dstImageGUID=8140d2e7-6908-438e-a646-23e58a33913e I think maybe some command did not been excuted after I export the vm as OVA. I can find the similar case here: https://access.redhat.com/solutions/1298893 I have used the tool unlock_entity.sh but cannot find any disk that is locked. So what should I do to delete the disk that is been locked? yours Adam

2 1

ovirt-imagio-proxy upload speed slow
by Dev Ops 18 Sep '19

18 Sep '19

I am working on integrating a backup solution for our ovirt environment and having issues with the time it takes to backup the VM's. This backup solution is simply taking a snapshot and making a clone and backing the clone up to a backup server. A VM that is 100 gig takes 52 minutes to back up. The same VM doing a file backup using the same product, and bypassing their rhv plugin, takes 14 minutes. So the throughput is there but the ovirt imageio-proxy process seems to be what manages how images are uploaded and is officially my bottle neck. Load is not high on the engine or kvm hosts. I had bumped up the Upload image size from 100MB to 10gig weeks ago and that didn't seem to help. [root@blah-lab-engine ~]# engine-config -a |grep Upload UploadImageChunkSizeKB: 10240000 version: general [root@bgl-vms-engine ~]# rpm -qa |grep ovirt-image ovirt-imageio-proxy-1.4.6-1.el7.noarch ovirt-imageio-common-1.4.6-1.el7.x86_64 ovirt-imageio-proxy-setup-1.4.6-1.el7.noarch I have seen bugs reported to redhat about this but I am running above the affected releases. engine software is 4.2.8.2-1.el7 Any idea what we can tweak to open up this bottleneck?

5 8

Re: Cannot Increase Hosted Engine VM Memory
by Douglas Duckworth 04 Sep '19

04 Sep '19

Yes, I do. Gold crown indeed. It's the "HostedEngine" as seen attached! Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit<https://scu.med.cornell.edu> Weill Cornell Medicine 1300 York Avenue New York, NY 10065 E: doug(a)med.cornell.edu<mailto:doug@med.cornell.edu> O: 212-746-6305 F: 212-746-8690 On Wed, Jan 23, 2019 at 12:02 PM Simone Tiraboschi <stirabos(a)redhat.com<mailto:stirabos@redhat.com>> wrote: On Wed, Jan 23, 2019 at 5:51 PM Douglas Duckworth <dod2014(a)med.cornell.edu<mailto:dod2014@med.cornell.edu>> wrote: Hi Simone Can I get help with this issue? Still cannot increase memory for Hosted Engine. From the logs it seams that the engine is trying to hotplug memory to the engine VM which is something it should not happen. The engine should simply update engine VM configuration in the OVF_STORE and require a reboot of the engine VM. Quick question, in the VM panel do you see a gold crown symbol on the Engine VM? Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit<https://scu.med.cornell.edu> Weill Cornell Medicine 1300 York Avenue New York, NY 10065 E: doug(a)med.cornell.edu<mailto:doug@med.cornell.edu> O: 212-746-6305 F: 212-746-8690 On Thu, Jan 17, 2019 at 8:08 AM Douglas Duckworth <dod2014(a)med.cornell.edu<mailto:dod2014@med.cornell.edu>> wrote: Sure, they're attached. In "first attempt" the error seems to be: 2019-01-17 07:49:24,795-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-29) [680f82b3-7612-4d91-afdc-43937aa298a2] EVENT_ID: FAILED_HOT_SET_MEMORY_NOT_DIVIDABLE(2,048), Failed to hot plug memory to VM HostedEngine. Amount of added memory (4000MiB) is not dividable by 256MiB. Followed by: 2019-01-17 07:49:24,814-05 WARN [org.ovirt.engine.core.bll.UpdateRngDeviceCommand] (default task-29) [26f5f3ed] Validation of action 'UpdateRngDevice' failed for user admin@internal-authz. Reasons: ACTION_TYPE_FAILED_VM_IS_RUNNING 2019-01-17 07:49:24,815-05 ERROR [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-29) [26f5f3ed] Updating RNG device of VM HostedEngine (adf14389-1563-4b1a-9af6-4b40370a825b) failed. Old RNG device = VmRngDevice:{id='VmDeviceId:{deviceId='6435b2b5-163c-4f0c-934e-7994da60dc89', vmId='adf14389-1563-4b1a-9af6-4b40370a825b'}', device='virtio', type='RNG', specParams='[source=urandom]', address='', managed='true', plugged='true', readOnly='false', deviceAlias='', customProperties='null', snapshotId='null', logicalName='null', hostDevice='null'}. New RNG device = VmRngDevice:{id='VmDeviceId:{deviceId='6435b2b5-163c-4f0c-934e-7994da60dc89', vmId='adf14389-1563-4b1a-9af6-4b40370a825b'}', device='virtio', type='RNG', specParams='[source=urandom]', address='', managed='true', plugged='true', readOnly='false', deviceAlias='', customProperties='null', snapshotId='null', logicalName='null', hostDevice='null'}. In "second attempt" I used values that are dividable by 256 MiB so that's no longer present. Though same error: 2019-01-17 07:56:59,795-05 INFO [org.ovirt.engine.core.vdsbroker.SetAmountOfMemoryVDSCommand] (default task-22) [7059a48f] START, SetAmountOfMemoryVDSCommand(HostName = ovirt-hv1.med.cornell.edu<http://ovirt-hv1.med.cornell.edu>, Params:{hostId='cdd5ffda-95c7-4ffa-ae40-be66f1d15c30', vmId='adf14389-1563-4b1a-9af6-4b40370a825b', memoryDevice='VmDevice:{id='VmDeviceId:{deviceId='7f7d97cc-c273-4033-af53-bc9033ea3abe', vmId='adf14389-1563-4b1a-9af6-4b40370a825b'}', device='memory', type='MEMORY', specParams='[node=0, size=2048]', address='', managed='true', plugged='true', readOnly='false', deviceAlias='', customProperties='null', snapshotId='null', logicalName='null', hostDevice='null'}', minAllocatedMem='6144'}), log id: 50873daa 2019-01-17 07:56:59,855-05 INFO [org.ovirt.engine.core.vdsbroker.SetAmountOfMemoryVDSCommand] (default task-22) [7059a48f] FINISH, SetAmountOfMemoryVDSCommand, log id: 50873daa 2019-01-17 07:56:59,862-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-22) [7059a48f] EVENT_ID: HOT_SET_MEMORY(2,039), Hotset memory: changed the amount of memory on VM HostedEngine from 4096 to 4096 2019-01-17 07:56:59,881-05 WARN [org.ovirt.engine.core.bll.UpdateRngDeviceCommand] (default task-22) [28fd4c82] Validation of action 'UpdateRngDevice' failed for user admin@internal-authz. Reasons: ACTION_TYPE_FAILED_VM_IS_RUNNING 2019-01-17 07:56:59,882-05 ERROR [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-22) [28fd4c82] Updating RNG device of VM HostedEngine (adf14389-1563-4b1a-9af6-4b40370a825b) failed. Old RNG device = VmRngDevice:{id='VmDeviceId:{deviceId='6435b2b5-163c-4f0c-934e-7994da60dc89', vmId='adf14389-1563-4b1a-9af6-4b40370a825b'}', device='virtio', type='RNG', specParams='[source=urandom]', address='', managed='true', plugged='true', readOnly='false', deviceAlias='', customProperties='null', snapshotId='null', logicalName='null', hostDevice='null'}. New RNG device = VmRngDevice:{id='VmDeviceId:{deviceId='6435b2b5-163c-4f0c-934e-7994da60dc89', vmId='adf14389-1563-4b1a-9af6-4b40370a825b'}', device='virtio', type='RNG', specParams='[source=urandom]', address='', managed='true', plugged='true', readOnly='false', deviceAlias='', customProperties='null', snapshotId='null', logicalName='null', hostDevice='null'}. This message repeats throughout engine.log: 2019-01-17 07:55:43,270-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-89) [] EVENT_ID: VM_MEMORY_UNDER_GUARANTEED_VALUE(148), VM HostedEngine on host ovirt-hv1.med.cornell.edu<http://ovirt-hv1.med.cornell.edu> was guaranteed 8192 MB but currently has 4224 MB As you can see attached the host has plenty of memory. Thank you Simone! Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit<https://scu.med.cornell.edu> Weill Cornell Medicine 1300 York Avenue New York, NY 10065 E: doug(a)med.cornell.edu<mailto:doug@med.cornell.edu> O: 212-746-6305 F: 212-746-8690 On Thu, Jan 17, 2019 at 5:09 AM Simone Tiraboschi <stirabos(a)redhat.com<mailto:stirabos@redhat.com>> wrote: On Wed, Jan 16, 2019 at 8:22 PM Douglas Duckworth <dod2014(a)med.cornell.edu<mailto:dod2014@med.cornell.edu>> wrote: Sorry for accidental send. Anyway I try to increase physical memory however it won't go above 4096MB. The hypervisor has 64GB. Do I need to modify this value with Hosted Engine offline? No, it's not required. Can you please attach your engine.log for the relevant time frame? Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit<https://scu.med.cornell.edu> Weill Cornell Medicine 1300 York Avenue New York, NY 10065 E: doug(a)med.cornell.edu<mailto:doug@med.cornell.edu> O: 212-746-6305 F: 212-746-8690 On Wed, Jan 16, 2019 at 1:58 PM Douglas Duckworth <dod2014(a)med.cornell.edu<mailto:dod2014@med.cornell.edu>> wrote: Hello I am trying to increase Hosted Engine physical memory above 4GB Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit<https://scu.med.cornell.edu> Weill Cornell Medicine 1300 York Avenue New York, NY 10065 E: doug(a)med.cornell.edu<mailto:doug@med.cornell.edu> O: 212-746-6305 F: 212-746-8690 _______________________________________________ Users mailing list -- users(a)ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave(a)ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_site_pri…> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_communit…> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WGSXQVVPJJ2CR…<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ovirt.org_archiv…>

3 8

Hosted-engine inaccessible
by Tau Makgaile 29 Aug '19

29 Aug '19

Hi, I am currently experiencing a problem with my Hosted-engine. Hosted-engine disconnected after increasing / partition. The increase went well but after some time the hosted-enigine VM disconnected and has since been giving alerts such as* re-initializingFSM*. Though VMs underneth are running, Hosted-engine --vm-status: *"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail"* There is no backup to restore at the moment. I am looking for a way to bring it up without redeploying the hosted engine. Thanks in advance for your help. Kind regards, Tau

2 1

oVirt 3.6: Node went into Error state while migrations happening
by Christopher Cox 23 Aug '19

23 Aug '19

On the node in question, the metadata isn't coming across (state) wise. It shows VMs being in an unknown state (some are up and some are down), some show as migrating and there are 9 forever hung migrating tasks. We tried to bring up some of the down VMs that had a state of Down, but that ended up getting them the state of "Wait for Lauch", though those VMs are actually started. Right now, my plan is attempt a restart of vdsmd on the node in question. Just trying to get the node to a working state again. There a total of 9 nodes in our cluster, but we can't manage any VMs on the affected node right now. Is there a way in 3.6 to cancel the hung tasks? I'm worried that if vdsmd is restarted on the node, the tasks might be "attempted"... I really need them to be forgotten if possible. Ideally want all "Unknown" to return to either an "up" or "down" state (depending if the VM is up or down) and for "Wait for Launch" for those, to go to "up" and for all the "Migrating" to go to "up" or "down" (I think only one is actually down). I'm concerned that any attempt manually maniplate the state in the ovirt mgmt head db will be moot because the node will be queried for state and that state will be taken and override anything I attempt to do. Thoughts??

2 1

Agentless backup solutions
by femi adegoke 07 Aug '19

07 Aug '19

What backup solutions are people using? I'm only interested in host-level backups (no agents in the vm's) I am aware of the following products: https://storware.eu/en/storware-vprotect/ https://github.com/zipurman/oVIRT_Simple_Backup https://www.sepusa.com/ Any others?

9 16

port_security in external networks-API
by ada per 07 Aug '19

07 Aug '19

Hello everyone, I have the following python script that creates an external network, but now after updating ovirt a new option "network port security" exists that is set as enabled by default. How can i disable the network port security? thanks!! :) networks_service = connection.system_service().networks_service() # Use the "add" method to create new VM logical network in data center network = networks_service.add( network=types.Network( name= ext_net_name, description='Network for testing API', data_center=types.DataCenter( name='Default' ), usages=[types.NetworkUsage.VM], external_provider=types.OpenStackNetworkProvider( id=provider.id ) ),

2 3

hosted engine installation / multipath / iscsi
by Michael Frank 07 Aug '19

07 Aug '19

Hi, since several days i try to install the hosted engine initially to an iscsi multipath device without success. Some information on the environment: - Version 4.3.3 - using two 10gbe interfaces as single bond for the ovirtmgmt interface - using two 10gbe storage interfaces on each hypervisor for iscsi storage -- each storage interface is configured without any bonding, etc -- each storage interface lives in a separate vlan were also the iscsi Portals/target are available, the iscsi portals have 4x10ge interfaces each, (2 in vlan xx and 2 interfaces in vlan yy ) -- so; each storage interface is connected to two iscsi Portals via 4 interfaces The documentation here is for me unclear: https://ovirt.org/documentation/self-hosted/chap-Deploying_Self-Hosted_Engi… >Note: To specify more than one iSCSI target, you must enable multipathing before deploying the self-hosted engine. There is also a Multipath >Helper tool that generates a script to install and configure multipath with different options. This indicates for me that it should be possbile to install the HE directly on the /dev/mapper/mpath device which is availibale when I have prepared the host accordingly before installing the HE (log in to multiple iscsi targets, create proper multipath,conf, etc) - right ? I login to the two iscsi portals and get in sum 8 pathes, 4 from each interface and iscsi target. Basically I have then the mpath device on the hypervisor available and i can mount the mpath device and put data on it. In the cockpit interface the mount can also be activated and is recognized correctly. multipathd -ll and lsblk looks good. Everything seems to be fine. But when I run the "hosted-engine" --deploy, the last option while running the assistant is to enter the iscsi data. So, basically i just want to define my mpath device - when entering the data (ip, port)for the iscsi Portal I can see the 4 pathes of the single hosted Engine target, and when i choose the path where the "lun" is finally available it fails. I think in general this option is not that what i want to have here for using the multipath device. I' m lost - what is the usual way to install the HE on a multipath device ? Do i have to change the configuration of the storage interfaces or the iscsi network design? Did I missed something obvious ? Can I put in my multipath data into the answerfile to get rid of the last step of the assistant ? Can I use the related ansible role for specify the Mpath device which is available when activating the multipath service ? Is it not possible in general ?? : https://bugzilla.redhat.com/show_bug.cgi?id=1193961 Sorry in advance for the long mail .... 1!^^ br, michael Sent from a mobile device

3 2

bond for vm interfaces
by Edoardo Mazza 06 Aug '19

06 Aug '19

Hello everyone, I need to create a bond for vm interfaces but I don't kown what is the best solution, you can help me? Thanks Edoardo

3 2

Data Center Compatibility version
by Rick A 06 Aug '19

06 Aug '19

So I updated our environment a bit ago during a scheduled down time from 4.2 to 4.3. Everything went smoothly, but it looks like I forgot to update the Data Center Compatibility (See alert message below). My question is, can I just change the Data Center compatibility to 4.3 while the environment is running without causing issues, or do i need to shutdown all my VM's and put the hosts into maintenance mode? The compatibility version for everything else is already set to 4.3. Thanks Alert Message: "Data Center compatibility version is 4.2, which is lower than latest engine version 4.3. Please upgrade your Data Center to latest version to successfully finish upgrade of your setup"

2 1

[ANN] oVirt 4.3.5 Fifth Release Candidate is now available for testing
by Sandro Bonazzola 06 Aug '19

06 Aug '19

The oVirt Project is pleased to announce the availability of the oVirt 4.3.5 Fifth Release Candidate for testing, as of July 11th, 2019. While testing this release candidate please consider deeper testing on gluster upgrade since with this release we are switching from Gluster 5 to Gluster 6. This update is a release candidate of the fifth in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later * oVirt Node 4.3 (available for x86_64 only) See the release notes [1] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is laready available - oVirt Node is already available[2] Additional Resources: * Read more about the oVirt 4.3.5 release highlights: http://www.ovirt.org/release/4.3.5/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.5/ [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>

3 7

ovirt-release43-pre-4.3.6-0.1 - python-ioprocess RPM version
by Maton, Brett 06 Aug '19

06 Aug '19

Hi, I just ran yum update on my test cluster and ran into the following issue: I did notice that the python2-ioprocess is currently installed from the ovirt-4.2 repo... Any suggestions? Thanks, Brett Repo RPM: ovirt-release43-pre-4.3.6-0.1.rc1.el7.noarch yum -y upgrade ... Error: Package: vdsm-python-4.30.25-1.el7.noarch (ovirt-4.3-pre) Requires: python-ioprocess >= 1.2.1 Installed: python2-ioprocess-1.1.2-1.el7.x86_64 (@ovirt-4.2) python-ioprocess = 1.1.2 Available: python-ioprocess-0.16.1-1.el7.noarch (ovirt-4.3-centos-ovirt43) python-ioprocess = 0.16.1-1.el7 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest

2 1

4.3.5 update error
by Kapetanakis Giannis 06 Aug '19

06 Aug '19

Hi, Trying today engine-setup to update from 4.3.3 -> 4.3.5 I got this error: [ ERROR ] Failed to execute stage 'Setup validation': local variable 'snapshot_cl' referenced before assignment In log file I see: 2019-07-31 10:43:00,375+0300 DEBUG otopi.context context._executeMethod:145 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/db/schema.py", line 437, in _validation self._checkSnapshotCompatibilityVersion() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/db/schema.py", line 212, in _checkSnapshotCompatibilityVersion 'v': snapshot_cl, UnboundLocalError: local variable 'snapshot_cl' referenced before assignment 2019-07-31 10:43:00,381+0300 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Setup validation': local variable 'snapshot_cl' referenced before assignment Any hints? thanks Giannis

5 5

Use public-signed SSL certs?
by Chris Adams 02 Aug '19

02 Aug '19

I installed an SSL cert from a public CA (Let's Encrypt) on my engine, following this: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/ht… That gets the regular web UI working, but I can't upload an ISO. I assume that I need to do something with the imageio-proxy service on the engine, but not sure what... I tried replacing imageio-proxy.cer and imageio-proxy.key.nopass, but that didn't work. I'm trying to avoid ever needing to install a special CA cert in browsers. -- Chris Adams <cma(a)cmadams.net>

6 19

Re: Manual Migration not working and Dashboard broken after 4.3.4 update
by Strahil 02 Aug '19

02 Aug '19

I'm not sure, but I always thought that you need an agent for live migrations. You can always try installing either qemu-guest-agent or ovirt-guest-agent and check if live migration between hosts is possible. Have you set the new cluster/dc version ? Best Regards Strahil NikolovOn Jul 9, 2019 17:42, Neil <nwilson123(a)gmail.com> wrote: > > I remember seeing the bug earlier but because it was closed thought it was unrelated, this appears to be it.... > > https://bugzilla.redhat.com/show_bug.cgi?id=1670701 > > Perhaps I'm not understanding your question about the VM guest agent, but I don't have any guest agent currently installed on the VM, not sure if the output of my qemu-kvm process maybe answers this question?.... > > /usr/libexec/qemu-kvm -name guest=Headoffice.cbl-ho.local,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-Headoffice.cbl-ho.lo/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu Broadwell,vme=on,f16c=on,rdrand=on,hypervisor=on,arat=on,xsaveopt=on,abm=on,rtm=on,hle=on -m 8192 -realtime mlock=off -smp 8,maxcpus=64,sockets=16,cores=4,threads=1 -numa node,nodeid=0,cpus=0-7,mem=8192 -uuid 9a6561b8-5702-43dc-9e92-1dc5dfed4eef -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-3.1611.el7.centos,serial=4C4C4544-0034-5810-8033-C2C04F4E4B32,uuid=9a6561b8-5702-43dc-9e92-1dc5dfed4eef -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=31,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2019-07-09T10:26:53,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/59831b91-00a5-01e4-0294-000000000018/8a607f8a-542a-473c-bb18-25c05fe2a3d4/images/56e8240c-a172-4f52-b0c1-2bddc4f34f93/9f245467-d31d-4f5a-8037-7c5012a4aa84,format=qcow2,if=none,id=drive-virtio-disk0,serial=56e8240c-a172-4f52-b0c1-2bddc4f34f93,werror=stop,rerror=stop,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:5b,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,fd=35,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,fd=36,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5900,addr=10.0.1.11,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=8388608,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x8 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on > > Please shout if you need further info. > > Thanks. > > > > > > > On Tue, Jul 9, 2019 at 4:17 PM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: >> >> Shouldn't cause that problem. >> >> You have to find the bug in bugzilla and report a regression (if it's not closed) , or open a new one and report the regression. >> As far as I remember , only the dashboard was affected due to new features about vdo disk savings. >>

5 13

[ANN] oVirt 4.3.5 first async is now available
by Sandro Bonazzola 31 Jul '19

31 Jul '19

The oVirt Team has just released a new version of the following packages: - ovirt-ansible-disaster-recovery-1.2.0 - ovirt-ansible-hosted-engine-setup-1.0.24 - ovirt-engine-dwh-4.3.5.1 - ovirt-engine-metrics-1.3.3.3 - ovirt-release43-4.3.5.1 The async release includes additional fixes: - [BZ 1721073 <https://bugzilla.redhat.com/1721073>] Run metrics ansible role only if the ovirt-engine-metrics rpm installed - [BZ 1731871 <https://bugzilla.redhat.com/1731871>] Unable to install metrics VM on rhev - failed to connect to master VM - [BZ 1727064 <https://bugzilla.redhat.com/1727064>] Deployment from Bastion Machine fails with Ansible 2.8 on upstream - [BZ 1698888 <https://bugzilla.redhat.com/1698888>] Update task "Get bastion ssh public key" to use ansible module other then command - [BZ 1730384 <https://bugzilla.redhat.com/1730384>] Connection closed with an exception - [BZ 1683157 <https://bugzilla.redhat.com/1683157>] CVE-2019-10194 <https://access.redhat.com/security/cve/cve-2019-10194> ovirt-engine-metrics: disclosure of sensitive passwords in log files and ansible playbooks Notes: - oVirt Appliance is already available - oVirt Node is already available[2] oVirt Node and Appliance have been updated including above packages and: - latest CentOS Virt and Storage SIG updates: - ansible-2.8.3: https://github.com/ansible/ansible/blob/stable-2.8/changelogs/CHANGELOG-v2.… - glusterfs-6.4: https://docs.gluster.org/en/latest/release-notes/6.4 - cockpit 195: https://cockpit-project.org/blog/cockpit-195.html - openvswitch 2.11: http://openvswitch.org/releases/NEWS-2.11.0 Additional Resources: * Read more about the oVirt 4.3.5 release highlights: http://www.ovirt.org/release/4.3.5/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.5/ [2] http://resources.ovirt.org/pub/ovirt-4.3/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

oVirt 4.3.5 is now generally available
by Sandro Bonazzola 30 Jul '19

30 Jul '19

The oVirt Project is pleased to announce the general availability of oVirt 4.3.5 as of July 30th, 2019. This update is the fourth in a series of stabilization updates to the 4.3 series. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.6 or later (but < 8) * CentOS Linux (or similar) 7.6 or later (but < 8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.6 or later (but < 8) * CentOS Linux (or similar) 7.6 or later (but < 8) * oVirt Node 4.3 (available for x86_64 only) Due to Fedora 28 being now at end of life this release is missing experimental tech preview for x86_64 and s390x architectures for Fedora 28. We are working on Fedora 29 and 30 support and we may re-introduce experimental support for Fedora in next release. See the release notes [1] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is already available[2] Please note that released Appliance and Node are missing ansible-2.8.3 and Gluster 6.4. We already tested this release with these versions but due to an issue on CentOS signing server we didn’t got he builds on CentOS mirrors. Since a new RHEL batch update has been shipped last night we’ll respin once all the new packages will be available on CentOS mirrors. oVirt Node and Appliance have been updated including: - oVirt 4.3.5: http://www.ovirt.org/release/4.3.5/ - wildfly-17.0.0: https://wildfly.org/news/2019/06/10/WildFly17-Final-Released/ - Latest CentOS updates including: - CESA-2019:1294 Important CentOS 7 bind Security Update <https://lists.centos.org/pipermail/centos-announce/2019-June/023321.html> - CEBA-2019:1331 CentOS 7 glib2 BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-June/023325.html> - CEBA-2019:1340 CentOS 7 lvm2 BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023354.html> - CEBA-2019:1338 CentOS 7 glibc BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023352.html> - CESA-2019:1815 Moderate CentOS 7 java-1.8.0-openjdk Security Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023373.html> - CEBA-2019:1339 CentOS 7 kernel BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-June/023326.html> - CEBA-2019:1333 CentOS 7 libteam BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023353.html> - CEEA-2019:1612 CentOS 7 microcode_ctl Enhancement Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023344.html> - CESA-2019:1587 Important CentOS 7 python Security Update <https://lists.centos.org/pipermail/centos-announce/2019-June/023337.html> - CEBA-2019:1336 CentOS 7 systemd BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023356.html> - CEBA-2019:1703 CentOS 7 tzdata BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023363.html> - CESA-2019:1619 Important CentOS 7 vim Security Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023345.html> - CEBA-2019:1345 CentOS 7 fence-agents BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-June/023323.html> - CESA-2019:1579 Important CentOS 7 libvirt Security Update <https://lists.centos.org/pipermail/centos-announce/2019-June/023336.html> - - latest CentOS Virt and Storage SIG updates: - ansible-2.8.2: https://github.com/ansible/ansible/blob/stable-2.8/changelogs/CHANGELOG-v2.… - glusterfs-6.3: https://docs.gluster.org/en/latest/release-notes/6.3 - cockpit 195: https://cockpit-project.org/blog/cockpit-195.html Additional Resources: * Read more about the oVirt 4.3.5 release highlights: http://www.ovirt.org/release/4.3.5/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.5/ [2] http://resources.ovirt.org/pub/ovirt-4.3/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*

1 0

Leeroy Jenkins cloud plugin for oVirt
by Vrgotic, Marko 30 Jul '19

30 Jul '19

Hey oVirt, How are you all doing? Has anyone used or has experience with Leeroy Jenkins on oVIrt. We need to move master ci and its slaves to oVirt, but cannot find suitable plugin for it. For builds we run, Jenkins need to spawn slaves, for which plugin is required. Please help if possible. Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic

1 0

Trouble initializing new VM in 4.2.8.2-1.el7
by David Johnson 30 Jul '19

30 Jul '19

Good evening, Thanks in advance, I'm trying to set up a new VM in my cluster, and things appear to have hung up initializing the 80GB boot partition for a new Windows VM. The system has been sitting like this for 6 hours. What am I looking for to resolve this issue? Can I kill this process safely and start over? [image: image.png] [image: image.png] Regards, David Johnson Director of Development, Maxis Technology 844.696.2947 ext 702 (o) | 479.531.3590 (c) djohnson(a)maxistechnology.com [image: Maxis Techncology] <http://www.maxistechnology.com> www.maxistechnology.com *stay connected <http://www.linkedin.com/in/pojoguy>*

2 1

Moving data to new storage appliance
by David Johnson 30 Jul '19

30 Jul '19

Hi everyone, I'm sorry to bother y'all with another noob question. We are in the process of retiring the old storage appliance that backed our oVirt cluster in favor of a new appliance. I have migrated all of the active VM storage to the new appliance, but can't see how to migrate the "base versions". My understanding is that if I just drop the storage with the base versions then the derived VM's will cease to be functional. Please advise.

2 3

bond for vm interfaces
by Edoardo Mazza 29 Jul '19

29 Jul '19

Hello everyone, I need to create a bond for vm interfaces but I don't kown what is the best solution, you can help me? Thanks Edoardo

5 4

Re: Active-Passive DR: mutual for different storage domains possible?
by Gianluca Cecchi 29 Jul '19

29 Jul '19

On Thu, Jul 25, 2019 at 2:21 PM Eyal Shenitzky <eshenitz(a)redhat.com> wrote: > On Thu, Jul 25, 2019 at 3:02 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com> > wrote: > >> On Thu, Jul 25, 2019 at 1:54 PM Eyal Shenitzky <eshenitz(a)redhat.com> >> wrote: >> >>> >>> Please notice that a automation python scripts created in order to >>> facilitate the DR process. >>> You can find them under - path/to/your/dr/folder/files. >>> >>> You can use those scripts to generate the mapping, test the generated >>> mapping and start the failover/failback. >>> >>> I strongly recommend to use it. >>> >>> >> Yes, I have used it to create the disaster_recovery_vars.yml mapping file >> and then populating it with the secondary site information, thanks. >> My doubt was about any difference in playbook actions between "failover" >> (3.3) and "discreet failover test" (B.1), as the executed playbook and >> tags are the same. >> > > No, the only difference is that you disable the storage replication by > yourself, this way you can test the failover while the other "primary" site > is still active. > > First "discreet failover test" was a success!!! Great. Storage domain attached, templates imported and the only VM defined at source correctly started (at source I configured link down for the VM, inherited at target, so no collisions). Elapsed between beginning of ovirt connection, until first template import has been about 6 minutes. ... Template TOL76 has been successfully imported from the given configuration. 7/25/19 3:26:58 PM Storage Domain ovsd3910 was attached to Data Center SVIZ3-DR by admin@internal-authz 7/25/19 3:26:46 PM Storage Domains were attached to Data Center SVIZ3-DR by admin@internal-authz 7/25/19 3:26:46 PM Storage Domain ovsd3910 (Data Center SVIZ3-DR) was activated by admin@internal-authz 7/25/19 3:26:46 PM ... Storage Pool Manager runs on Host ovh201. (Address: ovh201.), Data Center SVIZ3-DR. 7/25/19 3:26:36 PM Data Center is being initialized, please wait for initialization to complete. 7/25/19 3:23:53 PM Storage Domain ovsd3910 was added by admin@internal-authz 7/25/19 3:20:43 PM Disk Profile ovsd3910 was successfully added (User: admin@internal-authz). 7/25/19 3:20:42 PM User admin@internal-authz connecting from '10.4.192.43' using session 'xxx' logged in. 7/25/19 3:20:35 PM Some notes: 1) iSCSI multipath my storage domains are iSCSI based and my hosts have two network cards to reach the storage. I'm using EQL that doesn't support bonding and has one portal that all initiators use. So in my primary env I configured "iSCSI Multipathing" tab in Compute --> Datacenter --> Datacenter_Name window. But this tab appears only when you activate the storage. So during the ansible playbook run the iSCSI connection has been activated through the "default" iscsi interface I can then: - configure "iSCSI Multipathing" - shutdown VM - put host into maintenance - remove the default iSCSI session that has not been removed on host iscsiadm -m session -r 6 -u - activate host now I have: [root@ov201 ~]# iscsiadm -m session tcp: [10] 10.10.100.8:3260,1 iqn.2001-05.com.equallogic:4-771816-99d82fc59-5bdd77031e05beac-ovsd3910 (non-flash) tcp: [9] 10.10.100.8:3260,1 iqn.2001-05.com.equallogic:4-771816-99d82fc59-5bdd77031e05beac-ovsd3910 (non-flash) [root@ov201 ~]# with # multipath -l 364817197c52fd899acbe051e0377dd5b dm-29 EQLOGIC ,100E-00 size=1.0T features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 23:0:0:0 sdb 8:16 active undef running `- 24:0:0:0 sdc 8:32 active undef running - start vm The I do a cleanup: 1. Detach the storage domains from the secondary site. 2. Enable storage replication between the primary and secondary storage domains. The storage domain remains as "unattached" in DR environment Then I executed the test again and during connection I got this error about 40 seconds after run of playbook TASK [oVirt.disaster-recovery : Import iSCSI storage domain] *************************************** An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400. failed: [localhost] (item=iqn.2001-05.com.equallogic:4-771816-99d82fc59-5bdd77031e05beac-ovsd3910) => {"ansible_loop_var": "dr_target", "changed": false, "dr_target": "iqn.2001-05.com.equallogic:4-771816-99d82fc59-5bdd77031e05beac-ovsd3910", "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400."} In webadmin gui of DR env I see: VDSM ov201 command CleanStorageDomainMetaDataVDS failed: Cannot obtain lock: "id=56eadc97-5731-40cf-8409-aff58d8ffd11, rc=-243, out=Cannot acquire Lease(name='SDM', path='/dev/56eadc97-5731-40cf-8409-aff58d8ffd11/leases', offset=1048576), err=(-243, 'Sanlock resource not acquired', 'Lease is held by another host')" 7/25/19 4:50:43 PM What could be the cause of this? In vdsm.log: 2019-07-25 16:50:43,196+0200 INFO (jsonrpc/1) [vdsm.api] FINISH forcedDetachStorageDomain error=Cannot obtain lock: "id=56eadc97-5731-40cf-8409-aff58d8ffd11, rc=-243, out=Cannot acquire Lease(name='SDM', path='/dev/56eadc97-5731-40cf-8409-aff58d8ffd11/leases', offset=1048576), err=(-243, 'Sanlock resource not acquired', 'Lease is held by another host')" from=::ffff:10.4.192.79,49038, flow_id=4bd330d1, task_id=c0dfac81-5c58-427d-a7d0-e8c695448d27 (api:52) 2019-07-25 16:50:43,196+0200 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='c0dfac81-5c58-427d-a7d0-e8c695448d27') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in forcedDetachStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 856, in forcedDetachStorageDomain self._detachStorageDomainFromOldPools(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 834, in _detachStorageDomainFromOldPools dom.acquireClusterLock(host_id) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 910, in acquireClusterLock self._manifest.acquireDomainLock(hostID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 527, in acquireDomainLock self._domainLock.acquire(hostID, self.getDomainLease()) File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", line 419, in acquire "Cannot acquire %s" % (lease,), str(e)) AcquireLockFailure: Cannot obtain lock: "id=56eadc97-5731-40cf-8409-aff58d8ffd11, rc=-243, out=Cannot acquire Lease(name='SDM', path='/dev/56eadc97-5731-40cf-8409-aff58d8ffd11/leases', offset=1048576), err=(-243, 'Sanlock resource not acquired', 'Lease is held by another host')" 2019-07-25 16:50:43,196+0200 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='c0dfac81-5c58-427d-a7d0-e8c695448d27') aborting: Task is aborted: 'Cannot obtain lock: "id=56eadc97-5731-40cf-8409-aff58d8ffd11, rc=-243, out=Cannot acquire Lease(name=\'SDM\', path=\'/dev/56eadc97-5731-40cf-8409-aff58d8ffd11/leases\', offset=1048576), err=(-243, \'Sanlock resource not acquired\', \'Lease is held by another host\')"' - code 651 (task:1181) 2019-07-25 16:50:43,197+0200 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH forcedDetachStorageDomain error=Cannot obtain lock: "id=56eadc97-5731-40cf-8409-aff58d8ffd11, rc=-243, out=Cannot acquire Lease(name='SDM', path='/dev/56eadc97-5731-40cf-8409-aff58d8ffd11/leases', offset=1048576), err=(-243, 'Sanlock resource not acquired', 'Lease is held by another host')" (dispatcher:83) 2019-07-25 16:50:43,197+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StorageDomain.detach failed (error 651) in 24.12 seconds (__init__:312) 2019-07-25 16:50:44,180+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:10.4.192.79,49038 (api:48) 2019-07-25 16:50:44,222+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:10.4.192.79,49038, task_id=8a7a0302-4ee3-49a8-a3f7-f9636a123765 (api:48) 2019-07-25 16:50:44,222+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={} from=::ffff:10.4.192.79,49038, task_id=8a7a0302-4ee3-49a8-a3f7-f9636a123765 (api:54) 2019-07-25 16:50:44,223+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:10.4.192.79,49038, task_id=fb09923c-0888-4c3f-9b8a-a7750592da22 (api:48) 2019-07-25 16:50:44,223+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:10.4.192.79,49038, task_id=fb09923c-0888-4c3f-9b8a-a7750592da22 (api:54) After putting host into maintenance + reboot of the host and re-run of playbook, all went well again.. 2) Mac Address pools I notice that the imported VM has preserved the link state (down in my case) but not the mac address. The mac address is the one defined in my target engine, that is different from source engine to avoid overlap of mac adresses Is this an option I can customize? In general a VM could have problems when changing its mac address... 3) Clean up dest DC after "discreet failover test" The guide says: 1. Detach the storage domains from the secondary site. 2. Enable storage replication between the primary and secondary storage domains. better to also restart the DR hosts? 4) VM consistency Can we say that all the imported VMs will be "crash consistent"? Thanks, Gianluca

2 5

Re: engine-setup failure on 4.1 -> 4.2
by Strahil 27 Jul '19

27 Jul '19

Try to remove the maintenance, wait 10 min and then put it back: hosted-engine --set-maintenance --mode=none; sleep 600; hosted-engine --set-maintenance --mode=global ; sleep 300 Best Regards, Strahil NikolovOn Jul 26, 2019 20:49, Alex K <rightkicktech(a)gmail.com> wrote: > > I repeated the same upgrade steps to another cluster and although I was receiving same warnings about the db the upgrade completed successfully. > > Is there a way I can manually inform engine db about the maintenance status? I was thinking that in this way the engine would proceed with the remaining steps. > > On Thu, Jul 25, 2019 at 3:55 PM Alex K <rightkicktech(a)gmail.com> wrote: >> >> Hi all, >> >> I have a self hosted engine setup, with 3 servers. >> I have successfully upgraded several other installations, from 4.1 to 4.2. >> On one of them I am encountering and issue with the engine-setup. >> >> I get the following warning: >> >> Found the following problems in PostgreSQL configuration for the Engine database: >> It is required to be at least '8192' >> >> Please note the following required changes in postgresql.conf on 'localhost': >> 'work_mem' is currently '1024'. It is required to be at least '8192'. >> postgresql.conf is usually in /var/lib/pgsql/data, /var/opt/rh/rh-postgresql95/lib/pgsql/data, or somewhere under /etc/postgresql* . You have to restart PostgreSQL after making these changes. >> The database requires these configurations values to be changed. Setup can fix them for you or abort. Fix automatically? (Yes, No) [Yes]: >> >> Then, if I select Yes to proceed, I get: >> >> [WARNING] This release requires PostgreSQL server 9.5.14 but the engine database is currently hosted on PostgreSQL server 9.2.24. >> >> Then finally: >> [ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. >> In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data. >> >> [ ERROR ] Failed to execute stage 'Setup validation': Hosted Engine setup detected, but Global Maintenance is not set. >> [ INFO ] Stage: Clean up >> Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20190725124653-hvekp2.log >> [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20190725125154-setup.conf' >> [ INFO ] Stage: Pre-termination >> [ INFO ] Stage: Termination >> [ ERROR ] Execution of setup failed >> >> I have put the cluster on global maintenance, though the engine thinks it is not. >> >> Are any steps that I may follow to avoid above? >> I am attaching also the last full setup log. >> Thank you! >> >> Alex >> >> >>

1 0