- Users - oVirt List Archives

Arbiter brick disk performance
by Leo David 24 Apr '19

24 Apr '19

Hello Everyone, I need to look into adding some enterprise grade sas disks ( both ssd and spinning ), and since the prices are not too low, I would like to benefit of replica 3 arbitrated. Therefore, I intend to buy some smaller disks for use them as arbiter brick. My question is, what performance ( regarding iops, througput ) the arbiter disks need to be. Should they be at least the same as the real data disks ? Knowing that they only keep metadata, I am thinking that will not be so much pressure on the arbiters. Any thoughts? Thank you ! -- Best regards, Leo David

1 0

Upgrading from 4.2.8 to 4.3.3 broke Node NG GlusterFS
by Andreas Elvers 24 Apr '19

24 Apr '19

Hi, I am currently upgrading my oVirt setup from 4.2.8 to 4.3.3.1. The setup consists of: Datacenter/Cluster Default: [fully upgraded to 4.3.3.1] 2 nodes (node04,node05)- NFS storage domain with self hosted engine Datacenter Luise: Cluster1: 3 nodes (node01,node02,node03) - Node NG with GlusterFS - Ceph Cinder storage domain [Node1 and Node3 are upgraded to 4.3.3.1, Node2 is on 4.2.8] Cluster2: 1 node (node06) - only Ceph Cinder storage domain [fully upgraded to 4.3.3.1] Problems started when upgrading Luise/Cluster1 with GlusterFS: (I always waited for GlusterFS to be fully synced before proceeding to the next step) - Upgrade node01 to 4.3.3 -> OK - Upgrade node03 to 4.3.3.1 -> OK - Upgrade node01 to 4.3.3.1 -> GlusterFS became unstable. I now get the error message: VDSM node03.infra.solutions.work command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=f3218bf7-6158-4b2b-b272-51cdc3280376, msdUUID=02a32017-cbe6-4407-b825-4e558b784157' And on node03 there is a problem with Gluster: node03#: ls -l /rhev/data-center/mnt/glusterSD/node01.infra.solutions.work:_vmstore ls: cannot access /rhev/data-center/mnt/glusterSD/node01.infra.solutions.work:_vmstore: Transport endpoint is not connected The directory is available on node01 and node02. The engine is reporting the brick on node03 as down. Node03 and Node06 are shown as NonOperational, because they are not able to access the gluster storage domain. A “gluster peer status” on node1, node2, and node3 shows all peers connected. “gluster volume heal vmstore info” shows for all nodes: gluster volume heal vmstore info Brick node01.infra.solutions.work:/gluster_bricks/vmstore/vmstore Status: Transport endpoint is not connected Number of entries: - Brick node02.infra.solutions.work:/gluster_bricks/vmstore/vmstore <gfid:0bcb7825-e649-4178-a899-c5cc04c95286> <gfid:71ec8035-f5a5-4e61-bb34-5ad9db28c0eb> <gfid:16d5961e-c3bb-4493-a51d-bf83074c4cc7> /02a32017-cbe6-4407-b825-4e558b784157/dom_md/ids /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.66 <gfid:5fe350e4-1eb5-4b6f-a3fb-42c98b7b2f8d> /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.60 /02a32017-cbe6-4407-b825-4e558b784157/images/a3a10398-9698-4b73-84d9-9735448e3534/6161e310-4ad6-42d9-8117-5a89c5b2b4b6 <gfid:8eb9fd30-fdb9-442b-9c54-8ba256d7981b> <gfid:c72001be-e7d3-4b34-bac5-9ab50b609eea> /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.96 <gfid:447ec09b-336e-4d2b-8338-f31329ee7a55> <gfid:9d7db516-d6fb-43d8-a069-dcbc1d72e62a> /.shard/d66880de-3fa1-4362-8c43-574a173c5f7d.133 <gfid:2412d449-d3ed-40ef-b7eb-d81bdf7c5c05> <gfid:0fae358b-2cdd-4064-b63c-7f31a35bc35a> /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.38 /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.67 /__DIRECT_IO_TEST__ <gfid:a7945526-9ff3-40fe-b1e2-3921117ef738> <gfid:e78e9c1f-ce6b-4871-b5bf-9bde34685b99> /02a32017-cbe6-4407-b825-4e558b784157/images/493188b2-c137-4440-99ee-43a753842a7d/9aa2d139-e3bd-406b-8fe0-b189123eaa73 <gfid:3aed3fb6-044a-4371-9302-e0bd54cbd794> /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.64 /.shard/d66880de-3fa1-4362-8c43-574a173c5f7d.132 <gfid:f7631be7-2ab5-4985-904d-69174c0e1267> <gfid:43001625-1aad-4032-a76e-4cc2a51de2b3> <gfid:6ae3fe7f-15c9-4103-960c-faba0ba59cb3> /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.44 /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.9 /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.69 <gfid:f20cc6f7-9391-4260-9238-3e1d0cabbfa3> /02a32017-cbe6-4407-b825-4e558b784157/images/12e647fb-20aa-4957-b659-05fa75a9215e/f7e4b2a3-ab84-4eb5-a4e7-7208ddad8156 <gfid:c540368a-4431-4405-9a59-e11a217d0ea6> <gfid:4e698a74-39dc-40a3-ac9c-14456420ab66> <gfid:afd48e71-ff23-42d7-aef4-b2e2167b75e8> <gfid:194589b3-0760-4150-80ef-d87376813835> <gfid:6e17ead1-88cc-4e3e-84fa-7495c4fc3a0e> /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.35 /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.32 <gfid:982d071f-2081-4371-b007-bc48d8167e7c> <gfid:e2285905-a8da-44f5-8c56-1f8a4d6326a8> /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.39 <gfid:956c7d9c-2f96-42d1-bf6e-57bc9e534f84> <gfid:16162bc7-201a-4842-a41a-af2cc4fb8a9e> /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.34 /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.68 Status: Connected Number of entries: 47 Brick node03.infra.solutions.work:/gluster_bricks/vmstore/vmstore /02a32017-cbe6-4407-b825-4e558b784157/images/12e647fb-20aa-4957-b659-05fa75a9215e/f7e4b2a3-ab84-4eb5-a4e7-7208ddad8156 <gfid:c540368a-4431-4405-9a59-e11a217d0ea6> <gfid:4e698a74-39dc-40a3-ac9c-14456420ab66> <gfid:afd48e71-ff23-42d7-aef4-b2e2167b75e8> <gfid:194589b3-0760-4150-80ef-d87376813835> <gfid:6e17ead1-88cc-4e3e-84fa-7495c4fc3a0e> <gfid:099284a6-9538-4f9a-928a-d9b704fe0735> <gfid:75d3c8f7-d67a-49a4-9cd4-7fff202df40d> <gfid:982d071f-2081-4371-b007-bc48d8167e7c> <gfid:e2285905-a8da-44f5-8c56-1f8a4d6326a8> <gfid:447ec09b-336e-4d2b-8338-f31329ee7a55> <gfid:956c7d9c-2f96-42d1-bf6e-57bc9e534f84> /.shard/d66880de-3fa1-4362-8c43-574a173c5f7d.133 <gfid:43001625-1aad-4032-a76e-4cc2a51de2b3> <gfid:6ae3fe7f-15c9-4103-960c-faba0ba59cb3> <gfid:1a0b2737-9172-4c51-aa77-e93e9671840c> <gfid:eb471e13-6749-4f62-b1f5-15a44f8990c2> <gfid:a7945526-9ff3-40fe-b1e2-3921117ef738> <gfid:e78e9c1f-ce6b-4871-b5bf-9bde34685b99> /02a32017-cbe6-4407-b825-4e558b784157/images/493188b2-c137-4440-99ee-43a753842a7d/9aa2d139-e3bd-406b-8fe0-b189123eaa73 <gfid:6b418e80-9f61-4d6e-ba77-8a1969d9a99b> <gfid:914c72d2-e45e-48f2-b7ef-5846b13f7a91> <gfid:2bd28bdb-1dc6-41d5-96be-c696f452e3f2> <gfid:9d7db516-d6fb-43d8-a069-dcbc1d72e62a> <gfid:f7631be7-2ab5-4985-904d-69174c0e1267> <gfid:16162bc7-201a-4842-a41a-af2cc4fb8a9e> /.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.44 <gfid:afc6d611-528d-441b-b74e-c5fae6672088> <gfid:707308e9-e8e5-487a-b0f1-a816720c4243> <gfid:5f1a81a7-7c42-4226-9142-1b5b35c2b1e9> <gfid:f20cc6f7-9391-4260-9238-3e1d0cabbfa3> <gfid:0bcb7825-e649-4178-a899-c5cc04c95286> <gfid:71ec8035-f5a5-4e61-bb34-5ad9db28c0eb> <gfid:16d5961e-c3bb-4493-a51d-bf83074c4cc7> /02a32017-cbe6-4407-b825-4e558b784157/dom_md/ids <gfid:7661180b-1917-4a7b-9749-5dfb826c4449> <gfid:5fe350e4-1eb5-4b6f-a3fb-42c98b7b2f8d> <gfid:a6197593-7e09-4d3f-b538-9cd1ebadd6c9> /02a32017-cbe6-4407-b825-4e558b784157/images/a3a10398-9698-4b73-84d9-9735448e3534/6161e310-4ad6-42d9-8117-5a89c5b2b4b6 <gfid:8eb9fd30-fdb9-442b-9c54-8ba256d7981b> <gfid:c72001be-e7d3-4b34-bac5-9ab50b609eea> <gfid:3aed3fb6-044a-4371-9302-e0bd54cbd794> /.shard/d66880de-3fa1-4362-8c43-574a173c5f7d.132 <gfid:2412d449-d3ed-40ef-b7eb-d81bdf7c5c05> <gfid:0fae358b-2cdd-4064-b63c-7f31a35bc35a> <gfid:c0ca2784-a8af-44b3-9091-a1eaf4c8676f> /__DIRECT_IO_TEST__ Status: Connected Number of entries: 47 On Node03 there are several self healing processes, that seem to be doing nothing. Oh well.. What now? Best regards, - Andreas

2 4

Unable to use MAC address starting with reserved value 0xFE
by Ricardo Alonso 24 Apr '19

24 Apr '19

Is there a way to use a mac starting with FE? The machine has a license requirement attached to the mac address, and when I try to start it, it fails with the message: VM is down with error. Exit message: unsupported configuration: Unable to use MAC address starting with reserved value 0xFE - 'fe:XX:XX:XX:XX:XX'

1 0

Upgrade Initiation fails without obvious reason
by Vrgotic, Marko 24 Apr '19

24 Apr '19

Dear oVIrt, Short platform summary: * 5 Hosts * 3/5 in Shared DC/Cluster with NetApp NFS storage * 1/5 with Local Storage * 1/5 with Local Storage * oVirt version 4.3.2 , recently upgraded from 4.2.8 Host related: ovirt-staging-hv-04 Host is moved to Maintenance. Selected Upgrade, following log lines are written: 2019-04-23 10:58:32,639Z INFO [org.ovirt.engine.core.bll.hostdeploy.UpgradeHostCommand] (default task-247) [11381573-01c0-4e48-8634-c577988a8f79] Running command: UpgradeHostCommand internal: false. Entities affected : ID: 9462386e-74d1-4000-96b0-0e081ec7d069 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN 2019-04-23 10:58:32,648Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-247) [11381573-01c0-4e48-8634-c577988a8f79] EVENT_ID: HOST_UPGRADE_STARTED(840), Host ovirt-staging-hv-04 upgrade was started (User: mvrgotic@ictv.com@ictv.com-authz<mailto:mvrgotic@ictv.com@ictv.com-authz>). However, the host is still in Maintenance mode, expected is Installing: [cid:image001.png@01D4F9D5.35D3A4C0] Tail YUM logs also shows the procedure did not start and packages are not being updated. Kindly awaiting you reply. Marko Vrgotic

3 2

Prevent 2 different VMs from running on the same host
by Paulo Silva 24 Apr '19

24 Apr '19

Hi, I have a cluster of 6 hosts using ovirt 4.3 and I want to make sure that 2 VMs are always started on different hosts. Is it possible to prevent 2 different VMs from running on the same physical host without specifying manually a different set of hosts where each VM can start running? Thanks -- Paulo Silva <paulojjs(a)gmail.com>

2 1

Scale out ovirt 4.3 (from 3 to 6 or 9 nodes) with hyperconverged setup and Gluster
by adrianquintero＠gmail.com 24 Apr '19

24 Apr '19

Hello, I have a 3 node Hyperconverged setup with gluster and added 3 new nodes to the cluster for a total of 6 servers. I am now taking advantage of more compute power but cant scale out my storage volumes. Current Hyperconverged setup: - host1.mydomain.com ---> Bricks: engine data1 vmstore1 - host2.mydomain.com ---> Bricks: engine data1 vmstore1 - host3.mydomain.com ---> Bricks: engine data1 vmstore1 From these 3 servers we get the following Volumes: - engine(host1:engine, host2:engine, host3:engine) - data1 (host1:data1, host2:data1, host3:data1) - vmstore1 (host1:vmstore1, host2:vmstore1, host3:vmstore11) The following are the newly added servers to the cluster, however as you can see there are no gluster bricks - host4.mydomain.com - host5.mydomain.com - host6.mydomain.com I know that the bricks must be added in sets of 3 and per the first 3 hosts that is how it was deployed thru the web UI. Questions: -how can i extend the gluster volumes engine, data1 and vmstore using host4, host5 and host6? -Do I need to configure gluster volumes manually through the OS CLI in order for them to span amongst all 6 servers? -If I configure the fail storage scenario manually will oVirt know about it?, will it still be hyperconverged? I have only seen 3 host hyperconverged setup examples with gluster, but have not found examples for 6, 9 or 12 host cluster with gluster. I know it might be a lack of understanding from my end on how ovirt and gluster integrate with one another. If you can point me in the right direction would be great. thanks,

4 6

Gluster and few iSCSI Datastores in one Data Center
by toslavik＠yandex.ru 23 Apr '19

23 Apr '19

I decided to add another cluster to the existing data center (Enable Virt Service + Enable Gluster Service). Three nodes. But after installing the nodes (without errors) Ovirt Engine loses them cyclically. From the logs you can see that this happens when you try to interrogate the connected block devices. These block devices are LVM for VM. I use several Datastores submitted via iSCSI. About 2000 virtual machines and more than 2300 LVMs. All in the neighbor cluster (one data center). Logs in https://drive.google.com/open?id=1ja7Usxx5YCFDgjD2X51z9tzn_ycPoC2g Why is this happening and what can be done in this case?

2 2

ovirt with kvm stand alone
by igalvarez＠gmail.com 23 Apr '19

23 Apr '19

I have 3 KVM stand alone servers with no external storage. My question is, can I add these 3 KVM servers to ovirt without impact on VMs already running on KVM side? Thanks.

3 3

Re: 4.3.x upgrade and issues with OVN
by Weber, Charles (NIH/NIA/IRP) [E] 23 Apr '19

23 Apr '19

Configuration with versions Next email will have log files. 2 sites First site: Bayview 4 nodes BL460 gen9 with 4 x 10G nics Node 1-3 have not been changed since 4.3.2 upgrade. These nodes have the network sync issue and cannot migrate VMs. OS Version: RHEL - 7 - 6.1810.2.el7.centos OS Description: CentOS Linux 7 (Core) Kernel Version: 3.10.0 - 957.10.1.el7.x86_64 KVM Version: 2.12.0 - 18.el7_6.3.1 LIBVIRT Version: libvirt-4.5.0-10.el7_6.6 VDSM Version: vdsm-4.30.11-1.el7 SPICE Version: 0.14.0 - 6.el7_6.1 CEPH Version: librbd1-10.2.5-4.el7 Open vSwitch Version: openvswitch-2.10.1-3.el7 Kernel Features: PTI: 1, IBRS: 0, RETP: 1 VNC Encryption: Disabled I evacuated node 4 and did update to 4.3.3, still had issues so tried ISO OVNode install. It does not have network sync issue. However I cannot upgrade all the nodes without scheduling cluster downtime. OS Version: RHEL - 7 - 6.1810.2.el7.centos OS Description: oVirt Node 4.3.3.1 Kernel Version: 3.10.0 - 957.10.1.el7.x86_64 KVM Version: 2.12.0 - 18.el7_6.3.1 LIBVIRT Version: libvirt-4.5.0-10.el7_6.6 VDSM Version: vdsm-4.30.13-1.el7 SPICE Version: 0.14.0 - 6.el7_6.1 CEPH Version: librbd1-10.2.5-4.el7 Open vSwitch Version: openvswitch-2.10.1-3.el7 Kernel Features: PTI: 1, IBRS: 0, RETP: 1, SSBD: 3 VNC Encryption: Disabled 1 engine, DL360g10. 4.3.3.5-1.el7 CentOS 7 patched up to today Storage: Dell Unity ISCSI What I see in the engine is that the first 3 nodes all have the public network listed as out of sync with the DC. I can migrate form node 4 to the other 3 nodes but cannot migrate off the other 3 nodes. I also cannot sync the network on the other three nodes. There have been no recent network changes. After the 4.3.3 upgrade I initially found some curious OVN errors in the logfile on nodes 3 and 4. Nodes 1 and 2 do not have these errors. However the engine did have 2 extra OVN ports defined. ovs-vsctl show be10cd3d-85fe-4985-9635-f447bfbc5e25 Bridge br-int fail_mode: secure Port "ovn-877214-0" Interface "ovn-877214-0" type: geneve options: {csum="true", key=flow, remote_ip="137.187.160.14"} error: "could not add network device ovn-877214-0 to ofproto (File exists)" Port "ovn-48e040-0" Interface "ovn-48e040-0" type: geneve options: {csum="true", key=flow, remote_ip="137.187.160.18"} Port "ovn-d6eaa1-0" Interface "ovn-d6eaa1-0" type: geneve options: {csum="true", key=flow, remote_ip="137.187.160.13"} error: "could not add network device ovn-d6eaa1-0 to ofproto (File exists)" Port br-int Interface br-int type: internal Port "ovn-f0f789-0" Interface "ovn-f0f789-0" type: geneve options: {csum="true", key=flow, remote_ip="137.187.160.13"} Second site:Harbor I upgraded to 4.3.2 and stopped any upgrades until I could deal with the migration and restarting VDSMD issues. Nodes are 3 supermicro 1U with 2x10G nics All 3 nodes are the same and from the 4.3.2 update. OS Version: RHEL - 7 - 6.1810.2.el7.centos OS Description: CentOS Linux 7 (Core) Kernel Version: 3.10.0 - 957.10.1.el7.x86_64 KVM Version: 2.12.0 - 18.el7_6.3.1 LIBVIRT Version: libvirt-4.5.0-10.el7_6.6 VDSM Version: vdsm-4.30.11-1.el7 SPICE Version: 0.14.0 - 6.el7_6.1 GlusterFS Version: [N/A] CEPH Version: librbd1-10.2.5-4.el7 Open vSwitch Version: openvswitch-2.10.1-3.el7 Kernel Features: PTI: 1, IBRS: 0, RETP: 1 VNC Encryption: Disabled 1 engine DL360 g10 with 2x10G nics running 4.3.2.1-1.el7 Storage: Dell Unity ISCSI The only issue with Harbor site while running 4.3.2 is that when I migrate a node it never finishes in the engine until I restart VDSMD on the original host. It did not exhibit this issue with 4.2.x

1 0

Undeploy oVirt Metrics Store
by toslavik＠yandex.ru 23 Apr '19

23 Apr '19

Deployed MetricStore on the engine and hosts as per the instructions. But using some time I realized that for me the functionality is redundant, enough data collected by DWH. https://www.ovirt.org/develop/release-management/features/metrics/metrics-s… Is there an instruction on how undeploy oVirt MetricStore?

2 4