September 2020 - Users - oVirt List Archives

supervdsm failing during network_caps
by Alan G 24 Mar '21

24 Mar '21

Hi, I have issues with one host where supervdsm is failing in network_caps. I see the following trace in the log. MainProcess|jsonrpc/1::ERROR::2020-01-06 03:01:05,558::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) Error in network_caps Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 98, in wrapper res = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 56, in network_caps return netswitch.configurator.netcaps(compatibility=30600) File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 317, in netcaps net_caps = netinfo(compatibility=compatibility) File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 325, in netinfo _netinfo = netinfo_get(vdsmnets, compatibility) File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/cache.py", line 150, in get return _stringify_mtus(_get(vdsmnets)) File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/cache.py", line 59, in _get ipaddrs = getIpAddrs() File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/addresses.py", line 72, in getIpAddrs for addr in nl_addr.iter_addrs(): File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/addr.py", line 33, in iter_addrs with _nl_addr_cache(sock) as addr_cache: File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/__init__.py", line 92, in _cache_manager cache = cache_allocator(sock) File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/libnl.py", line 469, in rtnl_addr_alloc_cache raise IOError(-err, nl_geterror(err)) IOError: [Errno 16] Message sequence number mismatch A restart of supervdsm will resolve the issue for a period, maybe 24 hours, then it will occur again. So I'm thinking it's resource exhaustion or a leak of some kind? Running 4.2.8.2 with VDSM at 4.20.46. I've had a look through the bugzilla and can't find an exact match, closest was this one https://bugzilla.redhat.com/show_bug.cgi?id=1666123 which seems to be a RHV only fix. Thanks, Alan

2 3

OVN and change of mgmt network
by Gianluca Cecchi 26 Jan '21

26 Jan '21

Hello, I previously had OVN running on engine (as OVN provider with northd and northbound and southbound DBs) and hosts (with OVN controller). After changing mgmt ip of hosts (engine has retained instead the same ip), I executed again on them the command: vdsm-tool ovn-config <ip_of_engine> <nel_local_ip_of_host> Now I think I have to clean up some things, eg: 1) On engine where I get these lines below systemctl status ovn-northd.service -l . . . Sep 29 14:41:42 ovmgr1 ovsdb-server[940]: ovs|00005|reconnect|ERR|tcp: 10.4.167.40:37272: no response to inactivity probe after 5 seconds, disconnecting Oct 03 11:52:00 ovmgr1 ovsdb-server[940]: ovs|00006|reconnect|ERR|tcp: 10.4.167.41:52078: no response to inactivity probe after 5 seconds, disconnecting The two IPs are the old ones of two hosts It seems that a restart of the services has fixed... Can anyone confirm if I have to do anything else? 2) On hosts (there are 3 hosts with OVN on ip 10.4.192.32/33/34) where I currently have this output [root@ov301 ~]# ovs-vsctl show 3a38c5bb-0abf-493d-a2e6-345af8aedfe3 Bridge br-int fail_mode: secure Port "ovn-1dce5b-0" Interface "ovn-1dce5b-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.32"} Port "ovn-ddecf0-0" Interface "ovn-ddecf0-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.33"} Port "ovn-fd413b-0" Interface "ovn-fd413b-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.168.74"} Port br-int Interface br-int type: internal ovs_version: "2.7.2" [root@ov301 ~]# The IPs of kind 10.4.192.x are ok. But there is a left-over of an old host I initially used for tests, corresponding to 10.4.168.74, that now doesn't exist anymore How can I clean records for 1) and 2)? Thanks, Gianluca

3 4

CentOS Stream support
by Michal Skrivanek 25 Jan '21

25 Jan '21

Hi all, we would like to ask about interest in community about oVirt moving to CentOS Stream. There were some requests before but it’s hard to see how many people would really like to see that. With CentOS releases lagging behind RHEL for months it’s interesting to consider moving to CentOS Stream as it is much more up to date and allows us to fix bugs faster, with less workarounds and overhead for maintaining old code. E.g. our current integration tests do not really pass on CentOS 8.1 and we can’t really do much about that other than wait for more up to date packages. It would also bring us closer to make oVirt run smoothly on RHEL as that is also much closer to Stream than it is to outdated CentOS. So..would you like us to support CentOS Stream? We don’t really have capacity to run 3 different platforms, would you still want oVirt to support CentOS Stream if it means “less support” for regular CentOS? There are some concerns about Stream being a bit less stable, do you share those concerns? Thank you for your comments, michal

5 8

encrypted GENEVE traffic
by Pavel Nakonechnyi 18 Jan '21

18 Jan '21

Dear oVirt Community, From my understanding oVirt does not support Open vSwitch IPSEC tunneling for GENEVE traffic (which is described on pages http://docs.openvswitch.org/en/latest/howto/ipsec/ and http://docs.openvswitch.org/en/latest/tutorials/ipsec/) Are there plans to introduce such support? (or explicitly not to..) Is it possible to somehow manually configure such tunneling for existing virtual networks? (even in a limited way) Alternatively, is it possible to deploy oVirt on top of the tunneled (i.e. via VXLAN, IPSec) interfaces? This will allow to encrypt all management traffic. Such requirement arises when using oVirt deployment on third-party premises with untrusted network. Thank in advance for any clarifications. :) -- WBR, Pavel +32478910884

2 7

oVirt 4.4: Self-hosted engine deployment fails with backup restore from 4.3 engine
by Oliver Leinfelder 11 Dec '20

11 Dec '20

Hi there, I'm a bit puzzled about an possible upgrade paths from a 4.3 cluster to version 4.4 in a self-hosted engine environment. My idea was: Set up a new host with a clean ovirt node 4.4 installation, then deploy the hosted engine on this with a restored backup from the production cluster and go from there. This however fails with the following error: 2020-05-27 00:17:08,886+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:103 {'msg': 'non-zero return code', 'cmd': ['engine-setup', '--accept-defaults', '--config-append=/root/ovirt-engine-answers'], 'stdout': "[ INFO ] Stage: Initializing\n[ INFO ] Stage: Environment setup\n C onfiguration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf, /root/ovirt-engine-answers\n Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20200527001657-fyeueu.log\n Version: otop i-1.9.1 (otopi-1.9.1-1.el8)\n[ INFO ] DNF Downloading 1 files, 0.00KB\n[ INFO ] DNF Downloaded CentOS-8 - AppStream\n[ INFO ] DNF Downloading 1 files, 0.00KB\n[ INFO ] DNF Downloaded CentOS-8 - Base\n[ INFO ] DNF Downloading 1 files, 0.00KB\n [...] ... anwsers from backup config follow .... [...] 2020-05-27 00:17:12,396+0200 DEBUG otopi.context context._executeMethod:145 method exception Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 403, in _closeup r = ah.run() File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 229, in run raise RuntimeError(_('Failed executing ansible-playbook')) Is this approach (restoring from 4.3) generally supposed to work? If not, what is the appropriate upgrade path? Thank you! Regards Oli

3 7

Can't import some VMs after storage domain detach and reattach to new datacenter.
by m black 09 Dec '20

09 Dec '20

7 14

"gluster-ansible-roles is not installed on Host" error on Cockpit
by Hesham Ahmed 26 Nov '20

26 Nov '20

On a new 4.3.1 oVirt Node installation, when trying to deploy HCI (also when trying adding a new gluster volume to existing clusters) using Cockpit, an error is displayed "gluster-ansible-roles is not installed on Host. To continue deployment, please install gluster-ansible-roles on Host and try again". There is no package named gluster-ansible-roles in the repositories: [root@localhost ~]# yum install gluster-ansible-roles Loaded plugins: enabled_repos_upload, fastestmirror, imgbased-persist, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade This system is not registered with an entitlement server. You can use subscription-manager to register. Loading mirror speeds from cached hostfile * ovirt-4.3-epel: mirror.horizon.vn No package gluster-ansible-roles available. Error: Nothing to do Uploading Enabled Repositories Report Cannot upload enabled repos report, is this client registered? This is due to check introduced here: https://gerrit.ovirt.org/#/c/98023/1/dashboard/src/helpers/AnsibleUtil.js Changing the line from: [ "rpm", "-qa", "gluster-ansible-roles" ], { "superuser":"require" } to [ "rpm", "-qa", "gluster-ansible" ], { "superuser":"require" } resolves the issue. The above code snippet is installed at /usr/share/cockpit/ovirt-dashboard/app.js on oVirt node and can be patched by running "sed -i 's/gluster-ansible-roles/gluster-ansible/g' /usr/share/cockpit/ovirt-dashboard/app.js && systemctl restart cockpit"

2 1

ovirt-imageio-proxy not working after updating SSL certificates with a wildcard cert issued by AlphaSSL (intermediate)
by Lynn Dixon 22 Nov '20

22 Nov '20

All, I recently bought a wildcard certificate for my lab domain (shadowman.dev) and I replaced all the certs on my RHV4.3 machine per our documentation. The WebUI presents the certs successfully and without any issues, and everything seemed to be fine, until I tried to upload a disk image (or an ISO) to my storage domain. I get this error in the events tab: https://share.getcloudapp.com/p9uPvegx [image: image.png] I also see that the disk is showing up in my storage domain, but its showing "Paused by System" and I can't do anything with it. I cant even delete it! I have tried following this document to fix the issue, but it didn't work: https://access.redhat.com/solutions/4148361 I am seeing this error pop into my engine.log: https://pastebin.com/kDLSEq1A And I see this error in my image-proxy.log: WARNING 2020-07-24 15:26:34,802 web:137:web:(log_error) ERROR [172.17.0.30] PUT /tickets/ [403] Error verifying signed ticket: Invalid ovirt ticket (data='------my_ticket_data-----', reason=Untrusted certificate) [request=0.002946/1] Now, when I bought my wildcard, I was given a root certificate for the CA, as well as a separate intermediate CA certificate from the provider. Likewise, they gave me a certificate and a private key of course. The root and intermediate CA's certificates have been added to /etc/pki/ca-trust/source/anchors/ and I did an update-ca-trust. I also started experiencing issues with the ovpn network provider at the same time I replaced the SSL certs, but I disregarded it at the time, but now I am thinking its related. Any advice on what to look for to fix the ovirt-imageio-proxy? Thanks! *Lynn Dixon* | Red Hat Certified Architect #100-006-188 *Solutions Architect* | NA Commercial Google Voice: 423-618-1414 Cell/Text: 423-774-3188 Click here to view my Certification Portfolio <http://red.ht/1XMX2Mi>

6 8

Single Node HCI upgrade procedure from CentOS7/oVirt 4.3 to CentOS8/oVirt 4.4?
by thomas＠hoberg.net 11 Nov '20

11 Nov '20

I can hear you saying: "You did understand that single node HCI is just a toy, right?" For me the primary use of a single node HCI is adding some disaster resilience in small server edge type scenarios, where a three node HCI provides the fault tolerance: 3+1 with a bit of distance, warm or even cold stand-by, potentially manual switch and reduced workload in case disaster strikes. Of course, another 3nHCI would be better, but who gets that type of budget, right? What I am trying say: If you want oVirt to gain market share, try to give HCI more love. And while you're at it, try to make expanding from 1nHCI to 3nHCI (and higher counts) a standard operational procedure to allow expanding a disaster stand-by into a production setup, while the original 3nHCI is being rebuilt. For me low-budget HCI is where oVirt has its biggest competitive advantage against vSan and Nutanix, so please don't treat the HCI/gluster variant like an unwanted child any more. In the mean-time OVA imports (from 4.3.10 exports) on my 4.4.2 1nHCI fail again, which I'll report separately.

2 1

Problem with Cluster-wise BIOS Settings in oVirt 4.4
by Rodrigo G. López 10 Nov '20

10 Nov '20

Hi all, We are running an oVirt 4.4 Hosted Engine as a VM, and after changing the Cluster's BIOS type from Q35 with Legacy BIOS (the default one after installation) to Preexistent, the VM fails with the following error: XML error: The device at PCI address 0000:00:02.0 cannot be plugged into the PCI controller with index='0'. It requires a controller that accepts a pcie-root-port. We need it so that we can run imported VMs from a previous version of oVirt, namely 4.0. Applying the BIOS settings individually works but as an attempt to generalize the settings we decided to apply it to the full cluster. Tell me if you need more data. Cheers, -rodri

4 6

POWER9 (ppc64le) Support on oVirt 4.4.1
by Vinícius Ferrão 03 Nov '20

03 Nov '20

Hello, I was using oVirt 4.3.10 with IBM AC922 (POWER9 / ppc64le) without any issues. Since I’ve moved to 4.4.1 I can’t add the AC922 machine to the engine anymore, it complains with the following error: The host CPU does not match the Cluster CPU type and is running in degraded mode. It is missing the following CPU flags: model_POWER9, powernv. Any ideia of what’s may be happening? The engine runs on x86_64, and I was using this way on 4.3.10. Machine info: timebase : 512000000 platform : PowerNV model : 8335-GTH machine : PowerNV 8335-GTH firmware : OPAL MMU : Radix Thanks,

3 18

Gluster Domain Storage full
by suporte＠logicworks.pt 27 Oct '20

27 Oct '20

Hello, I'm running oVirt Version 4.3.4.3-1.el7. I have a small GlusterFS Domain storage brick on a dedicated filesystem serving only one VM. The VM filled all the Domain storage. The Linux filesystem has 4.1G available and 100% used, the mounted brick has 0GB available and 100% used I can not do anything with this disk, for example, if I try to move it to another Gluster Domain Storage get the message: Error while executing action: Cannot move Virtual Disk. Low disk space on Storage Domain Any idea? Thanks -- Jose Ferradeira http://www.logicworks.pt

2 12

adminstration portal wont complete load, looping
by Philip Brown 26 Oct '20

26 Oct '20

I have an odd situation: When I go to https://ovengine/ovirt-engine/webadmin/?locale=en_US after authentication passes... it shows the top banner of oVirt OPEN VIRTUALIZATION MANAGER and the Loading ... in the center. but never gets past that. Any suggestions on how I could investigate and fix this? background: I recently updated certs to be signed wildcard certs, but this broke consoles somehow. So I restored the original certs, and restarted things... but got stuck with this. Interestingly, the VM portal loads fine. But not the admin portal. -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

3 8

Latest ManagedBlockDevice documentation
by Michael Thomas 15 Oct '20

15 Oct '20

I'm looking for the latest documentation for setting up a Managed Block Device storage domain so that I can move some of my VM images to ceph rbd. I found this: https://ovirt.org/develop/release-management/features/storage/cinderlib-int… ...but it has a big note at the top that it is "...not user documentation and should not be treated as such." The oVirt administration guide[1] does not talk about managed block devices. I've found a few mailing list threads that discuss people setting up a Managed Block Device with ceph, but didn't see any links to documentation steps that folks were following. Is the Managed Block Storage domain a supported feature in oVirt 4.4.2, and if so, where is the documentation for using it? --Mike [1]ovirt.org/documentation/administration_guide/

4 19

ovirt broker how to modify network_test dns ip address?
by wodel youchi 11 Oct '20

11 Oct '20

Hi, How to modify the ip address of the DNS server used by ovirt-broker for testing? Regards.

2 1

Moving VM disks from one storage domain to another. Automate?
by Green, Jacob Allen /C 08 Oct '20

08 Oct '20

I am looking for an automated way, via Ansible to move a VM disk from one storage domain to another. I found the following, https://docs.ansible.com/ansible/latest/modules/ovirt_disk_module.html and while it mentions copying a VM disk image from one domain to another it does not mention a live storage migration. Which is what I am looking to do. I want to take roughly 100 VMs and move their disk images from one domain to another that is available to the datacenter in some automated/scripted fashion. I am just curious if anyone out there has had to do this and how they tackled it. Or perhaps I am missing some easy obvious way, other than clicking all the disks and clicking move. However from the looks of it, if I did click all the disk and selected move, it appears RHV tries to do them all at once, which is probably not ideal, I would like it to move the disks in a serial One after another fashion, to conserve throughput and IO. I also did not see anything on Ansible galaxy or the ovirt github that would do this. Thank you.

3 2

oVirt Survey Autumn 2020
by Sandro Bonazzola 06 Oct '20

06 Oct '20

As we continue to develop oVirt 4.4, the Development and Integration teams at Red Hat would value insights on how you are deploying the oVirt environment. Please help us to hit the mark by completing this short survey. The survey will close on October 18th 2020. If you're managing multiple oVirt deployments with very different use cases or very different deployments you can consider answering this survey multiple times. *Please note the answers to this survey will be publicly accessible*. This survey is under oVirt Privacy Policy available at https://www.ovirt.org/site/privacy-policy.html . The survey is available https://forms.gle/bPvEAdRyUcyCbgEc7 -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

3 4

oVirt Node 4.4.2 is now generally available
by Sandro Bonazzola 06 Oct '20

06 Oct '20

oVirt Node 4.4.2 is now generally available The oVirt project is pleased to announce the general availability of oVirt Node 4.4.2 , as of September 25th, 2020. This release completes the oVirt 4.4.2 release published on September 17th Important notes before you install / upgrade Please note that oVirt 4.4 only supports clusters and data centers with compatibility version 4.2 and above. If clusters or data centers are running with an older compatibility version, you need to upgrade them to at least 4.2 (4.3 is recommended). Please note that in RHEL 8 / CentOS 8 several devices that worked on EL7 are no longer supported. For example, the megaraid_sas driver is removed. If you use Enterprise Linux 8 hosts you can try to provide the necessary drivers for the deprecated hardware using the DUD method (See the users’ mailing list thread on this at https://lists.ovirt.org/archives/list/users@ovirt.org/thread/NDSVUZSESOXEFJ… ) How to prevent hosts entering emergency mode after upgrade from oVirt 4.4.1 Due to Bug 1837864 <https://bugzilla.redhat.com/show_bug.cgi?id=1837864> - Host enter emergency mode after upgrading to latest build If you have your root file system on a multipath device on your hosts you should be aware that after upgrading from 4.4.1 to 4.4.2 you may get your host entering emergency mode. In order to prevent this be sure to upgrade oVirt Engine first, then on your hosts: 1. Remove the current lvm filter while still on 4.4.1, or in emergency mode (if rebooted). 2. Reboot. 3. Upgrade to 4.4.2 (redeploy in case of already being on 4.4.2). 4. Run vdsm-tool config-lvm-filter to confirm there is a new filter in place. 5. Only if not using oVirt Node: - run "dracut --force --add multipath” to rebuild initramfs with the correct filter configuration 6. Reboot. Documentation - If you want to try oVirt as quickly as possible, follow the instructions on the Download <https://ovirt.org/download/> page. - For complete installation, administration, and usage instructions, see the oVirt Documentation <https://ovirt.org/documentation/>. - For upgrading from a previous version, see the oVirt Upgrade Guide <https://ovirt.org/documentation/upgrade_guide/>. - For a general overview of oVirt, see About oVirt <https://ovirt.org/community/about.html>. What’s new in oVirt Node 4.4.2 Release? oVirt Node has been updated, including: - oVirt 4.4.2: http://www.ovirt.org/release/4.4.2/ - Ansible 2.9.13: https://github.com/ansible/ansible/blob/stable-2.9/changelogs/CHANGELOG-v2.… - Glusterfs 7.7: https://docs.gluster.org/en/latest/release-notes/7.7/ - Advanced Virtualization 8.2.1 See the release notes [1] for installation instructions and a list of new features and bugs fixed. Additional resources: - Read more about the oVirt 4.4.2 release highlights: http://www.ovirt.org/release/4.4.2/ - Get more oVirt project updates on Twitter: https://twitter.com/ovirt - Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.4.2/ [2] http://resources.ovirt.org/pub/ovirt-4.4/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

6 27

Re: OVN Geneve tunnels not been established
by Konstantinos Betsis 06 Oct '20

06 Oct '20

I did a restart of the ovn-controller, this is the output of the ovn-controller.log 2020-09-11T10:54:07.566Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log 2020-09-11T10:54:07.568Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2020-09-11T10:54:07.568Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2020-09-11T10:54:07.570Z|00004|main|INFO|OVS IDL reconnected, force recompute. 2020-09-11T10:54:07.571Z|00005|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: connecting... 2020-09-11T10:54:07.571Z|00006|main|INFO|OVNSB IDL reconnected, force recompute. 2020-09-11T10:54:07.685Z|00007|stream_ssl|WARN|SSL_connect: unexpected SSL connection close 2020-09-11T10:54:07.685Z|00008|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: connection attempt failed (Protocol error) 2020-09-11T10:54:08.685Z|00009|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: connecting... 2020-09-11T10:54:08.800Z|00010|stream_ssl|WARN|SSL_connect: unexpected SSL connection close 2020-09-11T10:54:08.800Z|00011|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: connection attempt failed (Protocol error) 2020-09-11T10:54:08.800Z|00012|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: waiting 2 seconds before reconnect 2020-09-11T10:54:10.802Z|00013|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: connecting... 2020-09-11T10:54:10.917Z|00014|stream_ssl|WARN|SSL_connect: unexpected SSL connection close 2020-09-11T10:54:10.917Z|00015|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: connection attempt failed (Protocol error) 2020-09-11T10:54:10.917Z|00016|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: waiting 4 seconds before reconnect 2020-09-11T10:54:14.921Z|00017|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: connecting... 2020-09-11T10:54:15.036Z|00018|stream_ssl|WARN|SSL_connect: unexpected SSL connection close 2020-09-11T10:54:15.036Z|00019|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: connection attempt failed (Protocol error) 2020-09-11T10:54:15.036Z|00020|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: continuing to reconnect in the background but suppressing further logging I have also done the vdsm-tool ovn-config OVIRT_ENGINE_IP OVIRTMGMT_NETWORK_DC This is how the OVIRT_ENGINE_IP is provided in the ovn controller, i can redo it if you wan. After the restart of the ovn-controller the OVIRT ENGINE still shows only two geneve connections one with DC01-host02 and DC02-host01. Chassis "c4b23834-aec7-4bf8-8be7-aa94a50a6144" hostname: "dc02-host01" Encap geneve ip: "DC02-host01_IP" options: {csum="true"} Chassis "be3abcc9-7358-4040-a37b-8d8a782f239c" hostname: "DC01-host02" Encap geneve ip: "DC01-host02" options: {csum="true"} I've re-done the vdsm-tool command and nothing changed.... again....with the same errors as the systemctl restart ovn-controller On Fri, Sep 11, 2020 at 1:49 PM Dominik Holler <dholler(a)redhat.com> wrote: > Please include ovirt-users list in your reply, to share the knowledge and > experience with the community! > > On Fri, Sep 11, 2020 at 12:12 PM Konstantinos Betsis <k.betsis(a)gmail.com> > wrote: > >> Ok below the output per node and DC >> DC01 >> node01 >> >> [root@dc01-node01 ~]# ovs-vsctl --no-wait get open . >> external-ids:ovn-remote >> "ssl:*OVIRT_ENGINE_IP*:6642" >> [root@ dc01-node01 ~]# ovs-vsctl --no-wait get open . >> external-ids:ovn-encap-type >> geneve >> [root@ dc01-node01 ~]# ovs-vsctl --no-wait get open . >> external-ids:ovn-encap-ip >> >> "*OVIRTMGMT_IP_DC01-NODE01*" >> >> node02 >> >> [root@dc01-node02 ~]# ovs-vsctl --no-wait get open . >> external-ids:ovn-remote >> "ssl:*OVIRT_ENGINE_IP*:6642" >> [root@ dc01-node02 ~]# ovs-vsctl --no-wait get open . >> external-ids:ovn-encap-type >> geneve >> [root@ dc01-node02 ~]# ovs-vsctl --no-wait get open . >> external-ids:ovn-encap-ip >> >> "*OVIRTMGMT_IP_DC01-NODE02*" >> >> DC02 >> node01 >> >> [root@dc02-node01 ~]# ovs-vsctl --no-wait get open . >> external-ids:ovn-remote >> "ssl:*OVIRT_ENGINE_IP*:6642" >> [root@ dc02-node01 ~]# ovs-vsctl --no-wait get open . >> external-ids:ovn-encap-type >> geneve >> [root@ dc02-node01 ~]# ovs-vsctl --no-wait get open . >> external-ids:ovn-encap-ip >> >> "*OVIRTMGMT_IP_DC02-NODE01*" >> >> > Looks good. > > >> DC01 node01 and node02 share the same VM networks and VMs deployed on top >> of them cannot talk to VM on the other hypervisor. >> > > Maybe there is a hint on ovn-controller.log on dc01-node02 ? Maybe > restarting ovn-controller creates more helpful log messages? > > You can also try restart the ovn configuration on all hosts by executing > vdsm-tool ovn-config OVIRT_ENGINE_IP LOCAL_OVIRTMGMT_IP > on each host, this would trigger > > https://github.com/oVirt/ovirt-provider-ovn/blob/master/driver/scripts/setu… > internally. > > >> So I would expect to see the same output for node01 to have a geneve >> tunnel to node02 and vice versa. >> >> > Me too. > > >> On Fri, Sep 11, 2020 at 12:14 PM Dominik Holler <dholler(a)redhat.com> >> wrote: >> >>> >>> >>> On Fri, Sep 11, 2020 at 10:53 AM Konstantinos Betsis <k.betsis(a)gmail.com> >>> wrote: >>> >>>> Hi Dominik >>>> >>>> OVN is selected as the default network provider on the clusters and the >>>> hosts. >>>> >>>> >>> sounds good. >>> This configuration is required already during the host is added to oVirt >>> Engine, because OVN is configured during this step. >>> >>> >>>> The "ovn-sbctl show" works on the ovirt engine and shows only two >>>> hosts, 1 per DC. >>>> >>>> Chassis "c4b23834-aec7-4bf8-8be7-aa94a50a6144" >>>> hostname: "dc01-node02" >>>> Encap geneve >>>> ip: "X.X.X.X" >>>> options: {csum="true"} >>>> Chassis "be3abcc9-7358-4040-a37b-8d8a782f239c" >>>> hostname: "dc02-node1" >>>> Encap geneve >>>> ip: "A.A.A.A" >>>> options: {csum="true"} >>>> >>>> >>>> The new node is not listed (dc01-node1). >>>> >>>> When executed on the nodes the same command (ovn-sbctl show) times-out >>>> on all nodes..... >>>> >>>> The output of the /var/log/openvswitch/ovn-conntroller.log lists on all >>>> logs >>>> >>>> 2020-09-11T08:46:55.197Z|07361|stream_ssl|WARN|SSL_connect: unexpected >>>> SSL connection close >>>> >>>> >>>> >>> Can you please compare the output of >>> >>> ovs-vsctl --no-wait get open . external-ids:ovn-remote >>> ovs-vsctl --no-wait get open . external-ids:ovn-encap-type >>> ovs-vsctl --no-wait get open . external-ids:ovn-encap-ip >>> >>> of the working hosts, e.g. dc01-node02, and the failing host dc01-node1? >>> This should point us the relevant difference in the configuration. >>> >>> Please include ovirt-users list in your replay, to share the knowledge >>> and experience with the community. >>> >>> >>> >>>> Thank you >>>> Best regards >>>> Konstantinos Betsis >>>> >>>> >>>> On Fri, Sep 11, 2020 at 11:01 AM Dominik Holler <dholler(a)redhat.com> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Thu, Sep 10, 2020 at 6:26 PM Konstantinos B <k.betsis(a)gmail.com> >>>>> wrote: >>>>> >>>>>> Hi all >>>>>> >>>>>> We have a small installation based on OVIRT 4.3. >>>>>> 1 Cluster is based on Centos 7 and the other on OVIRT NG Node image. >>>>>> >>>>>> The environment was stable till an upgrade took place a couple of >>>>>> months ago. >>>>>> As such we had to re-install one of the Centos 7 node and start from >>>>>> scratch. >>>>>> >>>>> >>>>> To trigger the automatic configuration of the host, it is required to >>>>> configure ovirt-provider-ovn as the default network provider for the >>>>> cluster before adding the host to oVirt. >>>>> >>>>> >>>>>> Even though the installation completed successfully and VMs are >>>>>> created, the following are not working as expected: >>>>>> 1. ovn geneve tunnels are not established with the other Centos 7 >>>>>> node in the cluster. >>>>>> 2. Centos 7 node is configured by ovirt engine however no geneve >>>>>> tunnel is established when "ovn-sbctl show" is issued on the engine. >>>>>> >>>>> >>>>> Does "ovn-sbctl show" list the hosts? >>>>> >>>>> >>>>>> 3. no flows are shown on the engine on port 6642 for the ovs db. >>>>>> >>>>>> Does anyone have any experience on how to troubleshoot OVN on ovirt? >>>>>> >>>>>> >>>>> /var/log/openvswitch/ovncontroller.log on the host should contain a >>>>> helpful hint. >>>>> >>>>> >>>>> >>>>>> Thank you >>>>>> _______________________________________________ >>>>>> Users mailing list -- users(a)ovirt.org >>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: >>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/LBVGLQJBWJF3E… >>>>>> >>>>>

3 30

Is it possible to change scheduler optimization settings of cluster using ansible or some other automation way
by Kushagra Agarwal 02 Oct '20

02 Oct '20

I was hoping if i can get some help with the below oVirt scenario: *Problem Statement*:- Is it possible to change scheduler optimization settings of cluster using ansible or some other automation way *Description*:- Do we have any ansible module or any other CLI based approach which can help us to change 'scheduler optimization' settings of cluster in oVIrt. Scheduler optimization settings of cluster can be found under Scheduling Policy tab ( Compute -> Clusters(select the cluster) -> click on edit and then navigate to scheduling policy Any help in this will be highly appreciated. Thanks, Kushagra

3 7

VM AutoStart
by Jeremey Wise 01 Oct '20

01 Oct '20

When I have to shut down cluster... ups runs out etc.. I need a sequence set of just a small number of VMs to "autostart" Normally I just use DNS FQND to connect to oVirt engine but as two of my VMs are a DNS HA cluster.. as well as NTP / SMTP /DHCP etc... I need those two infrastructure VMs to be auto boot. I looked at HA settings for those VMs but it seems to be watching for pause /resume.. but it does not imply or state auto start on clean first boot. Options? -- p <jeremey.wise(a)gmail.com>enguinpages

4 12

Power on VM - CLI / API
by Jeremey Wise 01 Oct '20

01 Oct '20

Can anyone post link. (with examples.. as most documentation for oVirt lacks this).. where I can power on a VM via CLI or API. As of now I cannot login to oVirt-Engine. No errors when I restart it.. https://lists.ovirt.org/archives/list/users@ovirt.org/thread/45KKF5TN5PRQ3R… BUt... I need to get VMs booted. I tried to follow: http://ovirt.github.io/ovirt-engine-api-model/master/ and my server's API web portal https://ovirte01.penguinpages.local/ovirt-engine/apidoc/#/documents/003_com… And.. even get POSTMAN (real newbie at that tool but ran into how to add exported .pem key from portal to session issues) # failed CLI example: Power on VM "ns01" ########### <<<< DRAFT :: 2020-09-30################ # Get key from oVirt engine and import. Ex: from ovirte01 into server 'thor curl -k ' https://ovirte01.penguinpages.local/ovirt-engine/services/pki-resource?reso…' -o ovirt01_ca.pem sudo cp ovirt01_ca.pem /etc/pki/ca-trust/source/anchors sudo update-ca-trust extract openssl s_client -connect ovirte01.penguinpages.local:443 -showcerts < /dev/null <snip... lots of key info back> # Use key during GET list of VMs ?? <meh? # Power on VM <<fubar still> curl -X POST https://ovirte01.penguinpages.local/post -H /ovirt-engine/api/vms/ns01/start HTTP/1.1 <https://ovirte01.penguinpages.local/post%20-H%20/ovirt-engine/api/vms/ns01/…> ##### I just need to power on VM -- p <jeremey.wise(a)gmail.com>enguinpages

3 2

java.lang.reflect.UndeclaredThrowableException - oVirt engine UI
by Jeremey Wise 01 Oct '20

01 Oct '20

I tried to post on website but .. it did not seem to work... so sorry if this is double posting. oVirt login this AM. accepted username and password but got java error. Restarted oVirt engine ## hosted-engine --set-maintenance --mode=global hosted-engine --vm-shutdown hosted-engine --vm-status #make sure that the status is shutdown before restarting hosted-engine --vm-start hosted-engine --vm-status #make sure the status is health before leaving maintenance mode hosted-engine --set-maintenance --mode=none ## [root@thor ~]# hosted-engine --vm-status --== Host thor.penguinpages.local (id: 1) status ==-- Host ID : 1 Host timestamp : 65342 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : thor.penguinpages.local Local maintenance : False stopped : False crc32 : 824c29fd conf_on_shared_storage : True local_conf_timestamp : 65342 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=65342 (Wed Sep 30 08:11:45 2020) host-id=1 score=3400 vm_conf_refresh_time=65342 (Wed Sep 30 08:11:45 2020) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host medusa.penguinpages.local (id: 3) status ==-- Host ID : 3 Host timestamp : 87556 Score : 3400 Engine status : {"vm": "up", "health": "good", "detail": "Up"} Hostname : medusa.penguinpages.local Local maintenance : False stopped : False crc32 : 63296a70 conf_on_shared_storage : True local_conf_timestamp : 87556 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=87556 (Wed Sep 30 08:11:39 2020) host-id=3 score=3400 vm_conf_refresh_time=87556 (Wed Sep 30 08:11:39 2020) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False [root@thor ~]# yum update -y Last metadata expiration check: 0:31:17 ago on Wed 30 Sep 2020 09:17:03 AM EDT. Dependencies resolved. Nothing to do. Complete! [root@thor ~]# Gogled around .. just found this thread. ## https://bugzilla.redhat.com/show_bug.cgi?id=1378045 # pgadmin connect to ovirte01.penguinpages.com as engine to db engine select mac_addr from vm_interface "00:16:3e:57:0d:47" "56:6f:86:41:00:01" "56:6f:86:41:00:00" "56:6f:86:41:00:02" "56:6f:86:41:00:03" "56:6f:86:41:00:04" "56:6f:86:41:00:05" "56:6f:86:41:00:15" "56:6f:86:41:00:16" "56:6f:86:41:00:17" "56:6f:86:41:00:18" "56:6f:86:41:00:19" # Note one field is "null" Question: 1) is this bad? 2) How do I fix? 3) Any idea on root cause? -- p <jeremey.wise(a)gmail.com>enguinpages

3 2

Gluster Volumes - Correct Peer Connection
by Jeremey Wise 30 Sep '20

30 Sep '20

I just noticed when HCI setup bult the gluster engine / data / vmstore volumes... it did use correctly the definition of 10Gb "back end" interfaces / hosts. But.. oVirt Engine is NOT referencing this. it lists bricks as 1Gb "managment / host" interfaces. Is this a GUI issue? I doubt this and how do I correct it? ### Data Volume Example Name: data Volume ID: 0ae7b487-8b87-4192-bd30-621d445902fe Volume Type: Replicate Replica Count: 3 Number of Bricks: 3 Transport Types: TCP Maximum no of snapshots: 256 Capacity: 999.51 GiB total, 269.02 GiB used, 730.49 GiB free, 297.91 GiB Guaranteed free, 78 Deduplication/Compression savings (%) medusa.penguinpages.local medusa.penguinpages.local:/gluster_bricks/data/data 25% OK odin.penguinpages.local odin.penguinpages.local:/gluster_bricks/data/data 25% OK thor.penguinpages.local thor.penguinpages.local:/gluster_bricks/data/data 25% OK # I have storage back end of 172.16.101.x which is 10Gb dedicated for replication. Peers reflect this [root@odin c4918f28-00ce-49f9-91c8-224796a158b9]# gluster peer status Number of Peers: 2 Hostname: thorst.penguinpages.local Uuid: 7726b514-e7c3-4705-bbc9-5a90c8a966c9 State: Peer in Cluster (Connected) Hostname: medusast.penguinpages.local Uuid: 977b2c1d-36a8-4852-b953-f75850ac5031 State: Peer in Cluster (Connected) [root@odin c4918f28-00ce-49f9-91c8-224796a158b9]# -- p <jeremey.wise(a)gmail.com>enguinpages

5 7

Version 4.4.2.6-1.el8 -Console Error: java.lang.reflect.UndeclaredThrowableException
by penguin pages 30 Sep '20

30 Sep '20

Got message this AM when tried to login to oVirt Engine which up till now has been working fine. I can supply username and password and get portal to choose "Administration Portal" or "VM Portal" I have tested both.. both have same response about java.lang.reflect.UndeclaredThrowableException I restarted the engine # hosted-engine --set-maintenance --mode=global hosted-engine --vm-shutdown hosted-engine --vm-status #make sure that the status is shutdown before restarting hosted-engine --vm-start hosted-engine --vm-status #make sure the status is health before leaving maintenance mode hosted-engine --set-maintenance --mode=none # --== Host thor.penguinpages.local (id: 1) status ==-- Host ID : 1 Host timestamp : 70359 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : thor.penguinpages.local Local maintenance : False stopped : False crc32 : 25adf6d0 conf_on_shared_storage : True local_conf_timestamp : 70359 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=70359 (Wed Sep 30 09:35:22 2020) host-id=1 score=3400 vm_conf_refresh_time=70359 (Wed Sep 30 09:35:22 2020) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host medusa.penguinpages.local (id: 3) status ==-- Host ID : 3 Host timestamp : 92582 Score : 3400 Engine status : {"vm": "up", "health": "good", "detail": "Up"} Hostname : medusa.penguinpages.local Local maintenance : False stopped : False crc32 : 623359d2 conf_on_shared_storage : True local_conf_timestamp : 92582 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=92582 (Wed Sep 30 09:35:25 2020) host-id=3 score=3400 vm_conf_refresh_time=92582 (Wed Sep 30 09:35:25 2020) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False ## I downloaded and installed key from portal.. thinking that may have been issue.. it was not. I googled around /searched forum and nothing jumped out. (only hit I found in forum https://lists.ovirt.org/pipermail/users/2015-June/033421.html but no note about fix)

2 2