Unable to add hosts to cluster & possible related routing issue
by David White
Hi all,
I used the oVirt installer via cockpit to setup a hyperconverged cluster on 3 physical hosts running Red Hat 8.3. I used the following two resources to guide my work:
- https://www.ovirt.org/documentation/gluster-hyperconverged/chap-Deploying...
- https://blogs.ovirt.org/2018/02/up-and-running-with-ovirt-4-2-and-gluster...
As I mentioned in a previous email a week ago, it appears as if the wizard successfully setup gluster on all 3 of the servers, and the wizard also successfully setup the ovirt engine on the first host.
However, the oVirt engine only recognizes the first host, and only the first host's physical resources are available to the cluster.
So I have Gluster 8 installed on the 3 hosts (RHEL 8), and oVirt 4.4.5 installed on the first host, along with the ovirt engine VM.
I run `gluster peer status` from the first node, can confirm that the other two physical hosts are healthy:
(example.com is replacing actual domain below)
[root@cha1-storage ~]# gluster peer statusNumber of Peers: 2Hostname: cha2-storage.mgt.example.comUuid: 240a7ab1-ab52-4e5b-98ed-d978f848835eState: Peer in Cluster (Connected)Hostname: cha3-storage.mgt.example.comUuid: 0563c3e8-237d-4409-a09a-ec51719b0da6State: Peer in Cluster (Connected)
I am now trying to get the other two hosts added to the Engine. I navigate to Compute -> Hosts, and click New, fill in the details (hostname, root password, etc..) and begin the installation on the additional hosts.
It keeps failing.
Checking the error logs in /var/log/ovirt-engine/host-deploy on the Engine VM, I see the following near the bottom:
"msg" : "Failed to download metadata for repo 'ovirt-4.4-epel': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=... [Failed to connect to mirrors.fedoraproject.org port 443: No route to host]",
I can confirm that the Engine and each of the hosts are able to get to mirrors.fedoraproject.org:
I've run the following on both the Engine, as well as each of the hosts (the first host where everything is installed, as well as the 2nd host where I'm trying to get it installed):
[root@ovirt-engine1 host-deploy]# curl -i https://mirrors.fedoraproject.org
HTTP/2 302
Note, that this traffic is going out the management network.
That may be an important distinction -- keep reading.
So this does lead me to another issue that may or may not be related.
I've discovered that, from the RHEL 8 host's perspective, the public facing network is unable to get out to the internet.
Management (Gluster & the Engine VM) is on: 10.1.0.0/24
Each of the hosts are able to ping each other and communicate with each other.
Each of the hosts are able to ping 8.8.8.8 whenever the frontend network interface is disabled (see below).
The frontend network is on: 192.168.0.0/24
Each of the hosts are able to ping each other on this network
But this network isn't able to get out to the internet (yet).
Obviously I need to fix the routing and figure out why the 192.168.0.0/24 network is unable to reach the internet.
But shouldn't all the traffic to install the ovirt functionality onto the 2nd host go out the management network?
So to summarize, I have a few questions:
- Why didn't the wizard properly install the ovirt functionality onto all 3 hosts to begin with when I did the initial installation?
- From the physical host's perspective, what should be the default route? The internal management, or the front-end network?
- Is the following statement accurate?
- The ovirt-engine's default route should be management -- it doesn't even have a front-end IP address.
- Why would the ovirt engine fail to install, and indicate that it cannot get to mirrors.fedoraproject.org, when clearly it can?
- Again, the ovirt-engine VM is able to curl that URL and gets a valid http response.
- Any other tips or suggestions on how to troubleshoot this?
Sent with ProtonMail Secure Email.
4 years
hosted engine vm don't boot after minor update
by a.cor＠outlook.com
Hello!
I have rhv 4.4.2 and NFS storage type for HE Domain Storage.
I updated to 4.4.4 by guide (https://www.ovirt.org/documentation/upgrade_guide/#Updating_a_self-hosted...) and reboot Hosted Engine VM. After this VM is not booting.
Connected to console I see next error:
BdsDxe: failed to load Boot0001 "UEFI Misc Device"...
After "Press any key" I got BIOS screen, go to Device Manager -> Drive Health Manager I see nothing.
I suggest that something happened with VM config.
Check /etc/ovirt-hosted-engine/hosted-engine.conf
All looks correctly (I checked UUID to Domain Storage and etc.)
Check /var/run/ovirt-hosted-engine-ha/vm.conf.
After encoding base64 I found something like this:
<source file="/rhev/data-center/0000-0000-0000-0000-0000/6916940d-b21b-4739-9d6b-4f7ff9e72a9e/images/9a17d31d-30f8-480d-a106-2fa7f0d48c40/a817c10b-0366-4c05-b512-e6beb8308b5d
All path is correct only zero after data-center/.
I changed it to real path, decode to base64 and try to start VM with hosted-engine --vm-start --vm-conf=PATH_TO_NEW_FILE.
Nothing. I see similar error.
Could someone help me in this case?
4 years
Hosted engine deployment fail because host version does not match cluster version
by emanuele.breuza＠e4company.com
Hello,
I'm Emanuele. I'm interested in setting up a 3 node HA oVirt cluster on RHEL 8.3. Following the oVirt Gluster Hyperconvergence guide, I managed to setup a GlusterFS storage and then I started the procedure to configure a self-hosted engine via cockpit. Unfortunately, I got stack at the "Prepare VM stage".
During this procedure, the automated process paused to let you verify inside the ovirt-engine that your host is OK. Since this is a brand new installation of oVirt, I have only one host configured and it is marked as not-good. The reason reported in the web page is that the host is compatible for v. 4.2, 4.3 and 4.4 but the Cluster is v. 4.5. I tried to execute the upgrade procedure on the host but it was unable to go into maintenance mode.
I tried both ovirt-release44 and ovirt-release-master RPM to load the latest repository but without success. Do you have any suggestion about how to fix the self-hosted engine procedure?
BTW, if I install my server with the oVirt node ISO the procedure works correctly and my host report also 4.5 into the compatibility version list.
Thank you very much for your help!!!
4 years
[ANN] oVirt 4.4.6 First Release Candidate is now available for testing
by Lev Veyde
oVirt 4.4.6 First Release Candidate is now available for testing
The oVirt Project is pleased to announce the availability of oVirt 4.4.6
First Release Candidate for testing, as of March 25th, 2021.
This update is the sixth in a series of stabilization updates to the 4.4
series.
How to prevent hosts entering emergency mode after upgrade from oVirt 4.4.1
Note: Upgrading from 4.4.2 GA or later should not require re-doing these
steps, if already performed while upgrading from 4.4.1 to 4.4.2 GA. These
are only required to be done once.
Due to Bug 1837864 <https://bugzilla.redhat.com/show_bug.cgi?id=1837864> -
Host enter emergency mode after upgrading to latest build
If you have your root file system on a multipath device on your hosts you
should be aware that after upgrading from 4.4.1 to 4.4.6 you may get your
host entering emergency mode.
In order to prevent this be sure to upgrade oVirt Engine first, then on
your hosts:
1.
Remove the current lvm filter while still on 4.4.1, or in emergency mode
(if rebooted).
2.
Reboot.
3.
Upgrade to 4.4.6 (redeploy in case of already being on 4.4.6).
4.
Run vdsm-tool config-lvm-filter to confirm there is a new filter in
place.
5.
Only if not using oVirt Node:
- run "dracut --force --add multipath” to rebuild initramfs with the
correct filter configuration
6.
Reboot.
Documentation
-
If you want to try oVirt as quickly as possible, follow the instructions
on the Download <https://ovirt.org/download/> page.
-
For complete installation, administration, and usage instructions, see
the oVirt Documentation <https://ovirt.org/documentation/>.
-
For upgrading from a previous version, see the oVirt Upgrade Guide
<https://ovirt.org/documentation/upgrade_guide/>.
-
For a general overview of oVirt, see About oVirt
<https://ovirt.org/community/about.html>.
Important notes before you try it
Please note this is a pre-release build.
The oVirt Project makes no guarantees as to its suitability or usefulness.
This pre-release must not be used in production.
Installation instructions
For installation instructions and additional information please refer to:
https://ovirt.org/documentation/
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 8.3 or newer
* CentOS Linux (or similar) 8.3 or newer
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures
for:
* Red Hat Enterprise Linux 8.3 or newer
* CentOS Linux (or similar) 8.3 or newer
* oVirt Node 4.4 based on CentOS Linux 8.3 (available for x86_64 only)
See the release notes [1] for installation instructions and a list of new
features and bugs fixed.
Notes:
- oVirt Appliance is already available for CentOS Linux 8
- oVirt Node NG is already available for CentOS Linux 8
- We found a few issues while testing on CentOS Stream so we are still
basing oVirt 4.4.6 Node and Appliance on CentOS Linux.
Additional Resources:
* Read more about the oVirt 4.4.6 release highlights:
http://www.ovirt.org/release/4.4.6/
* Get more oVirt project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:
http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.4.6/
[2] http://resources.ovirt.org/pub/ovirt-4.4-pre/iso/
--
Lev Veyde
Senior Software Engineer, RHCE | RHCVA | MCITP
Red Hat Israel
<https://www.redhat.com>
lev(a)redhat.com | lveyde(a)redhat.com
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
4 years
Ovirt Node (iso) - OK to enable Centos-BaseOS/Centos-Appstream repos ?
by morgan cox
Hi.
I have installed Ovirt nodes via the ovirt-node iso (centos8 based) - on a fresh install the standard CentOS repos are disabled (the ovirt 4-4 repo is enabled)
As part of our company hardening we need to install a few packages from the Centos repos.
Can I enable the CentOS-Linux-AppStream.repo + CentOS-Linux-BaseOS.repo repos or will this cause issues when we update the node ?
Thanks
4 years
Problem to create provider
by miguel.garcia＠toshibagcs.com
We have a problem to create the first provider in the ovirt cluster by following next steps:
Go to Administration - Provider - click Add
Filled up fields for new provider with next information:
Name: ovirt-provider-ovn
Type: External Network Provider
Network Plugin: oVirt Network Provider for OVN
Provider URL: https://ovirthostname:9696
Uncheck Read-only
Check Request authentication
username : admin@internal
Protocol HTTPS
Hostname ovirthostname
API port: 35357
Click Test
Result:
Test Failed (unknow error)
Log file:
2021-03-23 15:23:50,986-04 ERROR [org.ovirt.engine.core.bll.provider.network.openstack.ExternalNetworkProviderProxy] (default task-118) [501852ef-cd76-40d0-8ef3-6b0826b3c0dd] Failed to communicate with external provider 'ovirt-provider-ovn' due to error 'ConnectException: Connection refused (Connection refused)' [org.ovirt.engine.core.utils.servlet.ServletUtils] (default t2021-03-23 15:23:50,986-04 ERROR [org.ovirt.engine.core.bll.GetProviderCertificateChainQuery] (default task-118) [501852ef-cd76-40d0-8ef3-6b0826b3c0dd] Error in encoding certificate: EngineException: Connection refused (Connection refused) (Failed with error PROVIDER_FAILURE and code 5050)
I think I would expect to ask the certificate to complete connection but does not ask for certificate at all.
Any idea how to solve this?
4 years
VDSM command failed:Received fatal alert certificate_expire
by fmendoza＠red.com.sv
Recently Ovirt sent certificate expiration messages. And it does not let me manage the server VMs, I have looked for information about this and I can not find something that is clear. The version of Ovirt that I own is 3.6.3.4-1.el7.cents. I have at least 4 operational VMs of vital importance.
Thanks for the support
4 years
websockify + ovirt
by Pascal D
Hi,
I am trying to get the spice-web-client working with ovirt. One area where I am having difficulties is authentication.Looking at remote-viewer on linux I am able to see that the minimum fields to have a successful spice connection are the following:
[virt-viewer]
type=spice
host=70.xxx.176.xxx
port=5914
password=WQJQWCo+s8tK
tls-port=5915
tls-ciphers=DEFAULT
host-subject=O=xxxx.com,CN=d1c1v5.xxx.net
ca=-----BEGIN CERTIFICATE-----\nMIIDzDCCArSgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwSzELMAkGA1UEBhMCVVMxGDAWBgNVBAoM\nD2J1dHRlcmZseWl0LmNvbTEiMCAGA1UEAwwZb3YxLmJ1dHRlcmZseWl0LmNvbS40NTQ2NTAeFw0x\nOTA2MDQwMDMyMDVaFw0yOTA2MDIwMDMyMxxxxxxxxxxxxxxxxxxxxxxxxdXR0\nZXJmbHlpdC5jb20xIjAgBgNVBAMMGW92MS5idXR0ZXJmbHlpdC5jb20uNDU0NjUwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDD218EJkIJewgmeDFcUM7vEQ3RQ4nL9ZNEg+zORlruLKON\neZRDfgXei3XTt+VFUNTrxBjepf+yN3WjhVP+lDeDveZU/3OYKj9dSewlz7Mj1XTKE8DXDMIGYc79\nXUrcSoiEjCRG1eB+w+uyP4WK0AlJwGKav3AZuU5awjvYAftkW0RhOgdjp80ofuoC3K9TUPPjemtw\n3EWb4bjRcWiDUj8owfhhAHnb4RfacUSMQmYpVJ5YfRunYrCOixlOeGx7PkvXLqWmu2Rnrnk7TNn6\nv74fHh3ruHmZHLk2i6/yNoOAiJC/M8piCGZ3tiOcnPcYF2ZoX+Ud6BV69Hp6SxnF/eCXAgMBAAGj\ngbkwgbYwHQYDVR0OBBYEFAlrTpLGY5Dq6gtA7d7CXc1QAFmOMHQGA1UdIwRtMGuAFAlrTpLGY5Dq\n6gtA7d7CXc1QAFmOoU+kTTBLMQswCQYDVQQGEwJVUzEYMBYGA1UECgwPYnV0dGVyZmx5aXQuY29t\nMSIwIAYDVQQDDBlvdjEuYnV0dGVyZmx5aXQuY29tLjQ1NDY1ggIQADAPBgNVHRMBAf8EBTADAQH/\nMA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCA
QEAoC8Nx/s4Uafgc3iyzxbLPb/chQ8U\n7+lULXTq+ZLOuMDdu6UKt7qKZpJZK8ZhjFh/1yVOnpzm7Np+oP7TQlOUkup8X4HsfAwrCgNK1IT1\nETdbdMYD8HYFjxz/0xbnMkJAHfPEh1vtqplw3YhVgiAZfZfT8HzVY/xGkjurvxSyVjBSbn+4uao1\n6W9URt2rWTHn+XxoT+j+cx8vv1WKsynlMBtUjCFy8eR7ZDngRcM/9iRkRCGHJvWJmi1CRrQeE5RZ\nvBH0zE64J3cOJj4BSlN3wOYWiRq28XLB9epDDyZaRpnsqLCOq/+/LscM7iPW1acdCoCu68nJUwTQ\nh1Jh7vQjCQ==\n-----END CERTIFICATE-----\n
with this I can successfully connect to a vm. Now I would like to do the same from spice-web-client but websockify doesn't give me a tls-port. How to could I implement this? Is there a wrapper that exists that I can pass to websockify to do the authentication on the port + 1 (it seems it is always the next port)
Thanks in advance for your help
4 years
VM listed as unmanaged external
by miguel.garcia＠toshibagcs.com
We have a vm that got renamed with prefix "external-*" and in the status of resources for this vm are as unmanaged.
Is there a way to turn it back to manage vm and rollback how it was before?
4 years
supervdsm failing during network_caps
by Alan G
Hi,
I have issues with one host where supervdsm is failing in network_caps.
I see the following trace in the log.
MainProcess|jsonrpc/1::ERROR::2020-01-06 03:01:05,558::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) Error in network_caps
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 98, in wrapper
res = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 56, in network_caps
return netswitch.configurator.netcaps(compatibility=30600)
File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 317, in netcaps
net_caps = netinfo(compatibility=compatibility)
File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 325, in netinfo
_netinfo = netinfo_get(vdsmnets, compatibility)
File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/cache.py", line 150, in get
return _stringify_mtus(_get(vdsmnets))
File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/cache.py", line 59, in _get
ipaddrs = getIpAddrs()
File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/addresses.py", line 72, in getIpAddrs
for addr in nl_addr.iter_addrs():
File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/addr.py", line 33, in iter_addrs
with _nl_addr_cache(sock) as addr_cache:
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/__init__.py", line 92, in _cache_manager
cache = cache_allocator(sock)
File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/libnl.py", line 469, in rtnl_addr_alloc_cache
raise IOError(-err, nl_geterror(err))
IOError: [Errno 16] Message sequence number mismatch
A restart of supervdsm will resolve the issue for a period, maybe 24 hours, then it will occur again. So I'm thinking it's resource exhaustion or a leak of some kind?
Running 4.2.8.2 with VDSM at 4.20.46.
I've had a look through the bugzilla and can't find an exact match, closest was this one https://bugzilla.redhat.com/show_bug.cgi?id=1666123 which seems to be a RHV only fix.
Thanks,
Alan
4 years
