Engine and host certificates expired
by simon＠justconnect.ie
I have an environment where the engine wouldn’t start and the certificate expiry dates were as follows.
Host1 - 25th Sep 2022
Host2 - 11th Aug 2022
Host3 - 11th Aug 2022
I copied the vdsm certs from Host1 to Host2 & Host3
Engine then started on Host1 and then backed up.
Engine cert expiry 11th Aug 2022
I put the cluster into Global Maintenance mode and then tried:
‘engine-setup —offline’
Which failed as the validation check said the engine wasn’t in Global Maintenance mode even though ‘hosted-engine —vm-status said it was.
None of the Hosts are ‘GREEN’ (can’t remember what the status was as I’m writing this from memory) but their status is ’RED’.
There are VMs running on the 3 Hosts and I’m reluctant to restart anything at the moment.
Is there a way to refresh the engine certificate to get past this or do I need to restart vdsm service on each host to bring them back online as far as the engine is concerned?
The environment is currently at 4.4.6 and is to be upgraded to 4.5.2 next month.
Any help as always will be greatly appreciated.
Kind regards
Simon
2 years, 2 months
VDSNetworkException: protocol version not accepted by client
by Kilian Ries
Hi,
im running an oVirt 4.2.8 cluster with two nodes. A few days ago, my SSL certificates expired. After that, i changed all the certificates on the engine via "engine-upgrade" command and issued new vdsm client certificates. Then i copied the new certificates to my ovirt nodes and restarted vdsmd (systemctl restart vdsmd).
Now i'm still not able to connect to my ovirt nodes. In the engine log i can see the following error:
###
2022-09-01 18:25:51,822+02 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to /192.168.xx.xx
2022-09-01 18:25:51,827+02 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages The server selected protocol version TLS10 is not accepted by client preferences [TLS12]
2022-09-01 18:25:51,829+02 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-88) [] Unable to RefreshCapabilities: VDSNetworkException: VDSGenericException: VDSNetworkException: The server selected protocol version TLS10 is not accepted by client preferences [TLS12]
###
I searched my vdsm client config but i cannot see any specific TLS version set (every option with TLS is commented - seems to be the default):
###
$grep -R -i TLS /etc/vdsm/
/etc/vdsm/vdsm.conf:# ssl_protocol = tlsv1
/etc/vdsm/vdsm.conf:# https://docs.python.org/2/library/ssl.html. e.g. OP_NO_TLSv1,
/etc/vdsm/vdsm.conf:# OP_NO_TLSv1_1 By default tlv1, tlsv1.1 and tlsv1.2 are enabled.
###
On the engine i didn't find any setting to set a specific TLS version - there seems to have been a setting (VdsmSSLProtocol) but that got deprecated years ago.
Does anybody know why my engine is still not able to connect to the client vdsmd?
I also tried to set "ssl_protocol = tlsv1" via vdsm.conf but that didn't work ...
Thanks
Regards,
Kilian
PS:
Name : vdsm
Architektur : x86_64
Version : 4.19.37
Ausgabe : 1.el7.centos
Name : ovirt-engine
Architektur : noarch
Version : 4.2.8.2
Ausgabe : 1.el7
2 years, 2 months
oVirt 4.5.2 - ovirt-hosted-engine-setup fails with "error: Must be number, not str"}]" when creating ovirtmgmt network
by Thomas Simmons
Hello,
I am trying to deploy the latest oVirt (4.5.2), on a fully patched Rocky
8.6 system and am having and issue where "ovirt-hosted-engine-setup" is
failing when it tries to create the ovirtmgmt network with the error
"error: Must be number, not str"}]". When this happens, the engine setup
pauses and if I can login to the bootstrap engine UI and when I attempt to
manually assign the ovirtmgmt network to the correct nic on the host, I get
the same error message. This server has (2) active network interfaces - a
gigabit NIC that will be a VM network for all networks except gluster and
migration and a 40Gbps Infiniband adapter in connected mode (IPoIB) for
gluster and migration (I previously had these servers in the same hardware
configuration running oVirt 4.3 on CentOS 7 and would like to have the same
setup again - just with latest versions of EL and oVirt).
I don't believe it's related, however for transparency I should note that
the server is running kernel-lt from elrepo (5.4.212-1.el8.elrepo.x86_64)
because both native EL and elrepo support for my Infiniband HBA was dropped
in the standard EL8 kernel due to known bugs with that version of the
kernel. Thanks in advance for any assistance.
Here is the specific error from engine.log on the bootstrap engine. I see
similar messages in vdsm.log on the host.
2022-09-04 18:01:10,725-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] START,
HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com,
HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10',
vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]',
rollbackOnFailure='true', commitOnSuccess='true',
connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true',
bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt',
nicName='enp3s0', vlan='null', vmNetwork='true', stp='false',
properties='null', ipv4BootProtocol='STATIC_IP',
ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0',
ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]',
removedNetworks='[]', bonds='[]', removedBonds='[]',
clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id:
6bc2c376
2022-09-04 18:01:10,726-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] FINISH,
HostSetupNetworksVDSCommand, return: , log id: 6bc2c376
2022-09-04 18:01:11,251-04 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"Attempt to call function: <bound method Global.setupNetworks of
<vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt':
{'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True,
'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess':
True, 'connectivityCheck': 'true'}) error: Must be number, not str"}]
2022-09-04 18:01:11,252-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Failed in
'HostSetupNetworksVDS' method
2022-09-04 18:01:11,252-04 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"Attempt to call function: <bound method Global.setupNetworks of
<vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt':
{'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True,
'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess':
True, 'connectivityCheck': 'true'}) error: Must be number, not str"}]
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM vmh1.my.domain.com command
HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': "Attempt
to call function: <bound method Global.setupNetworks of <vdsm.API.Global
object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck':
'true'}) error: Must be number, not str"}
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Error:
VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS,
error = Internal JSON-RPC error: {'reason': "Attempt to call function:
<bound method Global.setupNetworks of <vdsm.API.Global object at
0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command
'HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com,
HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10',
vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]',
rollbackOnFailure='true', commitOnSuccess='true',
connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true',
bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt',
nicName='enp3s0', vlan='null', vmNetwork='true', stp='false',
properties='null', ipv4BootProtocol='STATIC_IP',
ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0',
ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]',
removedNetworks='[]', bonds='[]', removedBonds='[]',
clusterSwitchType='LEGACY', managementNetworkChanged='true'})' execution
failed: VDSGenericException: VDSErrorException: Failed to
HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt
to call function: <bound method Global.setupNetworks of <vdsm.API.Global
object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603
2022-09-04 18:01:11,262-04 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Host setup networks
finished. Lock released. Monitoring can run now for host 'vmh1.my.domain.com'
from data-center 'Default'
2022-09-04 18:01:11,262-04 ERROR
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command
'org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand' failed:
EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS,
error = Internal JSON-RPC error: {'reason': "Attempt to call function:
<bound method Global.setupNetworks of <vdsm.API.Global object at
0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603 (Failed with error
unexpected and code 16)
2022-09-04 18:01:11,263-04 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Lock freed to object
'EngineLock:{exclusiveLocks='[HOST_NETWORK1def9b77-b268-4a64-bac0-3e51c1d16b10=HOST_NETWORK]',
sharedLocks=''}'
2022-09-04 18:01:11,269-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID:
SETUP_NETWORK_FAILED_FOR_MANAGEMENT_NETWORK_CONFIGURATION(1,120), Failed to
configure management network on host vmh1.my.domain.com due to setup
networks failure.
2 years, 2 months
VM has been paused due to no Storage space error on ovirt 4.5
by Jirka Simon
Hi there,
last days we are facing issues with paused VMs (in past it was for few
second to resize lv device), but now it doesn't resume. we migrated to
4.5.2 cluster, this never happened before with the same storage.
there is almost notning in engine log
2022-09-06 09:47:11,160+02 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-9) [51eb7178] VM
'cfff0648-6502-4977-95a8-c6f95c723f6d'(cm1.util.prod.hq.slde
v.cz) moved from 'Up' --> 'Paused'
2022-09-06 09:47:11,264+02 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-9) [51eb7178] EVENT_ID: VM_PAUSED(1,025), VM
cm1.util.prod.hq.
sldev.cz has been paused.
2022-09-06 09:47:11,271+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-9) [51eb7178] EVENT_ID: VM_PAUSED_ENOSPC(138), VM
cm1.util.pro
d.hq.sldev.cz has been paused due to no Storage space error.
but there are erros with LVM in vdsmlog. (attached)
ovirt 4.5.2
OS - ovirt-node-ng.
Thank you for any hint.
Jirka
2 years, 2 months
storage domain
by izidor.stokin＠gmail.com
Hello,
One of our admins accidentally added one of available LUN disks to storage domain. Now we are not able to remove it as additional storage from it.
There was no data written on it.
on host it was removed from vg group (vgreduce) , but on site it is still in use (greyed out) (vgreduce had no issues removing pvdisk from it)
multipath -ll shows it as unused,
dmsetup status shows some values related to this lun:
OS: rhel 7.9
package ovirt 4.3
cluster: yes
Any idea how to remove it without corrupting datadomain ?
is it safe to remove it with dmsetum remove?
BRI
2 years, 2 months
Some problems with ovirt
by אריה קלטר
Hi,
I have two ovirt clusters.
1. 4.3 cluster with single node (was 3 nodes in the past)
2. 4.5.2 cluster with 3 nodes. the problem was also on oVirt-4.5 node with a matching engine version but the cluster configured on version 4.6 because I did have a problem upgrading one of the nodes, I also lost the engine because of gluster volume sync issue between the hosts, so I formatted the problematic node using the 4.5.2 node iso and created a new engine on it, and attached 2 of the existing nodes and did upgrade on the nodes using the new engine.
About the new cluster:
1. migration between hosts - almost all the time looks like it succeeds, but the VM is not responding after the successful migration.
2. run reboot inside of the VM - in a high percent of the time it just hung instead of reboot. With 100% CPU. In status "Reboot in Progress", but after some time the status changed to "Up" incorrectly. I tried on the clean images of ubuntu and centos from ovirt-image-repository. I imported a server from the 4.3 cluster to the new cluster, on the old cluster it reboots correctly but in the new cluster it doesn't. I tried it both with gluster and with nfs.
Any idea how to solve both problems?
Regards, Arye
2 years, 2 months
Re: How many oVirt cluster, hosts and VMs do you have running?
by Jonas Lindholm
I was looking at those studies but not sure how old they are.
What I can see is that our oVirt deployment seems bigger than any of those
studies but I can be wrong.
------ Original Message ------
Received: 01:08 AM MDT, 09/02/2022
From: Sandro Bonazzola <sbonazzo(a)redhat.com>
To: jlhm(a)usa.net Cc: oVirt Users <users(a)ovirt.org>
Subject:
Re: [ovirt-users] How many oVirt cluster, hosts and VMs do you have running?
> Would something like an oVirt counter help?
> https://github.com/oVirt/ovirt-site/pull/3056
>
>
> Il giorno ven 2 set 2022 alle ore 08:26 ha scritto:
>
> > Hi, just trying to understand how our oVirt deployment compare to others.
> > - 75 cluster (and 75 Data centers as we map them 1 to 1 due to security
> > requirements) spanning +3 data centers
> > - 328 oVirt servers
> > - +1900 VMs running
> >
> > Majority are still on 4.3 (CentOS 7) but our engines (4 of them) runs
> > RedHat 8/oVirt 4.4. Working to upgrade all hypervisors to RedHat 8 (or
> > Rocky 8)/oVirt 4.4.
> > When that is done we will start to upgrade to latest oVirt version but it
> > takes time due to the size of our environment and we move slowly to
ensure
> > stability.
> > _______________________________________________
> > Users mailing list -- users(a)ovirt.org
> > To unsubscribe send an email to users-leave(a)ovirt.org
> > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > oVirt Code of Conduct:
> > https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> >
https://lists.ovirt.org/archives/list/users@ovirt.org/message/LM3GRRJDXVM...
> >
>
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D PERFORMANCE & SCALE
>
> Red Hat EMEA
>
> sbonazzo(a)redhat.com
>
>
> *Red Hat respects your work life balance. Therefore there is no need to
> answer this email out of your office hours.*
>
2 years, 2 months
How many oVirt cluster, hosts and VMs do you have running?
by jlhm＠usa.net
Hi, just trying to understand how our oVirt deployment compare to others.
- 75 cluster (and 75 Data centers as we map them 1 to 1 due to security requirements) spanning +3 data centers
- 328 oVirt servers
- +1900 VMs running
Majority are still on 4.3 (CentOS 7) but our engines (4 of them) runs RedHat 8/oVirt 4.4. Working to upgrade all hypervisors to RedHat 8 (or Rocky 8)/oVirt 4.4.
When that is done we will start to upgrade to latest oVirt version but it takes time due to the size of our environment and we move slowly to ensure stability.
2 years, 2 months
