7:06 p.m.
Hi everyone,
We have had a pair of Ovirt clusters at work, starting with the 3.x. I
replaced last year with 4.x cluster on new machines. 4.2 worked great, but
when I upgraded to 4.3.2 and now .3 I immediately ran into host networking
issues resulting in hung or unable totally to migrate VMs.
Configuration notes:
1. I use commercial star.x.x.x SSL cert on engine installed per Ovirt.org
instructions
2. I have 2 untagged NICs assigned to ovirtmgmt and Public, both on the
same IP range network. I also have another set of VLANs tagged on various
NICs. 1 cluster uses HP blades with lots of NICs, the other uses supermicro
1Us with only 2 NICs.
3. I upgraded both engines and hosts to 4.3.2 from 4.2.8. I started seeing
hung migrations cleared by restarting vdsm.
4. All computers involved run current CentOS 7.6
5. Both clusters use iscsi on dedicated tagged storage VLAN, seems fine.
6. I upgraded 1 engine and 1 host to 4.3.3 and things got worse. I have not
updated any hosts or the second engine since.
7. The 4 hosts on the 4.3.3 engine now have out of sync error that refuses
to clear for my Public network.
8. 2 OV 4.3.2 nodes had the following errors with 2 ovn ports. Here is
example, perhaps related to the genev_sys_6081 error.
ovn-d6eaa1-0: attempting to add tunnel port with same config as port
'ovn-f0f789-0' (::->137.187.160.13
ovn-877214-0: attempting to add tunnel port with same config as port
'ovn-483528-0'
9. I deleted and uninstalled all ovirt related rpms on one node, then did
clean node install using latest 4.3 release. Same errors.
10. I downloaded latest node iso, installed on same host, upgraded to to
4.3.3.1 node and joined cluster. The node installation has no errors and
can migrate to my other hosts. All networks are in sync. The migration
hangs. Restarting VDSMD clears the hung migration.
11. None of the other hosts can migrate VMs to the new node.
Here are excerpts from 3.2 node on 3.3 engine log file.
Apr 17, 2019, 2:43:04 PM
Check for available updates on host BRCVN3 was completed successfully with
message 'ovirt-host, cockpit-ovirt-dashboard, vdsm-client, ovirt-release43,
ovirt-host, ovirt-hosted-engine-setup, vdsm-api, vdsm-jsonrpc,
ovirt-ansible-hosted-engine-setup, ovirt-host-dependencies and 12 others.
To see all packages check engine.log.'.
oVirt
Apr 17, 2019, 2:43:04 PM
Host BRCVN3 has available updates: ovirt-host, cockpit-ovirt-dashboard,
vdsm-client, ovirt-release43, ovirt-host, ovirt-hosted-engine-setup,
vdsm-api, vdsm-jsonrpc, ovirt-ansible-hosted-engine-setup,
ovirt-host-dependencies and 12 others. To see all packages check
engine.log..
oVirt
Apr 17, 2019, 2:27:39 PM
Host BRCVN3's following network(s) are not synchronized with their Logical
Network configuration: Public.
oVirt
Apr 16, 2019, 5:53:05 PM
Failed to sync all host BRCVN3 networks
c1ba631c-7be5-4be0-abbb-a37b6bb7d26d
oVirt
Apr 16, 2019, 5:53:05 PM
(1/1): Failed to apply changes on host BRCVN3. (User: admin@internal-authz)
455a260f
oVirt
Apr 16, 2019, 5:53:05 PM
VDSM BRCVN3 command HostSetupNetworksVDS failed: Bridge Public has
interfaces set([u'vnet19', u'vnet12', u'vnet10',
u'vnet11', u'vnet16',
u'vnet17', u'vnet14', u'vnet15', u'vnet0',
u'vnet2', u'vnet3', u'vnet4',
u'vnet7', u'vnet8', u'vnet9']) connected
oVirt
Apr 16, 2019, 5:52:59 PM
(1/1): Applying network's changes on host BRCVN3. (User:
admin@internal-authz)
455a260f
oVirt
Apr 16, 2019, 5:32:07 PM
Check for available updates on host BRCVN3 was completed successfully with
message 'no updates found.'.
oVirt
Apr 16, 2019, 5:16:40 PM
Host BRCVN3's following network(s) are not synchronized with their Logical
Network configuration: Public.
oVirt
Apr 16, 2019, 3:33:36 PM
Migration failed (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4).
bd21f294-0170-47d1-845e-c64c52abbab4
oVirt
Apr 16, 2019, 3:33:36 PM
Migration started (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4, User:
admin@internal-authz).
bd21f294-0170-47d1-845e-c64c52abbab4
oVirt
Apr 16, 2019, 2:49:36 PM
Migration failed (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4).
7c04f7a6-8f88-4c64-99f9-e621724fc7ff
oVirt
Apr 16, 2019, 2:49:36 PM
Migration started (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4, User:
admin@internal-authz).
7c04f7a6-8f88-4c64-99f9-e621724fc7ff
oVirt
Apr 16, 2019, 2:22:03 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 2:11:02 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 2:10:02 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 2:05:46 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 2:03:16 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 2:02:16 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 2:00:16 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 1:59:01 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 1:57:31 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 1:57:05 PM