Hi everyone,

We have had a pair of Ovirt clusters at work, starting with the 3.x. I replaced last year with 4.x cluster on new machines. 4.2 worked great, but when I upgraded to 4.3.2 and now .3 I immediately ran into host networking issues resulting in hung or unable totally to migrate VMs.

Configuration notes:

1. I use commercial star.x.x.x SSL cert on engine installed per Ovirt.org instructions

2. I have 2 untagged NICs assigned to ovirtmgmt and Public, both on the same IP range network. I also have another set of VLANs tagged on various NICs. 1 cluster uses HP blades with lots of NICs, the other uses supermicro 1Us with only 2 NICs. 

3. I upgraded both engines and hosts to 4.3.2 from 4.2.8. I started seeing hung migrations cleared by restarting vdsm. 

4. All computers involved run current CentOS 7.6

5. Both clusters use iscsi on dedicated tagged storage VLAN, seems fine.

6. I upgraded 1 engine and 1 host to 4.3.3 and things got worse. I have not updated any hosts or the second engine since.

7. The 4 hosts on the 4.3.3 engine now have out of sync error that refuses to clear for my Public network.

8. 2 OV 4.3.2 nodes had the following errors with 2 ovn ports.  Here is example, perhaps related to the genev_sys_6081 error.

ovn-d6eaa1-0: attempting to add tunnel port with same config as port 'ovn-f0f789-0' (::->137.187.160.13

ovn-877214-0: attempting to add tunnel port with same config as port 'ovn-483528-0'

9. I deleted and uninstalled all ovirt related rpms on one node, then did clean node install using latest 4.3 release. Same errors.

10. I downloaded latest node iso, installed on same host, upgraded to to 4.3.3.1 node and joined cluster. The node installation has no errors and can migrate to my other hosts. All networks are in sync. The migration hangs. Restarting VDSMD clears the hung migration.

11. None of the other hosts can migrate VMs to the new node.

Here are excerpts from 3.2 node on 3.3 engine log file. 

	Apr 17, 2019, 2:43:04 PM	Check for available updates on host BRCVN3 was completed successfully with message 'ovirt-host, cockpit-ovirt-dashboard, vdsm-client, ovirt-release43, ovirt-host, ovirt-hosted-engine-setup, vdsm-api, vdsm-jsonrpc, ovirt-ansible-hosted-engine-setup, ovirt-host-dependencies and 12 others. To see all packages check engine.log.'.		oVirt
	Apr 17, 2019, 2:43:04 PM	Host BRCVN3 has available updates: ovirt-host, cockpit-ovirt-dashboard, vdsm-client, ovirt-release43, ovirt-host, ovirt-hosted-engine-setup, vdsm-api, vdsm-jsonrpc, ovirt-ansible-hosted-engine-setup, ovirt-host-dependencies and 12 others. To see all packages check engine.log..		oVirt
	Apr 17, 2019, 2:27:39 PM	Host BRCVN3's following network(s) are not synchronized with their Logical Network configuration: Public.		oVirt
	Apr 16, 2019, 5:53:05 PM	Failed to sync all host BRCVN3 networks	c1ba631c-7be5-4be0-abbb-a37b6bb7d26d	oVirt
	Apr 16, 2019, 5:53:05 PM	(1/1): Failed to apply changes on host BRCVN3. (User: admin@internal-authz)	455a260f	oVirt
	Apr 16, 2019, 5:53:05 PM	VDSM BRCVN3 command HostSetupNetworksVDS failed: Bridge Public has interfaces set([u'vnet19', u'vnet12', u'vnet10', u'vnet11', u'vnet16', u'vnet17', u'vnet14', u'vnet15', u'vnet0', u'vnet2', u'vnet3', u'vnet4', u'vnet7', u'vnet8', u'vnet9']) connected		oVirt
	Apr 16, 2019, 5:52:59 PM	(1/1): Applying network's changes on host BRCVN3. (User: admin@internal-authz)	455a260f	oVirt
	Apr 16, 2019, 5:32:07 PM	Check for available updates on host BRCVN3 was completed successfully with message 'no updates found.'.		oVirt
	Apr 16, 2019, 5:16:40 PM	Host BRCVN3's following network(s) are not synchronized with their Logical Network configuration: Public.		oVirt
	Apr 16, 2019, 3:33:36 PM	Migration failed (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4).	bd21f294-0170-47d1-845e-c64c52abbab4	oVirt
	Apr 16, 2019, 3:33:36 PM	Migration started (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4, User: admin@internal-authz).	bd21f294-0170-47d1-845e-c64c52abbab4	oVirt
	Apr 16, 2019, 2:49:36 PM	Migration failed (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4).	7c04f7a6-8f88-4c64-99f9-e621724fc7ff	oVirt
	Apr 16, 2019, 2:49:36 PM	Migration started (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4, User: admin@internal-authz).	7c04f7a6-8f88-4c64-99f9-e621724fc7ff	oVirt
	Apr 16, 2019, 2:22:03 PM	VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'}		oVirt
	Apr 16, 2019, 2:11:02 PM	VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'}		oVirt
	Apr 16, 2019, 2:10:02 PM	VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'}		oVirt
	Apr 16, 2019, 2:05:46 PM	VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'}		oVirt
	Apr 16, 2019, 2:03:16 PM	VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'}		oVirt
	Apr 16, 2019, 2:02:16 PM	VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'}		oVirt
	Apr 16, 2019, 2:00:16 PM	VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'}		oVirt
	Apr 16, 2019, 1:59:01 PM	VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'}		oVirt
	Apr 16, 2019, 1:57:31 PM	VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'}		oVirt
	Apr 16, 2019, 1:57:05 PM

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/U6HCH5YDEOKFZEN4TYRFNNBRNHXGWFSR/