[ovirt-users] new host addition to OVN cluster fails with Connectivity check failed, rolling back

Hello, I have a a oVirt 4.3 installation with two clusters. One of the cluster has switch type as OVS. I'm trying to add a second host to this cluster. I did a clean install of the OS, configured bond0 and bond0.2306 as the VLAN interface. I was able to add the host to the cluster. When I go to setup networks and drag ovirtmgmt onto bond0, I notice in the host that it was able to create the vdsm and br_int bridges, create ovirtmgmt interface and assign the IP on top of it. However I also notice that the bond0.2306 interface also exists with the ip assigned. Then it rolls back the config removing the bridges. I checked the supervdsm log and see that it's rolling back because "connectivity::48::root::(check) Connectivity check failed, rolling back" I'm pasting the relevant lines from supervdsm below MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:53,381::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return network_caps with {'bridges': {}, 'bondings': {'bond0': {'ipv4addrs': [], 'active_slave': '', 'ad_aggregator_id': '1', 'netmask': '', 'ad_partner_mac': '44:38:39:ff:01:33', 'hwaddr': '7c:d3:0a:60:e9:48', 'speed': 20000, 'gateway': '', 'ipv6autoconf': True, 'addr': '', 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '9000', 'dhcpv4': False, 'switch': 'legacy', 'ipv4defaultroute': False, 'slaves': ['eno1', 'eno2'], 'ipv6gateway': '::', 'opts': {'mode': '4'}}}, 'nameservers': ['10.222.0.6', '10.333.0.6'], 'nics': {'eno1': {'permhwaddr': '7c:d3:0a:60:e9:48', 'ipv6autoconf': True, 'addr': '', 'speed': 10000, 'dhcpv6': False, 'ipv6addrs': [], 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': '7c:d3:0a:60:e9:48', 'mtu': '9000', 'ipv6gateway': '::', 'gateway': ''}, 'eno2': {'permhwaddr': '7c:d3:0a:60:e9:49', 'ipv6autoco nf': True, 'addr': '', 'speed': 10000, 'dhcpv6': False, 'ipv6addrs': [], 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': '7c:d3:0a:60:e9:48', 'mtu': '9000', 'ipv6gateway': '::', 'gateway': ''}, 'eno3': {'ipv6autoconf': True, 'addr': '', 'speed': 10000, 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': '7c:d3:0a:60:e9:4a', 'ipv6gateway': '::', 'gateway': ''}, 'eno4': {'ipv6autoconf': True, 'addr': '', 'speed': 10000, 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': '7c:d3:0a:60:e9:4b', 'ipv6gateway': '::', 'gateway': ''}, 'enp0s20f0u1u6': {'ipv6autoconf': True, 'addr': '', 'speed': 0, 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': '7e:d3:0a:60:e9:4f', 'ipv6gateway': ' ::', 'gateway': ''}}, 'supportsIPv6': True, 'vlans': {'bond0.2306': {'iface': 'bond0', 'ipv6autoconf': True, 'addr': '10.119.6.237', 'dhcpv6': False, 'ipv6addrs': [], 'vlanid': 2306, 'mtu': '9000', 'dhcpv4': False, 'netmask': '255.255.255.0', 'ipv4defaultroute': True, 'ipv4addrs': ['10.119.6.237/24'], 'ipv6gateway': '::', 'gateway': '10.119.6.1'}}, 'networks': {}} MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,243::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call get_pti with () {} MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,243::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return get_pti with -1 MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,244::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call get_retp with () {} MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,244::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return get_retp with -1 MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,244::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call get_ibrs with () {} MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,245::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return get_ibrs with 1 MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,245::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call get_ssbd with () {} MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,245::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return get_ssbd with -1 MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,246::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call check_qemu_conf_contains with ('vnc_tls', '1') {} MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,250::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return check_qemu_conf_contains with True MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,304::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({u'ovirtmgmt': {u'ipv6autoconf': True, u'vlan': u'2306', u'ipaddr': u'10.119.6.237', u'switch': u'ovs', u'mtu': 9000, u'bonding': u'bond0', u'dhcpv6': False, u'STP': u'no', u'bridged': u'true', u'netmask': u'255.255.255.0', u'gateway': u'10.119.6.1', u'defaultRoute': True}}, {}, {u'connectivityCheck': u'true', u'connectivityTimeout': 120, u'commitOnSuccess': False}) {} MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:55,305::api::211::root::(setupNetworks) Setting up network according to configuration: networks:{u'ovirtmgmt': {u'ipv6autoconf': True, u'vlan': u'2306', u'ipaddr': u'10.119.6.237', u'bonding': u'bond0', u'mtu': 9000, u'switch': u'ovs', u'dhcpv6': False, u'STP': u'no', u'bridged': u'true', u'netmask': u'255.255.255.0', u'gateway': u'10.119.6.1', u'defaultRoute': True}}, bondings:{}, options:{u'connectivityCheck': u'true', u'connectivityTimeout': 120, u'commitOnSuccess': False} MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,323::cmdutils::133::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,342::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,408::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,408::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,445::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:55,459::netconfpersistence::69::root::(setBonding) Adding bond0({'nics': ['eno1', 'eno2'], 'switch': 'legacy', 'options': 'mode=4'}) MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:55,460::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({u'ipv6autoconf': True, 'nameservers': ['10.222.0.6', '10.333.0.6'], u'vlan': 2306, u'ipaddr': u'10.119.6.237', u'switch': u'ovs', u'mtu': 9000, u'bonding': u'bond0', u'dhcpv6': False, 'stp': False, u'bridged': True, u'netmask': u'255.255.255.0', u'gateway': u'10.119.6.1', u'defaultRoute': True, 'bootproto': 'none'}) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,462::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-95 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,909::commands::219::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:55,909::hooks::114::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err= MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,911::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,911::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,947::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,953::configurator::265::root::(_remove_networks) Removing networks: [] MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,955::setup::41::root::(remove_bonds) Removing bonds: [] MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,958::ifacquire::70::root::(acquire) Acquiring ifaces: set([u'eno1', u'eno2', u'bond0']) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,959::cmdutils::133::root::(exec_cmd) /sbin/ifdown eno1 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:56,606::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:56,609::cmdutils::133::root::(exec_cmd) /sbin/ifdown eno2 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:57,268::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:57,271::cmdutils::133::root::(exec_cmd) /sbin/ifdown bond0 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:58,940::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:58,940::setup::61::root::(edit_bonds) Editing bonds: [u'bond0'] MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:58,946::netconfpersistence::69::root::(setBonding) Adding bond0({'nics': [], 'switch': u'ovs', 'options': {u'mode': '4'}}) MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:58,949::sysfs_driver::104::root::(set_options) Bond bond0 options set: {u'mode': u'4'}. MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:58,950::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev eno1 down (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:58,956::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:59,000::sysfs_driver::78::root::(add_slaves) Slave eno1 has been added to bond bond0. MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,001::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev eno2 down (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,009::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:59,053::sysfs_driver::78::root::(add_slaves) Slave eno2 has been added to bond bond0. MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:59,053::netconfpersistence::69::root::(setBonding) Adding bond0({u'hwaddr': u'7c:d3:0a:60:e9:48', u'nics': [u'eno1', u'eno2'], u'switch': u'ovs', u'options': u'mode=4'}) netlink/events::DEBUG::2022-04-21 11:26:59,055::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363790243584)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe64d610>>, args=(), kwargs={}) netlink/events::DEBUG::2022-04-21 11:26:59,057::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363781850880)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe5e9550>>, args=(), kwargs={}) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,058::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev bond0 up (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,067::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 netlink/events::DEBUG::2022-04-21 11:26:59,071::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363781850880)> netlink/events::DEBUG::2022-04-21 11:26:59,073::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363781850880)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe5e91d0>>, args=(), kwargs={}) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,074::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev eno1 up (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,081::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 netlink/events::DEBUG::2022-04-21 11:26:59,083::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363781850880)> netlink/events::DEBUG::2022-04-21 11:26:59,084::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363781850880)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe5e91d0>>, args=(), kwargs={}) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,085::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev eno2 up (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,090::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 netlink/events::DEBUG::2022-04-21 11:26:59,092::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363781850880)> netlink/events::DEBUG::2022-04-21 11:26:59,093::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363790243584)> MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,219::cmdutils::133::root::(exec_cmd) /sbin/ip addr flush dev eno1 scope global (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,225::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,340::cmdutils::133::root::(exec_cmd) /sbin/ip addr flush dev eno2 scope global (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,346::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,346::setup::92::root::(add_bonds) Creating bonds: [] MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,346::configurator::274::root::(_add_networks) Adding networks: [u'ovirtmgmt'] MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,347::ifacquire::70::root::(acquire) Acquiring ifaces: set([u'bond0']) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,348::cmdutils::133::root::(exec_cmd) /sbin/ifdown eno1 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:00,004::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:00,006::cmdutils::133::root::(exec_cmd) /sbin/ifdown eno2 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:00,657::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:00,660::cmdutils::133::root::(exec_cmd) /sbin/ifdown bond0 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,360::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,361::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- add-br vdsmbr_TtLDFxN1 -- set Bridge vdsmbr_TtLDFxN1 other-config:hwaddr="7a:b2:a6:5b:96:78" -- add-port vdsmbr_TtLDFxN1 bond0 -- set Port bond0 other_config:vdsm_level=southbound -- add-port vdsmbr_TtLDFxN1 ovirtmgmt -- set Port ovirtmgmt other_config:vdsm_level=northbound -- set Interface ovirtmgmt type=internal -- set Port ovirtmgmt tag=2306 -- set Interface ovirtmgmt mtu_request=9000 -- set Interface bond0 mtu_request=9000 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,361::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- add-br vdsmbr_TtLDFxN1 -- set Bridge vdsmbr_TtLDFxN1 'other-config:hwaddr="7a:b2:a6:5b:96:78"' -- add-port vdsmbr_TtLDFxN1 bond0 -- set Port bond0 other_config:vdsm_level=southbound -- add-port vdsmbr_TtLDFxN1 ovirtmgmt -- set Port ovirtmgmt other_config:vdsm_level=northbound -- set Interface ovirtmgmt type=internal -- set Port ovirtmgmt tag=2306 -- set Interface ovirtmgmt mtu_request=9000 -- set Interface bond0 mtu_request=9000 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,704::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::INFO::2022-04-21 11:27:02,705::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({u'ipv6autoconf': True, u'nameservers': [u'10.129.0.60', u'10.229.0.60'], u'vlan': 2306, u'ipaddr': u'10.119.6.237', u'bonding': u'bond0', u'mtu': 9000, u'switch': u'ovs', u'dhcpv6': False, u'stp': False, u'bridged': True, u'netmask': u'255.255.255.0', u'gateway': u'10.119.6.1', u'defaultRoute': True, u'bootproto': u'none'}) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,705::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,706::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,745::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,753::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- set open . external-ids:ovn-bridge-mappings="ovirtmgmt:vdsmbr_TtLDFxN1" MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,753::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings="ovirtmgmt:vdsmbr_TtLDFxN1"' (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,794::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 netlink/events::DEBUG::2022-04-21 11:27:02,796::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363790243584)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe64d590>>, args=(), kwargs={}) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,797::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev ovirtmgmt up (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,805::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 netlink/events::DEBUG::2022-04-21 11:27:02,807::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363790243584)> netlink/events::DEBUG::2022-04-21 11:27:02,808::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363790243584)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe64d590>>, args=(), kwargs={}) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,809::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev bond0 up (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,816::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 netlink/events::DEBUG::2022-04-21 11:27:02,819::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363790243584)> netlink/events::DEBUG::2022-04-21 11:27:02,820::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363790243584)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe64d590>>, args=(), kwargs={}) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,821::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev eno1 up (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,867::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 netlink/events::DEBUG::2022-04-21 11:27:02,870::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363790243584)> netlink/events::DEBUG::2022-04-21 11:27:02,871::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363790243584)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe64d590>>, args=(), kwargs={}) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,872::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev eno2 up (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,920::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 netlink/events::DEBUG::2022-04-21 11:27:02,922::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363790243584)> MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,923::cmdutils::133::root::(exec_cmd) /sbin/ip addr flush dev ovirtmgmt scope global (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,930::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,931::cmdutils::133::root::(exec_cmd) /sbin/ip -4 addr add dev ovirtmgmt 10.119.6.237/255.255.255.0 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,938::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,938::cmdutils::133::root::(exec_cmd) /sbin/ip -4 route add default via 10.119.6.1 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,945::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,947::connectivity::46::root::(check) Checking connectivity... MainProcess|jsonrpc/3::DEBUG::2022-04-21 11:27:10,538::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call ksmTune with ({u'run': 0, u'merge_across_nodes': 1},) {} MainProcess|jsonrpc/3::DEBUG::2022-04-21 11:27:10,538::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return ksmTune with None MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:04,075::connectivity::48::root::(check) Connectivity check failed, rolling back MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,075::logutils::319::root::(_report_stats) ThreadedHandler is ok in the last 136 seconds (max pending: 3) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,075::ifacquire::76::root::(_rollback) Acquiring transaction failed, reverting ifaces: [u'eno1', u'eno2', u'bond0'] MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,077::cmdutils::133::root::(exec_cmd) /sbin/ifup eno1 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,485::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,488::cmdutils::133::root::(exec_cmd) /sbin/ifup eno2 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,897::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,900::cmdutils::133::root::(exec_cmd) /sbin/ifup bond0 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,045::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = './network-functions: line 119: .: /etc/sysconfig/network-scripts/: is a directory\n'; <rc> = 0 MainProcess|jsonrpc/2::WARNING::2022-04-21 11:29:06,046::netconfpersistence::294::root::(__exit__) Failed setup transaction,reverting to last known good network. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 249, in _setup_ovs connectivity.check(options) File "/usr/lib/python2.7/site-packages/vdsm/network/connectivity.py", line 50, in check 'connectivity check failed') ConfigNetworkError: (10, 'connectivity check failed') MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:06,047::api::211::root::(setupNetworks) Setting up network according to configuration: networks:{u'ovirtmgmt': {'remove': True}}, bondings:{u'bond0': {'remove': True}}, options:{'connectivityCheck': 0, 'inRollback': True} MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,061::cmdutils::133::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,079::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,153::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,153::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,190::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:06,326::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'bonding': u'bond0', 'ipv6autoconf': True, 'nameservers': [], 'vlan': 2306, 'ipaddr': '10.119.6.237', 'switch': 'ovs', 'mtu': 9000, 'netmask': '255.255.255.0', 'dhcpv6': False, 'stp': False, 'bridged': True, 'defaultRoute': False, 'bootproto': 'none'}) MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:06,327::netconfpersistence::69::root::(setBonding) Adding bond0({'nics': ['eno1', 'eno2'], 'switch': 'legacy', 'options': 'mode=4'}) MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:06,327::netconfpersistence::63::root::(removeNetwork) Removing network ovirtmgmt MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,334::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-95 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,911::commands::219::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:06,911::hooks::114::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err= MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,918::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,918::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,956::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,091::configurator::265::root::(_remove_networks) Removing networks: [u'ovirtmgmt'] MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,219::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- set Interface bond0 mtu_request=1500 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,220::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- set Interface bond0 mtu_request=1500 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,425::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,426::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- del-port ovirtmgmt -- del-port bond0 -- del-br vdsmbr_TtLDFxN1 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,426::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- del-port ovirtmgmt -- del-port bond0 -- del-br vdsmbr_TtLDFxN1 (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,949::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,950::netconfpersistence::65::root::(removeNetwork) Network ovirtmgmt not found for removal MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,951::setup::41::root::(remove_bonds) Removing bonds: [u'bond0'] MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:08,165::sysfs_driver::69::root::(destroy) Bond bond0 has been destroyed. MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,166::netconfpersistence::76::root::(removeBonding) bond0 not found for removal MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,172::ifacquire::70::root::(acquire) Acquiring ifaces: set([]) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,172::setup::61::root::(edit_bonds) Editing bonds: [] MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,172::setup::92::root::(add_bonds) Creating bonds: [] MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,172::configurator::274::root::(_add_networks) Adding networks: [] MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,172::ifacquire::70::root::(acquire) Acquiring ifaces: set([]) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,173::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,173::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,211::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,217::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- set open . external-ids:ovn-bridge-mappings="" MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,217::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,247::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:08,249::netconfpersistence::231::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:08,253::netconfpersistence::181::root::(save) Saved new config RunningConfig({}, {}, {}) to [/var/lib/vdsm/staging/netconf/nets,/var/lib/vdsm/staging/netconf/bonds,/var/lib/vdsm/staging/netconf/devices] MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,256::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-95 /usr/libexec/vdsm/hooks/after_network_setup/30_ethtool_options (cwd None) MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,434::commands::219::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:08,434::hooks::114::root::(_runHooksDir) /usr/libexec/vdsm/hooks/after_network_setup/30_ethtool_options: rc=0 err= MainProcess|jsonrpc/2::ERROR::2022-04-21 11:29:08,436::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 101, in wrapper res = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 228, in setupNetworks _setup_networks(networks, bondings, options, net_info) File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 249, in _setup_networks networks, bondings, options, net_info, in_rollback) File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 157, in _rollback six.reraise(excType, value, tb) File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 140, in _rollback yield File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 249, in _setup_networks networks, bondings, options, net_info, in_rollback) File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 139, in setup _setup(networks, bondings, options, in_rollback, net_info) File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 159, in _setup _setup_ovs(ovs_nets, ovs_bonds, options, net_info, in_rollback) File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 249, in _setup_ovs connectivity.check(options) File "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", line 295, in __exit__ raise ne.RollbackIncomplete(config_diff, ex_type, ex_value) ConfigNetworkError: (10, 'connectivity check failed') MainProcess|jsonrpc/5::DEBUG::2022-04-21 12:01:20,062::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call network_caps with () {} MainProcess|jsonrpc/5::DEBUG::2022-04-21 12:01:20,062::logutils::319::root::(_report_stats) ThreadedHandler is ok in the last 1935 seconds (max pending: 3) MainProcess|jsonrpc/5::DEBUG::2022-04-21 12:01:20,076::cmdutils::133::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/5::DEBUG::2022-04-21 12:01:20,098::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0 Regards, Ravi