[ovirt-devel] Enable to create ovirtmgmt network on node

Wed Jul 19 09:46:07 UTC 2017

While creating a cluster, on one of the nodes creation of ovirtmgmt fails and node is not in active state.
Looking at supervdsm.log, below are few snippets of errors-

===============================
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,665::supervdsmServer::116::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'ovirtmgmt': {'ipv6autoconf': False, 'nic': 'em3', 'vlan': '802', 'mtu': 1500, 'switch': 'legacy', 'dhcpv6': False, 'bridged': 'false', 'defaultRoute': True, 'bootproto': 'dhcp'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {}
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,665::api::887::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'ipv6autoconf': False, 'nic': 'em3', 'vlan': '802', 'mtu': 1500, 'switch': 'legacy', 'dhcpv6': False, 'bridged': 'false', 'defaultRoute': True, 'bootproto': 'dhcp'}}, bondings:{}, options:{'connectivityCheck': 'true', 'connectivityTimeout': 120}
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,666::api::891::root::(setupNetworks) Validating configuration
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,670::libvirtconnection::161::root::(get) trying to connect libvirt
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:57,690::netinfo::503::root::(_getNetInfo) Obtaining info for net ovirtmgmt.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 482, in _getNetInfo
    data.update({'ports': ports(iface),
  File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 187, in ports
    return os.listdir('/sys/class/net/' + bridge + '/brif')
OSError: [Errno 2] No such file or directory: '/sys/class/net/ovirtmgmt/brif'
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,708::api::901::setupNetworks::(setupNetworks) Applying...
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:57,708::netconfpersistence::73::root::(setBonding) Adding bond0({'nics': [], 'options': 'mode=0'})
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,709::api::926::setupNetworks::(setupNetworks) Removing broken network 'ovirtmgmt'
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:57,717::netinfo::503::root::(_getNetInfo) Obtaining info for net ovirtmgmt.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 482, in _getNetInfo
    data.update({'ports': ports(iface),
  File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 187, in ports
    return os.listdir('/sys/class/net/' + bridge + '/brif')
OSError: [Errno 2] No such file or directory: '/sys/class/net/ovirtmgmt/brif'
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:57,726::api::463::root::(_delNetwork) Removing network ovirtmgmt with vlan=None, bonding=None, nics=[u'em3'],keep_bridge=False options={}
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,727::ifcfg::335::root::(_atomicNetworkBackup) Backed up ovirtmgmt
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:57,729::api::496::root::(_delNetwork) Removing network entity ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,729::models::188::root::(remove) Removing bridge Bridge(ovirtmgmt: Nic(em3))
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,729::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown ovirtmgmt (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,760::utils::689::root::(execCmd) FAILED: <err> = 'usage: ifdown <configuration>\n'; <rc> = 1
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,760::__init__::144::root::(_removeSourceRoute) Removing source route for device ovirtmgmt
MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31 14:54:57,761::utils::140::root::(rmFile) File: /etc/sysconfig/network-scripts/rule-ovirtmgmt already removed
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,761::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/rule-ovirtmgmt
MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31 14:54:57,761::utils::140::root::(rmFile) File: /etc/sysconfig/network-scripts/route-ovirtmgmt already removed
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,762::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/route-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,762::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/brctl delbr ovirtmgmt (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,769::utils::689::root::(execCmd) FAILED: <err> = "bridge ovirtmgmt doesn't exist; can't delete it\n"; <rc> = 1
MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31 14:54:57,770::utils::140::root::(rmFile) File: /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt already removed
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,770::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,773::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:58,403::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:58,404::ifcfg::382::root::(_atomicBackup) Backed up /etc/sysconfig/network-scripts/ifcfg-em3
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:58,408::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/bin/systemd-run --scope --unit=5c224eed-6483-4d73-90a7-755800e81eed --slice=vdsm-dhclient /usr/sbin/ifup em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,438::utils::689::root::(execCmd) SUCCESS: <err> = 'Running scope as unit 5c224eed-6483-4d73-90a7-755800e81eed.scope.\n'; <rc> = 0
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,439::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc filter del dev em3 pref 5000 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,446::utils::689::root::(execCmd) FAILED: <err> = 'RTNETLINK answers: Operation not supported\nWe have an error talking to the kernel\n'; <rc> = 2
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,446::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc qdisc show dev em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,452::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,453::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc class del dev em3 classid 0:1388 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,459::utils::689::root::(execCmd) FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,459::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc class show dev em3 parent 0: (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,465::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,466::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc qdisc del dev em3 root (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,472::utils::689::root::(execCmd) FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,472::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc qdisc del dev em3 ingress (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,478::utils::689::root::(execCmd) FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:59,479::netconfpersistence::67::root::(removeNetwork) Removing network ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,525::api::766::setupNetworks::(_add_missing_networks) Adding network 'ovirtmgmt'
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,525::api::281::root::(_addNetwork) validating network...
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:59,525::api::301::root::(_addNetwork) Adding network ovirtmgmt with vlan=802, bonding=None, nics=['em3'], bondingOptions=None, mtu=1500, bridged=False, defaultRoute=True,options={'switch': 'legacy', 'bootproto': 'dhcp', 'implicitBonding': True}
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:59,526::api::331::root::(_addNetwork) Configuring device em3.802
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,527::ifcfg::419::root::(_persistentBackup) backing up ifcfg-em3.802: # original file did not exist

MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,528::ifcfg::324::root::(writeBackupFile) Persistently backed up /var/lib/vdsm/netconfback/ifcfg-em3.802 (until next 'set safe config')
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,528::ifcfg::490::root::(writeConfFile) Writing to file /etc/sysconfig/network-scripts/ifcfg-em3.802 configuration:
# Generated by VDSM version 4.17.33-1.1.el7rhgs
DEVICE=em3.802
VLAN=yes
ONBOOT=yes
BOOTPROTO=dhcp
MTU=1500
DEFROUTE=yes
NM_CONTROLLED=no
IPV6INIT=no
SWITCH=legacy
HOTPLUG=no

MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,549::ifcfg::490::root::(writeConfFile) Writing to file /etc/sysconfig/network-scripts/ifcfg-em3 configuration:
# Generated by VDSM version 4.17.33-1.1.el7rhgs
DEVICE=em3
HWADDR=24:6e:96:28:f1:84
ONBOOT=yes
MTU=1500
NM_CONTROLLED=no
IPV6INIT=no
SWITCH=legacy

MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,552::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:55:00,121::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:55:00,122::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/bin/systemd-run --scope --unit=73eaf14d-6b47-4f7a-990f-03cf718ce510 --slice=vdsm-dhclient /usr/sbin/ifup em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:55:01,109::utils::689::root::(execCmd) SUCCESS: <err> = 'Running scope as unit 73eaf14d-6b47-4f7a-990f-03cf718ce510.scope.\n'; <rc> = 0
ifup-waiting-on-dhcp::DEBUG::2017-05-31 14:55:01,111::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/bin/systemd-run --scope --unit=f6e8cb52-bcec-4892-9b03-a5be989f5854 --slice=vdsm-dhclient /usr/sbin/ifup em3.802 (cwd None)
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:55:01,117::netconfpersistence::62::root::(setNetwork) Adding network ovirtmgmt({'ipv6autoconf': False, 'nic': 'em3', 'vlan': '802', 'mtu': 1500, 'switch': 'legacy', 'dhcpv6': False, 'bridged': 'false', 'defaultRoute': True, 'bootproto': 'dhcp'})
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:55:01,133::api::789::setupNetworks::(_check_connectivity) Checking connectivity...
ifup-waiting-on-dhcp::DEBUG::2017-05-31 14:56:04,720::utils::689::root::(execCmd) FAILED: <err> = 'Running scope as unit f6e8cb52-bcec-4892-9b03-a5be989f5854.scope.\n'; <rc> = 1
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:02,268::api::791::setupNetworks::(_check_connectivity) Connectivity check failed, rolling back
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:02,270::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown em3.802 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:02,879::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:02,880::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,478::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,479::ifcfg::349::root::(restoreAtomicNetworkBackup) Rolling back logical networks configuration (restoring atomic logical networks backup)
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,483::ifcfg::365::root::(restoreAtomicNetworkBackup) Restored ovirtmgmt
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,484::ifcfg::390::root::(restoreAtomicBackup) Rolling back configuration (restoring atomic backup)
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,484::ifcfg::399::root::(restoreAtomicBackup) Restored /etc/sysconfig/network-scripts/ifcfg-em3
MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31 14:57:03,484::utils::140::root::(rmFile) File: /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt already removed
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,484::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,485::ifcfg::395::root::(restoreAtomicBackup) Removing empty configuration backup /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,485::ifcfg::399::root::(restoreAtomicBackup) Restored /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,485::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/ifcfg-em3.802
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,485::ifcfg::395::root::(restoreAtomicBackup) Removing empty configuration backup /etc/sysconfig/network-scripts/ifcfg-em3.802
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,485::ifcfg::399::root::(restoreAtomicBackup) Restored /etc/sysconfig/network-scripts/ifcfg-em3.802
MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31 14:57:03,485::utils::140::root::(rmFile) File: /etc/sysconfig/network-scripts/route-ovirtmgmt already removed
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,485::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/route-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,485::ifcfg::395::root::(restoreAtomicBackup) Removing empty configuration backup /etc/sysconfig/network-scripts/route-ovirtmgmt
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,486::ifcfg::399::root::(restoreAtomicBackup) Restored /etc/sysconfig/network-scripts/route-ovirtmgmt
MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31 14:57:03,486::utils::140::root::(rmFile) File: /etc/sysconfig/network-scripts/rule-ovirtmgmt already removed
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,486::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/rule-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,486::ifcfg::395::root::(restoreAtomicBackup) Removing empty configuration backup /etc/sysconfig/network-scripts/rule-ovirtmgmt
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,486::ifcfg::399::root::(restoreAtomicBackup) Restored /etc/sysconfig/network-scripts/rule-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,487::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/bin/systemd-run --scope --unit=3954be4a-5adf-4cbf-ad6c-f6403d36657a --slice=vdsm-dhclient /usr/sbin/ifup em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:08,531::utils::689::root::(execCmd) SUCCESS: <err> = 'Running scope as unit 3954be4a-5adf-4cbf-ad6c-f6403d36657a.scope.\n'; <rc> = 0
MainProcess|jsonrpc.Executor/6::ERROR::2017-05-31 14:57:08,532::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
    res = func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer", line 243, in setupNetworks
    return setupNetworks(networks, bondings, **options)
  File "/usr/share/vdsm/network/api.py", line 943, in setupNetworks
    options, logger)
  File "/usr/share/vdsm/network/api.py", line 800, in _check_connectivity
    'connectivity check failed')
ConfigNetworkError: (10, 'connectivity check failed')

====================

Can somebody help me figuring out the issue here?

Regards,
Shubhendu