Re: [ovirt-devel] Unable to create ovirtmgmt network on node

While creating a cluster, on one of the nodes creation of ovirtmgmt fails and node is not in active state. Looking at supervdsm.log, below are few snippets of errors- =============================== MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,665::supervdsmServer::116::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'ovirtmgmt': {'ipv6autoconf': False, 'nic': 'em3', 'vlan': '802', 'mtu': 1500, 'switch': 'legacy', 'dhcpv6': False, 'bridged': 'false', 'defaultRoute': True, 'bootproto': 'dhcp'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,665::api::887::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'ipv6autoconf': False, 'nic': 'em3', 'vlan': '802', 'mtu': 1500, 'switch': 'legacy', 'dhcpv6': False, 'bridged': 'false', 'defaultRoute': True, 'bootproto': 'dhcp'}}, bondings:{}, options:{'connectivityCheck': 'true', 'connectivityTimeout': 120} MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,666::api::891::root::(setupNetworks) Validating configuration MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,670:: libvirtconnection::161::root::(get) trying to connect libvirt MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:57,690::netinfo::503::root::(_getNetInfo) Obtaining info for net ovirtmgmt. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 482, in _getNetInfo data.update({'ports': ports(iface), File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 187, in ports return os.listdir('/sys/class/net/' + bridge + '/brif') OSError: [Errno 2] No such file or directory: '/sys/class/net/ovirtmgmt/ brif' MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,708::api::901::setupNetworks::(setupNetworks) Applying... MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:57,708:: netconfpersistence::73::root::(setBonding) Adding bond0({'nics': [], 'options': 'mode=0'}) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,709::api::926::setupNetworks::(setupNetworks) Removing broken network 'ovirtmgmt' MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:57,717::netinfo::503::root::(_getNetInfo) Obtaining info for net ovirtmgmt. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 482, in _getNetInfo data.update({'ports': ports(iface), File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 187, in ports return os.listdir('/sys/class/net/' + bridge + '/brif') OSError: [Errno 2] No such file or directory: '/sys/class/net/ovirtmgmt/ brif' MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:57,726::api::463::root::(_delNetwork) Removing network ovirtmgmt with vlan=None, bonding=None, nics=[u'em3'],keep_bridge=False options={} MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,727::ifcfg::335::root::(_atomicNetworkBackup) Backed up ovirtmgmt MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:57,729::api::496::root::(_delNetwork) Removing network entity ovirtmgmt MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,729::models::188::root::(remove) Removing bridge Bridge(ovirtmgmt: Nic(em3)) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,729::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown ovirtmgmt (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,760::utils::689::root::(execCmd) FAILED: <err> = 'usage: ifdown <configuration>\n'; <rc> = 1 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,760::__init__::144::root::(_removeSourceRoute) Removing source route for device ovirtmgmt MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31 14:54:57,761::utils::140::root::(rmFile) File: /etc/sysconfig/network-scripts/rule-ovirtmgmt already removed MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,761::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/rule-ovirtmgmt MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31 14:54:57,761::utils::140::root::(rmFile) File: /etc/sysconfig/network-scripts/route-ovirtmgmt already removed MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,762::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/route-ovirtmgmt MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,762::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/brctl delbr ovirtmgmt (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,769::utils::689::root::(execCmd) FAILED: <err> = "bridge ovirtmgmt doesn't exist; can't delete it\n"; <rc> = 1 MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31 14:54:57,770::utils::140::root::(rmFile) File: /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt already removed MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,770::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:57,773::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown em3 (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:58,403::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:58,404::ifcfg::382::root::(_atomicBackup) Backed up /etc/sysconfig/network-scripts/ifcfg-em3 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:58,408::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/bin/systemd-run --scope --unit=5c224eed-6483-4d73-90a7-755800e81eed --slice=vdsm-dhclient /usr/sbin/ifup em3 (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,438::utils::689::root::(execCmd) SUCCESS: <err> = 'Running scope as unit 5c224eed-6483-4d73-90a7-755800e81eed.scope.\n'; <rc> = 0 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,439::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc filter del dev em3 pref 5000 (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,446::utils::689::root::(execCmd) FAILED: <err> = 'RTNETLINK answers: Operation not supported\nWe have an error talking to the kernel\n'; <rc> = 2 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,446::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc qdisc show dev em3 (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,452::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,453::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc class del dev em3 classid 0:1388 (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,459::utils::689::root::(execCmd) FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,459::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc class show dev em3 parent 0: (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,465::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,466::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc qdisc del dev em3 root (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,472::utils::689::root::(execCmd) FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,472::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc qdisc del dev em3 ingress (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,478::utils::689::root::(execCmd) FAILED: <err> = 'RTNETLINK answers: No such file or directory\n'; <rc> = 2 MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:59,479:: netconfpersistence::67::root::(removeNetwork) Removing network ovirtmgmt MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,525::api::766:: setupNetworks::(_add_missing_networks) Adding network 'ovirtmgmt' MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,525::api::281::root::(_addNetwork) validating network... MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:59,525::api::301::root::(_addNetwork) Adding network ovirtmgmt with vlan=802, bonding=None, nics=['em3'], bondingOptions=None, mtu=1500, bridged=False, defaultRoute=True,options={'switch': 'legacy', 'bootproto': 'dhcp', 'implicitBonding': True} MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:54:59,526::api::331::root::(_addNetwork) Configuring device em3.802 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,527::ifcfg::419::root::(_persistentBackup) backing up ifcfg-em3.802: # original file did not exist MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,528::ifcfg::324::root::(writeBackupFile) Persistently backed up /var/lib/vdsm/netconfback/ifcfg-em3.802 (until next 'set safe config') MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,528::ifcfg::490::root::(writeConfFile) Writing to file /etc/sysconfig/network-scripts/ifcfg-em3.802 configuration: # Generated by VDSM version 4.17.33-1.1.el7rhgs DEVICE=em3.802 VLAN=yes ONBOOT=yes BOOTPROTO=dhcp MTU=1500 DEFROUTE=yes NM_CONTROLLED=no IPV6INIT=no SWITCH=legacy HOTPLUG=no MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,549::ifcfg::490::root::(writeConfFile) Writing to file /etc/sysconfig/network-scripts/ifcfg-em3 configuration: # Generated by VDSM version 4.17.33-1.1.el7rhgs DEVICE=em3 HWADDR=24:6e:96:28:f1:84 ONBOOT=yes MTU=1500 NM_CONTROLLED=no IPV6INIT=no SWITCH=legacy MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:54:59,552::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown em3 (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:55:00,121::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:55:00,122::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/bin/systemd-run --scope --unit=73eaf14d-6b47-4f7a-990f-03cf718ce510 --slice=vdsm-dhclient /usr/sbin/ifup em3 (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:55:01,109::utils::689::root::(execCmd) SUCCESS: <err> = 'Running scope as unit 73eaf14d-6b47-4f7a-990f-03cf718ce510.scope.\n'; <rc> = 0 ifup-waiting-on-dhcp::DEBUG::2017-05-31 14:55:01,111::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/bin/systemd-run --scope --unit=f6e8cb52-bcec-4892-9b03-a5be989f5854 --slice=vdsm-dhclient /usr/sbin/ifup em3.802 (cwd None) MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:55:01,117:: netconfpersistence::62::root::(setNetwork) Adding network ovirtmgmt({'ipv6autoconf': False, 'nic': 'em3', 'vlan': '802', 'mtu': 1500, 'switch': 'legacy', 'dhcpv6': False, 'bridged': 'false', 'defaultRoute': True, 'bootproto': 'dhcp'}) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:55:01,133::api::789:: setupNetworks::(_check_connectivity) Checking connectivity... ifup-waiting-on-dhcp::DEBUG::2017-05-31 14:56:04,720::utils::689::root::(execCmd) FAILED: <err> = 'Running scope as unit f6e8cb52-bcec-4892-9b03-a5be989f5854.scope.\n'; <rc> = 1 MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:02,268::api::791:: setupNetworks::(_check_connectivity) Connectivity check failed, rolling back MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:02,270::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown em3.802 (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:02,879::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:02,880::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown em3 (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,478::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,479::ifcfg::349::root::(restoreAtomicNetworkBackup) Rolling back logical networks configuration (restoring atomic logical networks backup) MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,483::ifcfg::365::root::(restoreAtomicNetworkBackup) Restored ovirtmgmt MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,484::ifcfg::390::root::(restoreAtomicBackup) Rolling back configuration (restoring atomic backup) MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,484::ifcfg::399::root::(restoreAtomicBackup) Restored /etc/sysconfig/network-scripts/ifcfg-em3 MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31 14:57:03,484::utils::140::root::(rmFile) File: /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt already removed MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,484::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,485::ifcfg::395::root::(restoreAtomicBackup) Removing empty configuration backup /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,485::ifcfg::399::root::(restoreAtomicBackup) Restored /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,485::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/ifcfg-em3.802 MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,485::ifcfg::395::root::(restoreAtomicBackup) Removing empty configuration backup /etc/sysconfig/network-scripts/ifcfg-em3.802 MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,485::ifcfg::399::root::(restoreAtomicBackup) Restored /etc/sysconfig/network-scripts/ifcfg-em3.802 MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31 14:57:03,485::utils::140::root::(rmFile) File: /etc/sysconfig/network-scripts/route-ovirtmgmt already removed MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,485::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/route-ovirtmgmt MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,485::ifcfg::395::root::(restoreAtomicBackup) Removing empty configuration backup /etc/sysconfig/network-scripts/route-ovirtmgmt MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,486::ifcfg::399::root::(restoreAtomicBackup) Restored /etc/sysconfig/network-scripts/route-ovirtmgmt MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31 14:57:03,486::utils::140::root::(rmFile) File: /etc/sysconfig/network-scripts/rule-ovirtmgmt already removed MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,486::ifcfg::292::root::(_removeFile) Removed file /etc/sysconfig/network-scripts/rule-ovirtmgmt MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,486::ifcfg::395::root::(restoreAtomicBackup) Removing empty configuration backup /etc/sysconfig/network-scripts/rule-ovirtmgmt MainProcess|jsonrpc.Executor/6::INFO::2017-05-31 14:57:03,486::ifcfg::399::root::(restoreAtomicBackup) Restored /etc/sysconfig/network-scripts/rule-ovirtmgmt MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:03,487::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/bin/systemd-run --scope --unit=3954be4a-5adf-4cbf-ad6c-f6403d36657a --slice=vdsm-dhclient /usr/sbin/ifup em3 (cwd None) MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31 14:57:08,531::utils::689::root::(execCmd) SUCCESS: <err> = 'Running scope as unit 3954be4a-5adf-4cbf-ad6c-f6403d36657a.scope.\n'; <rc> = 0 MainProcess|jsonrpc.Executor/6::ERROR::2017-05-31 14:57:08,532::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper res = func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer", line 243, in setupNetworks return setupNetworks(networks, bondings, **options) File "/usr/share/vdsm/network/api.py", line 943, in setupNetworks options, logger) File "/usr/share/vdsm/network/api.py", line 800, in _check_connectivity 'connectivity check failed') ConfigNetworkError: (10, 'connectivity check failed') ==================== Can somebody help me figuring out the issue here? Regards, Shubhendu _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel