4:46 a.m.
While creating a cluster, on one of the nodes creation of ovirtmgmt fails and node is not
in active state.
Looking at supervdsm.log, below are few snippets of errors-
===============================
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,665::supervdsmServer::116::SuperVdsm.ServerCallback::(wrapper) call setupNetworks
with ({'ovirtmgmt': {'ipv6autoconf': False, 'nic': 'em3',
'vlan': '802', 'mtu': 1500, 'switch': 'legacy',
'dhcpv6': False, 'bridged': 'false', 'defaultRoute': True,
'bootproto': 'dhcp'}}, {}, {'connectivityCheck': 'true',
'connectivityTimeout': 120}) {}
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,665::api::887::setupNetworks::(setupNetworks) Setting up network according to
configuration: networks:{'ovirtmgmt': {'ipv6autoconf': False,
'nic': 'em3', 'vlan': '802', 'mtu': 1500,
'switch': 'legacy', 'dhcpv6': False, 'bridged':
'false', 'defaultRoute': True, 'bootproto': 'dhcp'}},
bondings:{}, options:{'connectivityCheck': 'true',
'connectivityTimeout': 120}
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,666::api::891::root::(setupNetworks) Validating configuration
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,670::libvirtconnection::161::root::(get) trying to connect libvirt
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:54:57,690::netinfo::503::root::(_getNetInfo) Obtaining info for net ovirtmgmt.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 482, in
_getNetInfo
data.update({'ports': ports(iface),
File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 187, in ports
return os.listdir('/sys/class/net/' + bridge + '/brif')
OSError: [Errno 2] No such file or directory: '/sys/class/net/ovirtmgmt/brif'
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,708::api::901::setupNetworks::(setupNetworks) Applying...
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:54:57,708::netconfpersistence::73::root::(setBonding) Adding bond0({'nics': [],
'options': 'mode=0'})
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,709::api::926::setupNetworks::(setupNetworks) Removing broken network
'ovirtmgmt'
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:54:57,717::netinfo::503::root::(_getNetInfo) Obtaining info for net ovirtmgmt.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 482, in
_getNetInfo
data.update({'ports': ports(iface),
File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 187, in ports
return os.listdir('/sys/class/net/' + bridge + '/brif')
OSError: [Errno 2] No such file or directory: '/sys/class/net/ovirtmgmt/brif'
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:54:57,726::api::463::root::(_delNetwork) Removing network ovirtmgmt with vlan=None,
bonding=None, nics=[u'em3'],keep_bridge=False options={}
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,727::ifcfg::335::root::(_atomicNetworkBackup) Backed up ovirtmgmt
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:54:57,729::api::496::root::(_delNetwork) Removing network entity ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,729::models::188::root::(remove) Removing bridge Bridge(ovirtmgmt: Nic(em3))
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,729::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown
ovirtmgmt (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,760::utils::689::root::(execCmd) FAILED: <err> = 'usage: ifdown
<configuration>\n'; <rc> = 1
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,760::__init__::144::root::(_removeSourceRoute) Removing source route for device
ovirtmgmt
MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31
14:54:57,761::utils::140::root::(rmFile) File:
/etc/sysconfig/network-scripts/rule-ovirtmgmt already removed
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,761::ifcfg::292::root::(_removeFile) Removed file
/etc/sysconfig/network-scripts/rule-ovirtmgmt
MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31
14:54:57,761::utils::140::root::(rmFile) File:
/etc/sysconfig/network-scripts/route-ovirtmgmt already removed
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,762::ifcfg::292::root::(_removeFile) Removed file
/etc/sysconfig/network-scripts/route-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,762::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/brctl
delbr ovirtmgmt (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,769::utils::689::root::(execCmd) FAILED: <err> = "bridge ovirtmgmt
doesn't exist; can't delete it\n"; <rc> = 1
MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31
14:54:57,770::utils::140::root::(rmFile) File:
/etc/sysconfig/network-scripts/ifcfg-ovirtmgmt already removed
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,770::ifcfg::292::root::(_removeFile) Removed file
/etc/sysconfig/network-scripts/ifcfg-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:57,773::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown
em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:58,403::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> =
0
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:58,404::ifcfg::382::root::(_atomicBackup) Backed up
/etc/sysconfig/network-scripts/ifcfg-em3
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:58,408::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5
/usr/bin/systemd-run --scope --unit=5c224eed-6483-4d73-90a7-755800e81eed
--slice=vdsm-dhclient /usr/sbin/ifup em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,438::utils::689::root::(execCmd) SUCCESS: <err> = 'Running scope as
unit 5c224eed-6483-4d73-90a7-755800e81eed.scope.\n'; <rc> = 0
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,439::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc
filter del dev em3 pref 5000 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,446::utils::689::root::(execCmd) FAILED: <err> = 'RTNETLINK answers:
Operation not supported\nWe have an error talking to the kernel\n'; <rc> = 2
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,446::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc
qdisc show dev em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,452::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> =
0
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,453::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc
class del dev em3 classid 0:1388 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,459::utils::689::root::(execCmd) FAILED: <err> = 'RTNETLINK answers: No
such file or directory\n'; <rc> = 2
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,459::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc
class show dev em3 parent 0: (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,465::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> =
0
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,466::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc
qdisc del dev em3 root (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,472::utils::689::root::(execCmd) FAILED: <err> = 'RTNETLINK answers: No
such file or directory\n'; <rc> = 2
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,472::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/tc
qdisc del dev em3 ingress (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,478::utils::689::root::(execCmd) FAILED: <err> = 'RTNETLINK answers: No
such file or directory\n'; <rc> = 2
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:54:59,479::netconfpersistence::67::root::(removeNetwork) Removing network ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,525::api::766::setupNetworks::(_add_missing_networks) Adding network
'ovirtmgmt'
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,525::api::281::root::(_addNetwork) validating network...
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:54:59,525::api::301::root::(_addNetwork) Adding network ovirtmgmt with vlan=802,
bonding=None, nics=['em3'], bondingOptions=None, mtu=1500, bridged=False,
defaultRoute=True,options={'switch': 'legacy', 'bootproto':
'dhcp', 'implicitBonding': True}
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:54:59,526::api::331::root::(_addNetwork) Configuring device em3.802
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,527::ifcfg::419::root::(_persistentBackup) backing up ifcfg-em3.802: # original
file did not exist
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,528::ifcfg::324::root::(writeBackupFile) Persistently backed up
/var/lib/vdsm/netconfback/ifcfg-em3.802 (until next 'set safe config')
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,528::ifcfg::490::root::(writeConfFile) Writing to file
/etc/sysconfig/network-scripts/ifcfg-em3.802 configuration:
# Generated by VDSM version 4.17.33-1.1.el7rhgs
DEVICE=em3.802
VLAN=yes
ONBOOT=yes
BOOTPROTO=dhcp
MTU=1500
DEFROUTE=yes
NM_CONTROLLED=no
IPV6INIT=no
SWITCH=legacy
HOTPLUG=no
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,549::ifcfg::490::root::(writeConfFile) Writing to file
/etc/sysconfig/network-scripts/ifcfg-em3 configuration:
# Generated by VDSM version 4.17.33-1.1.el7rhgs
DEVICE=em3
HWADDR=24:6e:96:28:f1:84
ONBOOT=yes
MTU=1500
NM_CONTROLLED=no
IPV6INIT=no
SWITCH=legacy
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:54:59,552::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown
em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:55:00,121::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> =
0
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:55:00,122::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5
/usr/bin/systemd-run --scope --unit=73eaf14d-6b47-4f7a-990f-03cf718ce510
--slice=vdsm-dhclient /usr/sbin/ifup em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:55:01,109::utils::689::root::(execCmd) SUCCESS: <err> = 'Running scope as
unit 73eaf14d-6b47-4f7a-990f-03cf718ce510.scope.\n'; <rc> = 0
ifup-waiting-on-dhcp::DEBUG::2017-05-31 14:55:01,111::utils::671::root::(execCmd)
/usr/bin/taskset --cpu-list 0-5 /usr/bin/systemd-run --scope
--unit=f6e8cb52-bcec-4892-9b03-a5be989f5854 --slice=vdsm-dhclient /usr/sbin/ifup em3.802
(cwd None)
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:55:01,117::netconfpersistence::62::root::(setNetwork) Adding network
ovirtmgmt({'ipv6autoconf': False, 'nic': 'em3', 'vlan':
'802', 'mtu': 1500, 'switch': 'legacy', 'dhcpv6':
False, 'bridged': 'false', 'defaultRoute': True,
'bootproto': 'dhcp'})
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:55:01,133::api::789::setupNetworks::(_check_connectivity) Checking connectivity...
ifup-waiting-on-dhcp::DEBUG::2017-05-31 14:56:04,720::utils::689::root::(execCmd) FAILED:
<err> = 'Running scope as unit
f6e8cb52-bcec-4892-9b03-a5be989f5854.scope.\n'; <rc> = 1
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:57:02,268::api::791::setupNetworks::(_check_connectivity) Connectivity check failed,
rolling back
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:02,270::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown
em3.802 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:02,879::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> =
0
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:02,880::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5 /usr/sbin/ifdown
em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:03,478::utils::689::root::(execCmd) SUCCESS: <err> = ''; <rc> =
0
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:57:03,479::ifcfg::349::root::(restoreAtomicNetworkBackup) Rolling back logical networks
configuration (restoring atomic logical networks backup)
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:57:03,483::ifcfg::365::root::(restoreAtomicNetworkBackup) Restored ovirtmgmt
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:57:03,484::ifcfg::390::root::(restoreAtomicBackup) Rolling back configuration
(restoring atomic backup)
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:57:03,484::ifcfg::399::root::(restoreAtomicBackup) Restored
/etc/sysconfig/network-scripts/ifcfg-em3
MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31
14:57:03,484::utils::140::root::(rmFile) File:
/etc/sysconfig/network-scripts/ifcfg-ovirtmgmt already removed
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:03,484::ifcfg::292::root::(_removeFile) Removed file
/etc/sysconfig/network-scripts/ifcfg-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:03,485::ifcfg::395::root::(restoreAtomicBackup) Removing empty configuration backup
/etc/sysconfig/network-scripts/ifcfg-ovirtmgmt
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:57:03,485::ifcfg::399::root::(restoreAtomicBackup) Restored
/etc/sysconfig/network-scripts/ifcfg-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:03,485::ifcfg::292::root::(_removeFile) Removed file
/etc/sysconfig/network-scripts/ifcfg-em3.802
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:03,485::ifcfg::395::root::(restoreAtomicBackup) Removing empty configuration backup
/etc/sysconfig/network-scripts/ifcfg-em3.802
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:57:03,485::ifcfg::399::root::(restoreAtomicBackup) Restored
/etc/sysconfig/network-scripts/ifcfg-em3.802
MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31
14:57:03,485::utils::140::root::(rmFile) File:
/etc/sysconfig/network-scripts/route-ovirtmgmt already removed
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:03,485::ifcfg::292::root::(_removeFile) Removed file
/etc/sysconfig/network-scripts/route-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:03,485::ifcfg::395::root::(restoreAtomicBackup) Removing empty configuration backup
/etc/sysconfig/network-scripts/route-ovirtmgmt
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:57:03,486::ifcfg::399::root::(restoreAtomicBackup) Restored
/etc/sysconfig/network-scripts/route-ovirtmgmt
MainProcess|jsonrpc.Executor/6::WARNING::2017-05-31
14:57:03,486::utils::140::root::(rmFile) File:
/etc/sysconfig/network-scripts/rule-ovirtmgmt already removed
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:03,486::ifcfg::292::root::(_removeFile) Removed file
/etc/sysconfig/network-scripts/rule-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:03,486::ifcfg::395::root::(restoreAtomicBackup) Removing empty configuration backup
/etc/sysconfig/network-scripts/rule-ovirtmgmt
MainProcess|jsonrpc.Executor/6::INFO::2017-05-31
14:57:03,486::ifcfg::399::root::(restoreAtomicBackup) Restored
/etc/sysconfig/network-scripts/rule-ovirtmgmt
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:03,487::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-5
/usr/bin/systemd-run --scope --unit=3954be4a-5adf-4cbf-ad6c-f6403d36657a
--slice=vdsm-dhclient /usr/sbin/ifup em3 (cwd None)
MainProcess|jsonrpc.Executor/6::DEBUG::2017-05-31
14:57:08,531::utils::689::root::(execCmd) SUCCESS: <err> = 'Running scope as
unit 3954be4a-5adf-4cbf-ad6c-f6403d36657a.scope.\n'; <rc> = 0
MainProcess|jsonrpc.Executor/6::ERROR::2017-05-31
14:57:08,532::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in
setupNetworks
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 243, in setupNetworks
return setupNetworks(networks, bondings, **options)
File "/usr/share/vdsm/network/api.py", line 943, in setupNetworks
options, logger)
File "/usr/share/vdsm/network/api.py", line 800, in _check_connectivity
'connectivity check failed')
ConfigNetworkError: (10, 'connectivity check failed')
====================
Can somebody help me figuring out the issue here?
Regards,
Shubhendu