For the record, we have realized that this failure is unrelated to the log supplied in bug https://bugzilla.redhat.com/show_bug.cgi?id=1417595 . That log is due to a test requesting dhcp address, not receiving it, and carrying on to next tests, while a Vdsm thread is still waiting for dhcp. The cause for this breakage is probably a broken jsonrpc connection, hopefully fixed by https://gerrit.ovirt.org/#/q/Idaa54767bb7e54bf13e89887ca34fa8e01ade420 On Thu, Mar 9, 2017 at 3:06 PM, Dan Kenigsberg <danken@redhat.com> wrote:

On Thu, Mar 9, 2017 at 10:00 AM, Daniel Belenky <dbelenky@redhat.com> wrote:

...

Test failed: basic_sanity.hotplug_nic

Link to failed job: test-repo_ovirt_experimental_4.1/917/

Link to all logs: logs from Jenkins

May be related to: https://bugzilla.redhat.com/show_bug.cgi?id=1417595

Error snippet from log: (from supervdsm.log):

ifup/VLAN100_Network::DEBUG::2017-03-08 22:24:22,832::commands::93::root::(execCmd) FAILED: <err> = 'Running scope as unit ff994cde-b66d-4d54-9e36-fc253682608f.scope.\n/etc/sysconfig/network-scripts/ifup-eth: line 297: 16985 Terminated /sbin/dhclient ${DHCLIENTARGS} ${DEVICE}\nCannot find device "VLAN100_Network"\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\n'; <rc> = 1 ifup/VLAN100_Network::ERROR::2017-03-08 22:24:22,832::utils::371::root::(wrapper) Unhandled exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 368, in wrapper return f(*a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 180, in run return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 924, in _exec_ifup _exec_ifup_by_name(iface.name, cgroup) File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 910, in _exec_ifup_by_name raise ConfigNetworkError(ERR_FAILED_IFUP, out[-1] if out else '') ConfigNetworkError: (29, 'Determining IPv6 information for VLAN100_Network... failed.')

Yes, this seems to be a reproduction of Vdsm bug 1417595 - hotplug_nic test fails in OST

Eddy, can you tell how could it be that the VLAN100_NETWORK bridge does not exist when ifup is called?