<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 8, 2016 at 3:19 PM, Dan Kenigsberg <span dir="ltr"><<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Thu, Sep 08, 2016 at 02:23:24PM +0300, Nir Soffer wrote:<br>
> On Thu, Sep 8, 2016 at 2:20 PM, Nir Soffer <<a href="mailto:nsoffer@redhat.com">nsoffer@redhat.com</a>> wrote:<br>
> > Seen this (new?) failure on master - please check:<br>
<br>
</span>Please rebase on top of the recent fix<br>
of <a href="https://gerrit.ovirt.org/63516" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/63516</a><br>
<div class="HOEnZb"><div class="h5"><br>
> ><br>
> > <a href="http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/216/console" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_master_check-patch-el7-<wbr>x86_64/216/console</a><br>
> ><br>
> > 0:59:51 ==============================<wbr>==============================<wbr>==========<br>
> > 10:59:51 FAIL: test_ip_info (network.netinfo_test.<wbr>TestNetinfo)<br>
> > 10:59:51 ------------------------------<wbr>------------------------------<wbr>----------<br>
> > 10:59:51 Traceback (most recent call last):<br>
> > 10:59:51 File<br>
> > "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-el7-x86_64/<wbr>vdsm/tests/testValidation.py",<br>
> > line 97, in wrapper<br>
> > 10:59:51 return f(*args, **kwargs)<br>
> > 10:59:51 File<br>
> > "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-el7-x86_64/<wbr>vdsm/tests/network/netinfo_<wbr>test.py",<br>
> > line 359, in test_ip_info<br>
> > 10:59:51 [IPV6_ADDR_CIDR]))<br>
> > 10:59:51 AssertionError: Tuples differ: ('192.0.2.2',<br>
> > '255.255.255.0',... != ('192.0.2.2', '255.255.255.0',...<br>
> > 10:59:51<br>
> > 10:59:51 First differing element 2:<br>
> > 10:59:51 ['<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>', '<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>', '<a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a>']<br>
> > 10:59:51 ['<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>', '<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>', '<a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a>', '<a href="http://192.0.2.3/24" rel="noreferrer" target="_blank">192.0.2.3/24</a>']<br>
> > 10:59:51<br>
> > 10:59:51 ('192.0.2.2',<br>
> > 10:59:51 '255.255.255.0',<br>
> > 10:59:51 - ['<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>', '<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>', '<a href="http://198.51.100.11/32'" rel="noreferrer" target="_blank">198.51.100.11/32'</a>],<br>
> > 10:59:51 + ['<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>', '<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>', '<a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a>',<br>
> > '<a href="http://192.0.2.3/24'" rel="noreferrer" target="_blank">192.0.2.3/24'</a>],<br>
> > 10:59:51 ?<br>
> > ++++++++++++++++<br>
> > 10:59:51<br>
> > 10:59:51 ['2607:f0d0:1002:51::4/64'])<br>
> > 10:59:51 -------------------- >> begin captured logging << --------------------<br>
> > 10:59:51 2016-09-08 10:59:40,924 DEBUG [root] (MainThread)<br>
> > /usr/bin/taskset --cpu-list 0-1 /sbin/ip link add name dummy_GX6CE<br>
> > type dummy (cwd None)<br>
> > 10:59:51 2016-09-08 10:59:40,944 DEBUG [root] (MainThread) SUCCESS:<br>
> > <err> = ''; <rc> = 0<br>
> > 10:59:51 2016-09-08 10:59:40,950 DEBUG [root] (MainThread)<br>
> > /usr/bin/taskset --cpu-list 0-1 /sbin/ip link set dev dummy_GX6CE up<br>
> > (cwd None)<br>
> > 10:59:51 2016-09-08 10:59:40,968 DEBUG [root] (MainThread) SUCCESS:<br>
> > <err> = ''; <rc> = 0<br>
> > 10:59:51 2016-09-08 10:59:40,971 DEBUG [root] (MainThread)<br>
> > /usr/bin/taskset --cpu-list 0-1 /sbin/ip -4 addr add dev dummy_GX6CE<br>
> > <a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a> (cwd None)<br>
> > 10:59:51 2016-09-08 10:59:40,988 DEBUG [root] (MainThread) SUCCESS:<br>
> > <err> = ''; <rc> = 0<br>
> > 10:59:51 2016-09-08 10:59:40,991 DEBUG [root] (MainThread)<br>
> > /usr/bin/taskset --cpu-list 0-1 /sbin/ip -4 addr add dev dummy_GX6CE<br>
> > <a href="http://192.0.2.3/24" rel="noreferrer" target="_blank">192.0.2.3/24</a> (cwd None)<br>
> > 10:59:51 2016-09-08 10:59:41,007 DEBUG [root] (MainThread) SUCCESS:<br>
> > <err> = ''; <rc> = 0<br>
> > 10:59:51 2016-09-08 10:59:41,010 DEBUG [root] (MainThread)<br>
> > /usr/bin/taskset --cpu-list 0-1 /sbin/ip -4 addr add dev dummy_GX6CE<br>
> > <a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a> (cwd None)<br>
> > 10:59:51 2016-09-08 10:59:41,037 DEBUG [root] (MainThread) SUCCESS:<br>
> > <err> = ''; <rc> = 0<br>
> > 10:59:51 2016-09-08 10:59:41,040 DEBUG [root] (MainThread)<br>
> > /usr/bin/taskset --cpu-list 0-1 /sbin/ip -6 addr add dev dummy_GX6CE<br>
> > 2607:f0d0:1002:51::4/64 (cwd None)<br>
> > 10:59:51 2016-09-08 10:59:41,067 DEBUG [root] (MainThread) SUCCESS:<br>
> > <err> = ''; <rc> = 0<br>
> > 10:59:51 2016-09-08 10:59:41,071 DEBUG [root] (MainThread)<br>
> > /usr/bin/taskset --cpu-list 0-1 /sbin/ip -4 addr add dev dummy_GX6CE<br>
> > <a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a> (cwd None)<br>
> > 10:59:51 2016-09-08 10:59:41,090 DEBUG [root] (MainThread) SUCCESS:<br>
> > <err> = ''; <rc> = 0<br>
> > 10:59:51 2016-09-08 10:59:41,102 DEBUG [root] (MainThread)<br>
> > /usr/bin/taskset --cpu-list 0-1 /sbin/ip link del dev dummy_GX6CE (cwd<br>
> > None)<br>
> > 10:59:51 2016-09-08 10:59:41,133 DEBUG [root] (MainThread) SUCCESS:<br>
> > <err> = ''; <rc> = 0<br>
> > 10:59:51 --------------------- >> end captured logging << ---------------------<br>
><br>
> Another instance here:<br>
> <a href="http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/212/console" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_master_check-patch-el7-<wbr>x86_64/212/console</a><br>
><br>
> 0:58:52 ==============================<wbr>==============================<wbr>==========<br>
> 10:58:52 FAIL: test_ip_info (network.netinfo_test.<wbr>TestNetinfo)<br>
> 10:58:52 ------------------------------<wbr>------------------------------<wbr>----------<br>
> 10:58:52 Traceback (most recent call last):<br>
> 10:58:52 File<br>
> "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-el7-x86_64/<wbr>vdsm/tests/testValidation.py",<br>
> line 97, in wrapper<br>
> 10:58:52 return f(*args, **kwargs)<br>
> 10:58:52 File<br>
> "/home/jenkins/workspace/vdsm_<wbr>master_check-patch-el7-x86_64/<wbr>vdsm/tests/network/netinfo_<wbr>test.py",<br>
> line 359, in test_ip_info<br>
> 10:58:52 [IPV6_ADDR_CIDR]))<br>
> 10:58:52 AssertionError: Tuples differ: ('192.0.2.2',<br>
> '255.255.255.0',... != ('192.0.2.2', '255.255.255.0',...<br>
> 10:58:52<br>
> 10:58:52 First differing element 2:<br>
> 10:58:52 ['<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>', '<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>', '<a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a>']<br>
> 10:58:52 ['<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>', '<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>', '<a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a>', '<a href="http://192.0.2.3/24" rel="noreferrer" target="_blank">192.0.2.3/24</a>']<br>
> 10:58:52<br>
> 10:58:52 ('192.0.2.2',<br>
> 10:58:52 '255.255.255.0',<br>
> 10:58:52 - ['<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>', '<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>', '<a href="http://198.51.100.11/32'" rel="noreferrer" target="_blank">198.51.100.11/32'</a>],<br>
> 10:58:52 + ['<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>', '<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>', '<a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a>',<br>
> '<a href="http://192.0.2.3/24'" rel="noreferrer" target="_blank">192.0.2.3/24'</a>],<br>
> 10:58:52 ?<br>
> ++++++++++++++++<br>
> 10:58:52<br>
> 10:58:52 ['2607:f0d0:1002:51::4/64'])<br>
> 10:58:52 -------------------- >> begin captured logging << --------------------<br>
> 10:58:52 2016-09-08 10:58:42,213 DEBUG [root] (MainThread)<br>
> /usr/bin/taskset --cpu-list 0-31 /sbin/ip link add name dummy_8awuY<br>
> type dummy (cwd None)<br>
> 10:58:52 2016-09-08 10:58:42,221 DEBUG [root] (MainThread) SUCCESS:<br>
> <err> = ''; <rc> = 0<br>
> 10:58:52 2016-09-08 10:58:42,224 DEBUG [root] (MainThread)<br>
> /usr/bin/taskset --cpu-list 0-31 /sbin/ip link set dev dummy_8awuY up<br>
> (cwd None)<br>
> 10:58:52 2016-09-08 10:58:42,237 DEBUG [root] (MainThread) SUCCESS:<br>
> <err> = ''; <rc> = 0<br>
> 10:58:52 2016-09-08 10:58:42,240 DEBUG [root] (MainThread)<br>
> /usr/bin/taskset --cpu-list 0-31 /sbin/ip -4 addr add dev dummy_8awuY<br>
> <a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a> (cwd None)<br>
> 10:58:52 2016-09-08 10:58:42,250 DEBUG [root] (MainThread) SUCCESS:<br>
> <err> = ''; <rc> = 0<br>
> 10:58:52 2016-09-08 10:58:42,252 DEBUG [root] (MainThread)<br>
> /usr/bin/taskset --cpu-list 0-31 /sbin/ip -4 addr add dev dummy_8awuY<br>
> <a href="http://192.0.2.3/24" rel="noreferrer" target="_blank">192.0.2.3/24</a> (cwd None)<br>
> 10:58:52 2016-09-08 10:58:42,259 DEBUG [root] (MainThread) SUCCESS:<br>
> <err> = ''; <rc> = 0<br>
> 10:58:52 2016-09-08 10:58:42,260 DEBUG [root] (MainThread)<br>
> /usr/bin/taskset --cpu-list 0-31 /sbin/ip -4 addr add dev dummy_8awuY<br>
> <a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a> (cwd None)<br>
> 10:58:52 2016-09-08 10:58:42,267 DEBUG [root] (MainThread) SUCCESS:<br>
> <err> = ''; <rc> = 0<br>
> 10:58:52 2016-09-08 10:58:42,269 DEBUG [root] (MainThread)<br>
> /usr/bin/taskset --cpu-list 0-31 /sbin/ip -6 addr add dev dummy_8awuY<br>
> 2607:f0d0:1002:51::4/64 (cwd None)<br>
> 10:58:52 2016-09-08 10:58:42,276 DEBUG [root] (MainThread) SUCCESS:<br>
> <err> = ''; <rc> = 0<br>
> 10:58:52 2016-09-08 10:58:42,278 DEBUG [root] (MainThread)<br>
> /usr/bin/taskset --cpu-list 0-31 /sbin/ip -4 addr add dev dummy_8awuY<br>
> <a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a> (cwd None)<br>
> 10:58:52 2016-09-08 10:58:42,289 DEBUG [root] (MainThread) SUCCESS:<br>
> <err> = ''; <rc> = 0<br>
> 10:58:52 2016-09-08 10:58:42,293 DEBUG [root] (MainThread)<br>
> /usr/bin/taskset --cpu-list 0-31 /sbin/ip link del dev dummy_8awuY<br>
> (cwd None)<br>
> 10:58:52 2016-09-08 10:58:42,322 DEBUG [root] (MainThread) SUCCESS:<br>
> <err> = ''; <rc> = 0<br>
> 10:58:52 --------------------- >> end captured logging << ---------------------<br>
</div></div></blockquote></div><br></div><div class="gmail_extra">This looks like a new one... I suspect that the kernel is sending multiple identical events.<br></div><div class="gmail_extra">We can try and fix it by waiting for a specific IP.<br><br></div></div>