<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 8, 2016 at 3:19 PM, Dan Kenigsberg <span dir="ltr">&lt;<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Thu, Sep 08, 2016 at 02:23:24PM +0300, Nir Soffer wrote:<br>
&gt; On Thu, Sep 8, 2016 at 2:20 PM, Nir Soffer &lt;<a href="mailto:nsoffer@redhat.com">nsoffer@redhat.com</a>&gt; wrote:<br>
&gt; &gt; Seen this (new?) failure on master - please check:<br>
<br>
</span>Please rebase on top of the recent fix<br>
of <a href="https://gerrit.ovirt.org/63516" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/63516</a><br>
<div class="HOEnZb"><div class="h5"><br>
&gt; &gt;<br>
&gt; &gt; <a href="http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/216/console" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_master_check-patch-el7-<wbr>x86_64/216/console</a><br>
&gt; &gt;<br>
&gt; &gt; 0:59:51 ==============================<wbr>==============================<wbr>==========<br>
&gt; &gt; 10:59:51 FAIL: test_ip_info (network.netinfo_test.<wbr>TestNetinfo)<br>
&gt; &gt; 10:59:51 ------------------------------<wbr>------------------------------<wbr>----------<br>
&gt; &gt; 10:59:51 Traceback (most recent call last):<br>
&gt; &gt; 10:59:51   File<br>
&gt; &gt; &quot;/home/jenkins/workspace/vdsm_<wbr>master_check-patch-el7-x86_64/<wbr>vdsm/tests/testValidation.py&quot;,<br>
&gt; &gt; line 97, in wrapper<br>
&gt; &gt; 10:59:51     return f(*args, **kwargs)<br>
&gt; &gt; 10:59:51   File<br>
&gt; &gt; &quot;/home/jenkins/workspace/vdsm_<wbr>master_check-patch-el7-x86_64/<wbr>vdsm/tests/network/netinfo_<wbr>test.py&quot;,<br>
&gt; &gt; line 359, in test_ip_info<br>
&gt; &gt; 10:59:51     [IPV6_ADDR_CIDR]))<br>
&gt; &gt; 10:59:51 AssertionError: Tuples differ: (&#39;192.0.2.2&#39;,<br>
&gt; &gt; &#39;255.255.255.0&#39;,... != (&#39;192.0.2.2&#39;, &#39;255.255.255.0&#39;,...<br>
&gt; &gt; 10:59:51<br>
&gt; &gt; 10:59:51 First differing element 2:<br>
&gt; &gt; 10:59:51 [&#39;<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>&#39;, &#39;<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>&#39;, &#39;<a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a>&#39;]<br>
&gt; &gt; 10:59:51 [&#39;<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>&#39;, &#39;<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>&#39;, &#39;<a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a>&#39;, &#39;<a href="http://192.0.2.3/24" rel="noreferrer" target="_blank">192.0.2.3/24</a>&#39;]<br>
&gt; &gt; 10:59:51<br>
&gt; &gt; 10:59:51   (&#39;192.0.2.2&#39;,<br>
&gt; &gt; 10:59:51    &#39;255.255.255.0&#39;,<br>
&gt; &gt; 10:59:51 -  [&#39;<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>&#39;, &#39;<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>&#39;, &#39;<a href="http://198.51.100.11/32&#39;" rel="noreferrer" target="_blank">198.51.100.11/32&#39;</a>],<br>
&gt; &gt; 10:59:51 +  [&#39;<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>&#39;, &#39;<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>&#39;, &#39;<a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a>&#39;,<br>
&gt; &gt; &#39;<a href="http://192.0.2.3/24&#39;" rel="noreferrer" target="_blank">192.0.2.3/24&#39;</a>],<br>
&gt; &gt; 10:59:51 ?<br>
&gt; &gt; ++++++++++++++++<br>
&gt; &gt; 10:59:51<br>
&gt; &gt; 10:59:51    [&#39;2607:f0d0:1002:51::4/64&#39;])<br>
&gt; &gt; 10:59:51 -------------------- &gt;&gt; begin captured logging &lt;&lt; --------------------<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:40,924 DEBUG   [root] (MainThread)<br>
&gt; &gt; /usr/bin/taskset --cpu-list 0-1 /sbin/ip link add name dummy_GX6CE<br>
&gt; &gt; type dummy (cwd None)<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:40,944 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:40,950 DEBUG   [root] (MainThread)<br>
&gt; &gt; /usr/bin/taskset --cpu-list 0-1 /sbin/ip link set dev dummy_GX6CE up<br>
&gt; &gt; (cwd None)<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:40,968 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:40,971 DEBUG   [root] (MainThread)<br>
&gt; &gt; /usr/bin/taskset --cpu-list 0-1 /sbin/ip -4 addr add dev dummy_GX6CE<br>
&gt; &gt; <a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a> (cwd None)<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:40,988 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:40,991 DEBUG   [root] (MainThread)<br>
&gt; &gt; /usr/bin/taskset --cpu-list 0-1 /sbin/ip -4 addr add dev dummy_GX6CE<br>
&gt; &gt; <a href="http://192.0.2.3/24" rel="noreferrer" target="_blank">192.0.2.3/24</a> (cwd None)<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:41,007 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:41,010 DEBUG   [root] (MainThread)<br>
&gt; &gt; /usr/bin/taskset --cpu-list 0-1 /sbin/ip -4 addr add dev dummy_GX6CE<br>
&gt; &gt; <a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a> (cwd None)<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:41,037 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:41,040 DEBUG   [root] (MainThread)<br>
&gt; &gt; /usr/bin/taskset --cpu-list 0-1 /sbin/ip -6 addr add dev dummy_GX6CE<br>
&gt; &gt; 2607:f0d0:1002:51::4/64 (cwd None)<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:41,067 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:41,071 DEBUG   [root] (MainThread)<br>
&gt; &gt; /usr/bin/taskset --cpu-list 0-1 /sbin/ip -4 addr add dev dummy_GX6CE<br>
&gt; &gt; <a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a> (cwd None)<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:41,090 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:41,102 DEBUG   [root] (MainThread)<br>
&gt; &gt; /usr/bin/taskset --cpu-list 0-1 /sbin/ip link del dev dummy_GX6CE (cwd<br>
&gt; &gt; None)<br>
&gt; &gt; 10:59:51 2016-09-08 10:59:41,133 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; &gt; 10:59:51 --------------------- &gt;&gt; end captured logging &lt;&lt; ---------------------<br>
&gt;<br>
&gt; Another instance here:<br>
&gt; <a href="http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/212/console" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_master_check-patch-el7-<wbr>x86_64/212/console</a><br>
&gt;<br>
&gt; 0:58:52 ==============================<wbr>==============================<wbr>==========<br>
&gt; 10:58:52 FAIL: test_ip_info (network.netinfo_test.<wbr>TestNetinfo)<br>
&gt; 10:58:52 ------------------------------<wbr>------------------------------<wbr>----------<br>
&gt; 10:58:52 Traceback (most recent call last):<br>
&gt; 10:58:52   File<br>
&gt; &quot;/home/jenkins/workspace/vdsm_<wbr>master_check-patch-el7-x86_64/<wbr>vdsm/tests/testValidation.py&quot;,<br>
&gt; line 97, in wrapper<br>
&gt; 10:58:52     return f(*args, **kwargs)<br>
&gt; 10:58:52   File<br>
&gt; &quot;/home/jenkins/workspace/vdsm_<wbr>master_check-patch-el7-x86_64/<wbr>vdsm/tests/network/netinfo_<wbr>test.py&quot;,<br>
&gt; line 359, in test_ip_info<br>
&gt; 10:58:52     [IPV6_ADDR_CIDR]))<br>
&gt; 10:58:52 AssertionError: Tuples differ: (&#39;192.0.2.2&#39;,<br>
&gt; &#39;255.255.255.0&#39;,... != (&#39;192.0.2.2&#39;, &#39;255.255.255.0&#39;,...<br>
&gt; 10:58:52<br>
&gt; 10:58:52 First differing element 2:<br>
&gt; 10:58:52 [&#39;<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>&#39;, &#39;<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>&#39;, &#39;<a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a>&#39;]<br>
&gt; 10:58:52 [&#39;<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>&#39;, &#39;<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>&#39;, &#39;<a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a>&#39;, &#39;<a href="http://192.0.2.3/24" rel="noreferrer" target="_blank">192.0.2.3/24</a>&#39;]<br>
&gt; 10:58:52<br>
&gt; 10:58:52   (&#39;192.0.2.2&#39;,<br>
&gt; 10:58:52    &#39;255.255.255.0&#39;,<br>
&gt; 10:58:52 -  [&#39;<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>&#39;, &#39;<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>&#39;, &#39;<a href="http://198.51.100.11/32&#39;" rel="noreferrer" target="_blank">198.51.100.11/32&#39;</a>],<br>
&gt; 10:58:52 +  [&#39;<a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a>&#39;, &#39;<a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a>&#39;, &#39;<a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a>&#39;,<br>
&gt; &#39;<a href="http://192.0.2.3/24&#39;" rel="noreferrer" target="_blank">192.0.2.3/24&#39;</a>],<br>
&gt; 10:58:52 ?<br>
&gt; ++++++++++++++++<br>
&gt; 10:58:52<br>
&gt; 10:58:52    [&#39;2607:f0d0:1002:51::4/64&#39;])<br>
&gt; 10:58:52 -------------------- &gt;&gt; begin captured logging &lt;&lt; --------------------<br>
&gt; 10:58:52 2016-09-08 10:58:42,213 DEBUG   [root] (MainThread)<br>
&gt; /usr/bin/taskset --cpu-list 0-31 /sbin/ip link add name dummy_8awuY<br>
&gt; type dummy (cwd None)<br>
&gt; 10:58:52 2016-09-08 10:58:42,221 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; 10:58:52 2016-09-08 10:58:42,224 DEBUG   [root] (MainThread)<br>
&gt; /usr/bin/taskset --cpu-list 0-31 /sbin/ip link set dev dummy_8awuY up<br>
&gt; (cwd None)<br>
&gt; 10:58:52 2016-09-08 10:58:42,237 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; 10:58:52 2016-09-08 10:58:42,240 DEBUG   [root] (MainThread)<br>
&gt; /usr/bin/taskset --cpu-list 0-31 /sbin/ip -4 addr add dev dummy_8awuY<br>
&gt; <a href="http://192.0.2.2/24" rel="noreferrer" target="_blank">192.0.2.2/24</a> (cwd None)<br>
&gt; 10:58:52 2016-09-08 10:58:42,250 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; 10:58:52 2016-09-08 10:58:42,252 DEBUG   [root] (MainThread)<br>
&gt; /usr/bin/taskset --cpu-list 0-31 /sbin/ip -4 addr add dev dummy_8awuY<br>
&gt; <a href="http://192.0.2.3/24" rel="noreferrer" target="_blank">192.0.2.3/24</a> (cwd None)<br>
&gt; 10:58:52 2016-09-08 10:58:42,259 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; 10:58:52 2016-09-08 10:58:42,260 DEBUG   [root] (MainThread)<br>
&gt; /usr/bin/taskset --cpu-list 0-31 /sbin/ip -4 addr add dev dummy_8awuY<br>
&gt; <a href="http://198.51.100.9/24" rel="noreferrer" target="_blank">198.51.100.9/24</a> (cwd None)<br>
&gt; 10:58:52 2016-09-08 10:58:42,267 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; 10:58:52 2016-09-08 10:58:42,269 DEBUG   [root] (MainThread)<br>
&gt; /usr/bin/taskset --cpu-list 0-31 /sbin/ip -6 addr add dev dummy_8awuY<br>
&gt; 2607:f0d0:1002:51::4/64 (cwd None)<br>
&gt; 10:58:52 2016-09-08 10:58:42,276 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; 10:58:52 2016-09-08 10:58:42,278 DEBUG   [root] (MainThread)<br>
&gt; /usr/bin/taskset --cpu-list 0-31 /sbin/ip -4 addr add dev dummy_8awuY<br>
&gt; <a href="http://198.51.100.11/32" rel="noreferrer" target="_blank">198.51.100.11/32</a> (cwd None)<br>
&gt; 10:58:52 2016-09-08 10:58:42,289 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; 10:58:52 2016-09-08 10:58:42,293 DEBUG   [root] (MainThread)<br>
&gt; /usr/bin/taskset --cpu-list 0-31 /sbin/ip link del dev dummy_8awuY<br>
&gt; (cwd None)<br>
&gt; 10:58:52 2016-09-08 10:58:42,322 DEBUG   [root] (MainThread) SUCCESS:<br>
&gt; &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>
&gt; 10:58:52 --------------------- &gt;&gt; end captured logging &lt;&lt; ---------------------<br>
</div></div></blockquote></div><br></div><div class="gmail_extra">This looks like a new one... I suspect that the kernel is sending multiple identical events.<br></div><div class="gmail_extra">We can try and fix it by waiting for a specific IP.<br><br></div></div>