This is a multi-part message in MIME format. --------------3067C91F8A26101021D62FD5 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Daniel, the link below is for failed test 002_bootstrap.add_secondary_storage_domains - is that the correct link? If so, the error is in node01 and from what I can see, there is a race between creating the target for the secondary domain and the mount command sent from the host. The reason I think its a race is because on the last try to mount the server we are getting a different error than timeout and the error suggests that we wither did not create the target or did not change the permission to the target in the nfs server. If you follow the log in node01, it looks like the connectStorageServer command is succeeding but the mount commands times out at first MountError: (32, ';mount.nfs: Connection timed out\n') and than we get a "no such file or directly from the server. MountError: (32, ';mount.nfs: mounting 192.168.201.3:/exports/nfs/exported failed, reason given by server: No such file or directory\n') Here's the two last mount errors from the vdsm log. http://pastebin.test.redhat.com/509066 Here is the link to the vdsm log itself: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/... Thanks, Dafna On 08/17/2017 01:29 PM, Dominik Holler wrote:
From my point of view the snippet from logs does not point to the reason to fail the test.
On Thu, 17 Aug 2017 12:26:23 +0300 Daniel Belenky <dbelenky@redhat.com> wrote:
Failed test: basic_suite_master/002_bootstrap
Version: oVirt master
Link to failed job (Jenkins): ovirt-master_change-queue-tester/1817/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/>
Link to logs (Jenkins): link <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/>
Suspected patch: Gerrit 80481/10 <https://gerrit.ovirt.org/#/c/80481/10>
Error snippet from logs:
*From host0*
MainThread::DEBUG::2017-08-17 05:03:20,501::cmd::63::root::(exec_sync_bytes) FAILED: <err> = ''; <rc> = 1 MainThread::ERROR::2017-08-17 05:03:20,502::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth0 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth0000badminStatus0002rx' command timed out.\n", '', 'eth0')
This error is expected [1] and will by fixed by REL 7.4.1 [2]. This error just blocks collecting lldp information and should not influence anything else.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1472722
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1479767
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--------------3067C91F8A26101021D62FD5 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">Daniel, the link below is for failed test <span style="color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial; display: inline ! important; float: none;">002_bootstrap.add_secondary_storage_domains - is that the correct link? </span><br> <span style="color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial; display: inline ! important; float: none;"></span><br> <span style="color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial; display: inline ! important; float: none;">If so, the error is in node01 and from what I can see, there is a race between creating the target for the secondary domain and the mount command sent from the host. <br> The reason I think its a race is because on the last try to mount the server we are getting a different error than timeout and the error suggests that we wither did not create the target or did not change the permission to the target in the nfs server. <br> <br> If you follow the log in node01, it looks like the connectStorageServer command is succeeding but the mount commands times out at first <br> <br> MountError: (32, ';mount.nfs: Connection timed out\n')<br> <br> and than we get a "no such file or directly from the server. <br> <br> MountError: (32, ';mount.nfs: mounting 192.168.201.3:/exports/nfs/exported failed, reason given by server: No such file or directory\n')<br> <br> Here's the two last mount errors from the vdsm log. <br> <br> <a class="moz-txt-link-freetext" href="http://pastebin.test.redhat.com/509066">http://pastebin.test.redhat.com/509066</a><br> <br> Here is the link to the vdsm log itself: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-host1/_var_log/vdsm/vdsm.log">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-host1/_var_log/vdsm/vdsm.log</a><br> <br> Thanks, <br> Dafna</span><br> <br> On 08/17/2017 01:29 PM, Dominik Holler wrote:<br> </div> <blockquote type="cite" cite="mid:20170817122949.68cad062@t460p"> <pre wrap="">From my point of view the snippet from logs does not point to the reason to fail the test. On Thu, 17 Aug 2017 12:26:23 +0300 Daniel Belenky <a class="moz-txt-link-rfc2396E" href="mailto:dbelenky@redhat.com"><dbelenky@redhat.com></a> wrote: </pre> <blockquote type="cite"> <pre wrap="">Failed test: basic_suite_master/002_bootstrap Version: oVirt master Link to failed job (Jenkins): ovirt-master_change-queue-tester/1817/ <a class="moz-txt-link-rfc2396E" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/"><http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/></a> Link to logs (Jenkins): link <a class="moz-txt-link-rfc2396E" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/"><http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/></a> Suspected patch: Gerrit 80481/10 <a class="moz-txt-link-rfc2396E" href="https://gerrit.ovirt.org/#/c/80481/10"><https://gerrit.ovirt.org/#/c/80481/10></a> Error snippet from logs: *From host0* MainThread::DEBUG::2017-08-17 05:03:20,501::cmd::63::root::(exec_sync_bytes) FAILED: <err> = ''; <rc> = 1 MainThread::ERROR::2017-08-17 05:03:20,502::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth0 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth0000badminStatus0002rx' command timed out.\n", '', 'eth0') </pre> </blockquote> <pre wrap=""> This error is expected [1] and will by fixed by REL 7.4.1 [2]. This error just blocks collecting lldp information and should not influence anything else. [1] <a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1472722">https://bugzilla.redhat.com/show_bug.cgi?id=1472722</a> [2] <a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1479767">https://bugzilla.redhat.com/show_bug.cgi?id=1479767</a> _______________________________________________ Devel mailing list <a class="moz-txt-link-abbreviated" href="mailto:Devel@ovirt.org">Devel@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/devel">http://lists.ovirt.org/mailman/listinfo/devel</a> </pre> </blockquote> <p><br> </p> </body> </html> --------------3067C91F8A26101021D62FD5--