[ ovirt-devel ] [ OST Failure Report ] [ oVirt Master ] [ 002_bootstrap ] [ 17/08/17 ]
Failed test: basic_suite_master/002_bootstrap Version: oVirt master Link to failed job (Jenkins): ovirt-master_change-queue-tester/1817/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/> Link to logs (Jenkins): link <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/> Suspected patch: Gerrit 80481/10 <https://gerrit.ovirt.org/#/c/80481/10> Error snippet from logs: *From host0* MainThread::DEBUG::2017-08-17 05:03:20,501::cmd::63::root::(exec_sync_bytes) FAILED: <err> = ''; <rc> = 1 MainThread::ERROR::2017-08-17 05:03:20,502::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth0 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth0000badminStatus0002rx' command timed out.\n", '', 'eth0') Thanks, -- DANIEL BELENKY RHV DEVOPS EMEA VIRTUALIZATION R&D <https://red.ht/sig>
From my point of view the snippet from logs does not point to the reason to fail the test. On Thu, 17 Aug 2017 12:26:23 +0300 Daniel Belenky <dbelenky@redhat.com> wrote:
Failed test: basic_suite_master/002_bootstrap
Version: oVirt master
Link to failed job (Jenkins): ovirt-master_change-queue-tester/1817/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/>
Link to logs (Jenkins): link <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/>
Suspected patch: Gerrit 80481/10 <https://gerrit.ovirt.org/#/c/80481/10>
Error snippet from logs:
*From host0*
MainThread::DEBUG::2017-08-17 05:03:20,501::cmd::63::root::(exec_sync_bytes) FAILED: <err> = ''; <rc> = 1 MainThread::ERROR::2017-08-17 05:03:20,502::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth0 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth0000badminStatus0002rx' command timed out.\n", '', 'eth0')
This error is expected [1] and will by fixed by REL 7.4.1 [2]. This error just blocks collecting lldp information and should not influence anything else. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1472722 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1479767
This is a multi-part message in MIME format. --------------3067C91F8A26101021D62FD5 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Daniel, the link below is for failed test 002_bootstrap.add_secondary_storage_domains - is that the correct link? If so, the error is in node01 and from what I can see, there is a race between creating the target for the secondary domain and the mount command sent from the host. The reason I think its a race is because on the last try to mount the server we are getting a different error than timeout and the error suggests that we wither did not create the target or did not change the permission to the target in the nfs server. If you follow the log in node01, it looks like the connectStorageServer command is succeeding but the mount commands times out at first MountError: (32, ';mount.nfs: Connection timed out\n') and than we get a "no such file or directly from the server. MountError: (32, ';mount.nfs: mounting 192.168.201.3:/exports/nfs/exported failed, reason given by server: No such file or directory\n') Here's the two last mount errors from the vdsm log. http://pastebin.test.redhat.com/509066 Here is the link to the vdsm log itself: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/... Thanks, Dafna On 08/17/2017 01:29 PM, Dominik Holler wrote:
From my point of view the snippet from logs does not point to the reason to fail the test.
On Thu, 17 Aug 2017 12:26:23 +0300 Daniel Belenky <dbelenky@redhat.com> wrote:
Failed test: basic_suite_master/002_bootstrap
Version: oVirt master
Link to failed job (Jenkins): ovirt-master_change-queue-tester/1817/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/>
Link to logs (Jenkins): link <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/>
Suspected patch: Gerrit 80481/10 <https://gerrit.ovirt.org/#/c/80481/10>
Error snippet from logs:
*From host0*
MainThread::DEBUG::2017-08-17 05:03:20,501::cmd::63::root::(exec_sync_bytes) FAILED: <err> = ''; <rc> = 1 MainThread::ERROR::2017-08-17 05:03:20,502::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth0 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth0000badminStatus0002rx' command timed out.\n", '', 'eth0')
This error is expected [1] and will by fixed by REL 7.4.1 [2]. This error just blocks collecting lldp information and should not influence anything else.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1472722
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1479767
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--------------3067C91F8A26101021D62FD5 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">Daniel, the link below is for failed test <span style="color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial; display: inline ! important; float: none;">002_bootstrap.add_secondary_storage_domains - is that the correct link? </span><br> <span style="color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial; display: inline ! important; float: none;"></span><br> <span style="color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial; display: inline ! important; float: none;">If so, the error is in node01 and from what I can see, there is a race between creating the target for the secondary domain and the mount command sent from the host. <br> The reason I think its a race is because on the last try to mount the server we are getting a different error than timeout and the error suggests that we wither did not create the target or did not change the permission to the target in the nfs server. <br> <br> If you follow the log in node01, it looks like the connectStorageServer command is succeeding but the mount commands times out at first <br> <br> MountError: (32, ';mount.nfs: Connection timed out\n')<br> <br> and than we get a "no such file or directly from the server. <br> <br> MountError: (32, ';mount.nfs: mounting 192.168.201.3:/exports/nfs/exported failed, reason given by server: No such file or directory\n')<br> <br> Here's the two last mount errors from the vdsm log. <br> <br> <a class="moz-txt-link-freetext" href="http://pastebin.test.redhat.com/509066">http://pastebin.test.redhat.com/509066</a><br> <br> Here is the link to the vdsm log itself: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-host1/_var_log/vdsm/vdsm.log">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-host1/_var_log/vdsm/vdsm.log</a><br> <br> Thanks, <br> Dafna</span><br> <br> On 08/17/2017 01:29 PM, Dominik Holler wrote:<br> </div> <blockquote type="cite" cite="mid:20170817122949.68cad062@t460p"> <pre wrap="">From my point of view the snippet from logs does not point to the reason to fail the test. On Thu, 17 Aug 2017 12:26:23 +0300 Daniel Belenky <a class="moz-txt-link-rfc2396E" href="mailto:dbelenky@redhat.com"><dbelenky@redhat.com></a> wrote: </pre> <blockquote type="cite"> <pre wrap="">Failed test: basic_suite_master/002_bootstrap Version: oVirt master Link to failed job (Jenkins): ovirt-master_change-queue-tester/1817/ <a class="moz-txt-link-rfc2396E" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/"><http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/></a> Link to logs (Jenkins): link <a class="moz-txt-link-rfc2396E" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/"><http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/></a> Suspected patch: Gerrit 80481/10 <a class="moz-txt-link-rfc2396E" href="https://gerrit.ovirt.org/#/c/80481/10"><https://gerrit.ovirt.org/#/c/80481/10></a> Error snippet from logs: *From host0* MainThread::DEBUG::2017-08-17 05:03:20,501::cmd::63::root::(exec_sync_bytes) FAILED: <err> = ''; <rc> = 1 MainThread::ERROR::2017-08-17 05:03:20,502::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth0 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth0000badminStatus0002rx' command timed out.\n", '', 'eth0') </pre> </blockquote> <pre wrap=""> This error is expected [1] and will by fixed by REL 7.4.1 [2]. This error just blocks collecting lldp information and should not influence anything else. [1] <a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1472722">https://bugzilla.redhat.com/show_bug.cgi?id=1472722</a> [2] <a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1479767">https://bugzilla.redhat.com/show_bug.cgi?id=1479767</a> _______________________________________________ Devel mailing list <a class="moz-txt-link-abbreviated" href="mailto:Devel@ovirt.org">Devel@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/devel">http://lists.ovirt.org/mailman/listinfo/devel</a> </pre> </blockquote> <p><br> </p> </body> </html> --------------3067C91F8A26101021D62FD5--
On Thu, Aug 17, 2017 at 12:26 PM, Daniel Belenky <dbelenky@redhat.com> wrote:
Failed test: basic_suite_master/002_bootstrap
Version: oVirt master
Link to failed job (Jenkins): ovirt-master_change-queue-tester/1817/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/>
Link to logs (Jenkins): link <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1817/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/>
Suspected patch: Gerrit 80481/10 <https://gerrit.ovirt.org/#/c/80481/10>
Error snippet from logs:
*From host0*
MainThread::DEBUG::2017-08-17 05:03:20,501::cmd::63::root::(exec_sync_bytes) FAILED: <err> = ''; <rc> = 1 MainThread::ERROR::2017-08-17 05:03:20,502::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth0 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth0000badminStatus0002rx' command timed out.\n", '', 'eth0')
where is this coming from? I'm seeing issues in host1 (vdsm log) : 2017-08-17 05:11:32,476-0400 ERROR (jsonrpc/6) [storage.HSM] Could not connect to storageServer (hsm:2351) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2348, in connectStorageServer conObj.connect() File "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 381, in connect return self._mountCon.connect() File "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 163, in connect six.reraise(t, v, tb) File "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 155, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 208, in mount timeout=timeout, cgroup=cgroup) File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 53, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 51, in <lambda> **kwargs) File "<string>", line 2, in mount File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) MountError: (32, ';mount.nfs: mounting 192.168.201.3:/exports/nfs/exported failed, reason given by server: No such file or directory\n') Y.
Thanks, --
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
participants (4)
-
Dafna Ron -
Daniel Belenky -
Dominik Holler -
Yaniv Kaul