This is a multi-part message in MIME format... ------------=_1508515087-31944-155 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit [ https://ovirt-jira.atlassian.net/browse/OVIRT-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=35180#comment-35180 ] Evgheni Dereveanchin edited comment on OVIRT-1712 at 10/20/17 3:58 PM: ----------------------------------------------------------------------- I rebooted ovirt-srv21 which was failing manual tests and [~dfodor] started a new build on it. It finished successfully and nothing was listening to port 8585 when I logged in to check after the job finished. I went through all of the bare metals and a few of them had port 8585 still occupied. They also had leftovers of ovirt-master_change-queue-tester in the Jenkins work directory, so this may be the job causing the issue. was (Author: ederevea): I rebooted ovirt-srv21 which was failing manual tests and [~dfodor] started a new build on it. It finished successfully and nothing was listening to port 8585 when I logged in to check after the job finished.. It may be some other job leaving this web server behind but it's definitely affected several bare metals so far. I'll try to log in to the idle ones and kill the python process if present but the root cause is yet to be identified.

Re: Manual OST fails --------------------

Key: OVIRT-1712 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1712 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: eyal edri Assignee: infra

Evgheni, Was there any change recently to Lago slaves? On Fri, Oct 20, 2017 at 11:05 AM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:

...

I attempted to run manual OST twice and both failed with below issue. Can someone take a look?

Thanks, Piotr

2017-10-20 07:59:12,485::log_utils.py::__exit__::607::ovirtlago.prefix: :DEBUG:: File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 111, in wrapper with utils.repo_server_context(args[0]): File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 100, in repo_server_context root_dir=prefix.paths.internal_repo(), File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76, in _create_http_server generate_request_handler(root_dir), File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ self.server_bind() File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind SocketServer.TCPServer.server_bind(self) File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind self.socket.bind(self.server_address) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args)

2017-10-20 07:59:12,485::cmd.py::do_run::365::root::ERROR::Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 166, in do_deploy prefix.deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 111, in wrapper with utils.repo_server_context(args[0]): File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 100, in repo_server_context root_dir=prefix.paths.internal_repo(), File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76, in _create_http_server generate_request_handler(root_dir), File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ self.server_bind() File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind SocketServer.TCPServer.server_bind(self) File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind self.socket.bind(self.server_address) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) error: [Errno 98] Address already in use _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

-- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

-- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100066) ------------=_1508515087-31944-155 Content-Type: text/html; charset="UTF-8" Content-Disposition: inline Content-Transfer-Encoding: 7bit <html><body> <pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1712?page=com.atlassian.jira.p... ]</pre> <h3>Evgheni Dereveanchin edited comment on OVIRT-1712 at 10/20/17 3:58 PM:</h3> <p>I rebooted ovirt-srv21 which was failing manual tests and [~dfodor] started a new build on it. It finished successfully and nothing was listening to port 8585 when I logged in to check after the job finished. I went through all of the bare metals and a few of them had port 8585 still occupied. They also had leftovers of ovirt-master_change-queue-tester in the Jenkins work directory, so this may be the job causing the issue.</p> <p>was (Author: ederevea): I rebooted ovirt-srv21 which was failing manual tests and [~dfodor] started a new build on it. It finished successfully and nothing was listening to port 8585 when I logged in to check after the job finished.. It may be some other job leaving this web server behind but it's definitely affected several bare metals so far. I'll try to log in to the idle ones and kill the python process if present but the root cause is yet to be identified.</p> <blockquote><h3>Re: Manual OST fails</h3> <pre> Key: OVIRT-1712 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1712 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: eyal edri Assignee: infra</pre> <p>Evgheni, Was there any change recently to Lago slaves? On Fri, Oct 20, 2017 at 11:05 AM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:</p> <blockquote><p>I attempted to run manual OST twice and both failed with below issue. Can someone take a look?</p> <p>Thanks, Piotr</p> <p>2017-10-20 07:59:12,485::log_utils.py::__exit__::607::ovirtlago.prefix: :DEBUG::</p> <pre>File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,</pre> <p>in wrapper</p> <pre> return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line</pre> <p>111, in wrapper</p> <pre> with utils.repo_server_context(args[0]): File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line</pre> <p>100, in repo_server_context</p> <pre> root_dir=prefix.paths.internal_repo(), File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,</pre> <p>in _create_http_server</p> <pre> generate_request_handler(root_dir), File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ self.server_bind() File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind SocketServer.TCPServer.server_bind(self) File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind self.socket.bind(self.server_address) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args)</pre> <p>2017-10-20 07:59:12,485::cmd.py::do_run::365::root::ERROR::Error occured, aborting Traceback (most recent call last):</p> <pre>File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, in</pre> <p>do_run</p> <pre> self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line</pre> <p>184, in do_run</p> <pre> self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in</pre> <p>wrapper</p> <pre> return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in</pre> <p>wrapper</p> <pre> return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 166,</pre> <p>in do_deploy</p> <pre> prefix.deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,</pre> <p>in wrapper</p> <pre> return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line</pre> <p>111, in wrapper</p> <pre> with utils.repo_server_context(args[0]): File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line</pre> <p>100, in repo_server_context</p> <pre> root_dir=prefix.paths.internal_repo(), File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,</pre> <p>in _create_http_server</p> <pre> generate_request_handler(root_dir), File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ self.server_bind() File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind SocketServer.TCPServer.server_bind(self) File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind self.socket.bind(self.server_address) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args)</pre> <p>error: [Errno 98] Address already in use <em>____________________________________________</em>_ Infra mailing list Infra@ovirt.org <a href="http://lists.ovirt.org/mailman/listinfo/infra">http://lists.ovirt.org/mailman/listinfo/infra</a></p></blockquote> <p>— Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <<a href="https://www.redhat.com/">https://www.redhat.com/</a>>; <<a href="https://red.ht/sig">https://red.ht/sig</a>>; TRIED. TESTED. TRUSTED. <<a href="https://redhat.com/trusted">https://redhat.com/trusted</a>>; phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)</p></blockquote> <p>— This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100066)</p> <img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYLyzY..." alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/> </body></html> ------------=_1508515087-31944-155--