[JIRA] (OVIRT-1658) Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
by eyal edri (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1658?page=com.atlassian.jir... ]
eyal edri commented on OVIRT-1658:
----------------------------------
Is it still an issue after a few 7.4.z batches were released?
> Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
> ----------------------------------------------------------------------
>
> Key: OVIRT-1658
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Barak Korren
> Assignee: Evgheni Dereveanchin
>
> Seems there is an issue with locale files in recent CentOS versions,
> and we're hitting it in slaves that have been updated recently.
> The symptoms are that Jenkins disconnects from the slave and then
> refuses to reconnect to it. The agent log in Jenkins shows:
> [09/19/17 15:42:56] [SSH] Connection closed.
> [09/19/17 15:58:48] [SSH] Opening SSH connection to
> vm0002.workers-phx.ovirt.org:22.
> [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being
> verified. Man-in-the-middle attacks may be possible against this
> connection.
> [09/19/17 15:58:48] [SSH] Authentication successful.
> SSH connection reports a garbage before a command execution.
> Check your .bashrc, .profile, and so on to make sure it is quiet.
> The received junk text is as follows:
> /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot
> change locale (en_US.utf8): No such file or directory
> /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE:
> cannot change locale (en_US.utf8): No such file or directory
> /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES:
> cannot change locale (en_US.utf8): No such file or directory
> /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC:
> cannot change locale (en_US.utf8): No such file or directory
> /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot
> change locale (en_US.utf8): No such file or directory
> null
> [09/19/17 15:58:48] Launch failed - cleaning up connection
> [09/19/17 15:58:48] [SSH] Connection closed.
> The same locale error messages can also be reproduced on the slave by
> running an interactive login from the console or 'su -'. When running
> 'locale -a' you can also see the en_US.UTF-8 locale is somehow
> missing.
> Looking around for this I found the following:
> https://github.com/CentOS/sig-cloud-instance-images/issues/71
> I tried downgrading glibc back to the version we had before, but that
> did not seem to resolve the issue. Eventually I managed to resolve it
> by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave.
> I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is
> attached to the staging Jenkins, but I've no reason to believe this
> won't start impacting production slaves.
> We need to research this further and find out if we need to do
> something to prevent this issue from surfacing on production slaves.
> --
> Barak Korren
> RHV DevOps team , RHCE, RHCi
> Red Hat EMEA
> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100075)
6 years, 3 months
[JIRA] (OVIRT-1712) OST jobs fails on "address already in use"
by eyal edri (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1712?page=com.atlassian.jir... ]
eyal edri updated OVIRT-1712:
-----------------------------
Status: In Progress (was: To Do)
> OST jobs fails on "address already in use"
> ------------------------------------------
>
> Key: OVIRT-1712
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1712
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: eyal edri
> Assignee: Gal Ben Haim
> Priority: High
>
> Evgheni,
> Was there any change recently to Lago slaves?
> On Fri, Oct 20, 2017 at 11:05 AM, Piotr Kliczewski <
> piotr.kliczewski(a)gmail.com> wrote:
> > I attempted to run manual OST twice and both failed with below issue.
> > Can someone take a look?
> >
> > Thanks,
> > Piotr
> >
> > 2017-10-20 07:59:12,485::log_utils.py::__exit__::607::ovirtlago.prefix:
> > :DEBUG::
> > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,
> > in wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line
> > 111, in wrapper
> > with utils.repo_server_context(args[0]):
> > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> > return self.gen.next()
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line
> > 100, in repo_server_context
> > root_dir=prefix.paths.internal_repo(),
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,
> > in _create_http_server
> > generate_request_handler(root_dir),
> > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> > self.server_bind()
> > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> > SocketServer.TCPServer.server_bind(self)
> > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> > self.socket.bind(self.server_address)
> > File "/usr/lib64/python2.7/socket.py", line 224, in meth
> > return getattr(self._sock,name)(*args)
> >
> > 2017-10-20 07:59:12,485::cmd.py::do_run::365::root::ERROR::Error
> > occured, aborting
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, in
> > do_run
> > self.cli_plugins[args.ovirtverb].do_run(args)
> > File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line
> > 184, in do_run
> > self._do_run(**vars(args))
> > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in
> > wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in
> > wrapper
> > return func(*args, prefix=prefix, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 166,
> > in do_deploy
> > prefix.deploy()
> > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,
> > in wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line
> > 111, in wrapper
> > with utils.repo_server_context(args[0]):
> > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> > return self.gen.next()
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line
> > 100, in repo_server_context
> > root_dir=prefix.paths.internal_repo(),
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,
> > in _create_http_server
> > generate_request_handler(root_dir),
> > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> > self.server_bind()
> > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> > SocketServer.TCPServer.server_bind(self)
> > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> > self.socket.bind(self.server_address)
> > File "/usr/lib64/python2.7/socket.py", line 224, in meth
> > return getattr(self._sock,name)(*args)
> > error: [Errno 98] Address already in use
> > _______________________________________________
> > Infra mailing list
> > Infra(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
> >
> --
> Eyal edri
> MANAGER
> RHV DevOps
> EMEA VIRTUALIZATION R&D
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100075)
6 years, 3 months
[JIRA] (OVIRT-1712) OST jobs fails on "address already in use"
by eyal edri (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1712?page=com.atlassian.jir... ]
eyal edri updated OVIRT-1712:
-----------------------------
Priority: High (was: Medium)
> OST jobs fails on "address already in use"
> ------------------------------------------
>
> Key: OVIRT-1712
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1712
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: eyal edri
> Assignee: Gal Ben Haim
> Priority: High
>
> Evgheni,
> Was there any change recently to Lago slaves?
> On Fri, Oct 20, 2017 at 11:05 AM, Piotr Kliczewski <
> piotr.kliczewski(a)gmail.com> wrote:
> > I attempted to run manual OST twice and both failed with below issue.
> > Can someone take a look?
> >
> > Thanks,
> > Piotr
> >
> > 2017-10-20 07:59:12,485::log_utils.py::__exit__::607::ovirtlago.prefix:
> > :DEBUG::
> > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,
> > in wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line
> > 111, in wrapper
> > with utils.repo_server_context(args[0]):
> > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> > return self.gen.next()
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line
> > 100, in repo_server_context
> > root_dir=prefix.paths.internal_repo(),
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,
> > in _create_http_server
> > generate_request_handler(root_dir),
> > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> > self.server_bind()
> > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> > SocketServer.TCPServer.server_bind(self)
> > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> > self.socket.bind(self.server_address)
> > File "/usr/lib64/python2.7/socket.py", line 224, in meth
> > return getattr(self._sock,name)(*args)
> >
> > 2017-10-20 07:59:12,485::cmd.py::do_run::365::root::ERROR::Error
> > occured, aborting
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, in
> > do_run
> > self.cli_plugins[args.ovirtverb].do_run(args)
> > File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line
> > 184, in do_run
> > self._do_run(**vars(args))
> > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in
> > wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in
> > wrapper
> > return func(*args, prefix=prefix, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 166,
> > in do_deploy
> > prefix.deploy()
> > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636,
> > in wrapper
> > return func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line
> > 111, in wrapper
> > with utils.repo_server_context(args[0]):
> > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> > return self.gen.next()
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line
> > 100, in repo_server_context
> > root_dir=prefix.paths.internal_repo(),
> > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76,
> > in _create_http_server
> > generate_request_handler(root_dir),
> > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
> > self.server_bind()
> > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind
> > SocketServer.TCPServer.server_bind(self)
> > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
> > self.socket.bind(self.server_address)
> > File "/usr/lib64/python2.7/socket.py", line 224, in meth
> > return getattr(self._sock,name)(*args)
> > error: [Errno 98] Address already in use
> > _______________________________________________
> > Infra mailing list
> > Infra(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
> >
> --
> Eyal edri
> MANAGER
> RHV DevOps
> EMEA VIRTUALIZATION R&D
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100075)
6 years, 3 months
[JIRA] (OVIRT-1734) Add CI jobs to test slaves on staging engine
by eyal edri (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1734?page=com.atlassian.jir... ]
eyal edri commented on OVIRT-1734:
----------------------------------
I think we now have more jobs running on staging, can we close this?
> Add CI jobs to test slaves on staging engine
> --------------------------------------------
>
> Key: OVIRT-1734
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1734
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Components: oVirt CI, oVirt Infra
> Reporter: Barak Korren
> Assignee: infra
> Labels: staging
>
> After the staging engine was installed, several VMs were created on it and added as slaves to the staging Jenkins.
> We need to add some jobs that will take advantage on those slaves so we can tell if they are working properly.
> The should probably just add CI jobs for the 'jenkins' repo. That repo is already configured to run some jobs on the staging Jenkins and is mostly maintained by our team, so and issues there will be visible to us.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100075)
6 years, 3 months
[JIRA] (OVIRT-1814) Re: Jenkins failing on fcraw
by eyal edri (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1814?page=com.atlassian.jir... ]
eyal edri updated OVIRT-1814:
-----------------------------
Resolution: Fixed
Status: Done (was: To Do)
vdsm fcraw jobs were disabled.
If at some point maintainers want to enable it, it should be done only after its verified to be working.
> Re: Jenkins failing on fcraw
> ----------------------------
>
> Key: OVIRT-1814
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1814
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: eyal edri
> Assignee: infra
>
> Adding infra-support to track this.
> Evgheni, can you have a look to see if its something on infra side?
> On Thu, Dec 21, 2017 at 2:10 PM, Milan Zamazal <mzamazal(a)redhat.com> wrote:
> > Hi, I experienced CI -1 on https://gerrit.ovirt.org/#/c/85669/1 due to
> > failing fcraw test. I can see that it's due to some broken package
> > dependencies on that system. I've heard that fcraw should run the tests
> > but not vote, however it put -1 score on the patch. Could it be fixed
> > please?
> >
> > Thanks,
> > Milan
> > _______________________________________________
> > Infra mailing list
> > Infra(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
> >
> --
> Eyal edri
> MANAGER
> RHV DevOps
> EMEA VIRTUALIZATION R&D
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100075)
6 years, 3 months