December 2017 - Infra - oVirt List Archives

[JIRA] (OVIRT-1658) Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
by eyal edri (oVirt JIRA) 26 Dec '17

26 Dec '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1658?page=com.atlassian.jira.… ] eyal edri commented on OVIRT-1658: ---------------------------------- Is it still an issue after a few 7.4.z batches were released? > Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale) > ---------------------------------------------------------------------- > > Key: OVIRT-1658 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Barak Korren > Assignee: Evgheni Dereveanchin > > Seems there is an issue with locale files in recent CentOS versions, > and we're hitting it in slaves that have been updated recently. > The symptoms are that Jenkins disconnects from the slave and then > refuses to reconnect to it. The agent log in Jenkins shows: > [09/19/17 15:42:56] [SSH] Connection closed. > [09/19/17 15:58:48] [SSH] Opening SSH connection to > vm0002.workers-phx.ovirt.org:22. > [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being > verified. Man-in-the-middle attacks may be possible against this > connection. > [09/19/17 15:58:48] [SSH] Authentication successful. > SSH connection reports a garbage before a command execution. > Check your .bashrc, .profile, and so on to make sure it is quiet. > The received junk text is as follows: > /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot > change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot > change locale (en_US.utf8): No such file or directory > null > [09/19/17 15:58:48] Launch failed - cleaning up connection > [09/19/17 15:58:48] [SSH] Connection closed. > The same locale error messages can also be reproduced on the slave by > running an interactive login from the console or 'su -'. When running > 'locale -a' you can also see the en_US.UTF-8 locale is somehow > missing. > Looking around for this I found the following: > https://github.com/CentOS/sig-cloud-instance-images/issues/71 > I tried downgrading glibc back to the version we had before, but that > did not seem to resolve the issue. Eventually I managed to resolve it > by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave. > I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is > attached to the staging Jenkins, but I've no reason to believe this > won't start impacting production slaves. > We need to research this further and find out if we need to do > something to prevent this issue from surfacing on production slaves. > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100075)

1 0

[JIRA] (OVIRT-1702) Jenkins STD-CI are running in dirty mock chroots
by eyal edri (oVirt JIRA) 26 Dec '17

26 Dec '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1702?page=com.atlassian.jira.… ] eyal edri updated OVIRT-1702: ----------------------------- Resolution: Done Status: Done (was: To Do) please reopen if its still relevant > Jenkins STD-CI are running in dirty mock chroots > ------------------------------------------------ > > Key: OVIRT-1702 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1702 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: infra > > Example: > http://jenkins.ovirt.org/job/ovirt-wgt_master_check-patch-fc26-x86_64/11/co… > *09:44:06* + ./automation/build-artifacts.sh*09:44:06* + [[ -d > exported-artifacts ]]*09:44:06* + mkdir -p > exported-artifacts*09:44:06* + [[ -d tmp.repos/SOURCES ]]*09:44:06* + > mkdir -p tmp.repos/SOURCES*09:44:06* + git clone > git://anongit.freedesktop.org/spice/spice-nsis*09:44:06* fatal: > destination path 'spice-nsis' already exists and is not an empty > directory. > jobs should run in clean chroots. > -- > SANDRO BONAZZOLA > ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > <http://www.teraplan.it/redhat-osd-2017/> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100075)

1 0

[JIRA] (OVIRT-1703) vm0152.workers-phx.ovirt.org ran out of file descriptors
by eyal edri (oVirt JIRA) 26 Dec '17

26 Dec '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1703?page=com.atlassian.jira.… ] eyal edri reassigned OVIRT-1703: -------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > vm0152.workers-phx.ovirt.org ran out of file descriptors > -------------------------------------------------------- > > Key: OVIRT-1703 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1703 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Daniel Belenky > Assignee: Evgheni Dereveanchin > > Link to the failed job: [http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3161/console|…] > I've opened this ticket to track this issue and see if we see it again sometime. I wasn't able to reboot the slave from the terminal, had to reboot it from the engine. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100075)

1 0

[JIRA] (OVIRT-1712) OST jobs fails on "address already in use"
by eyal edri (oVirt JIRA) 26 Dec '17

26 Dec '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1712?page=com.atlassian.jira.… ] eyal edri updated OVIRT-1712: ----------------------------- Status: In Progress (was: To Do) > OST jobs fails on "address already in use" > ------------------------------------------ > > Key: OVIRT-1712 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1712 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: eyal edri > Assignee: Gal Ben Haim > Priority: High > > Evgheni, > Was there any change recently to Lago slaves? > On Fri, Oct 20, 2017 at 11:05 AM, Piotr Kliczewski < > piotr.kliczewski(a)gmail.com> wrote: > > I attempted to run manual OST twice and both failed with below issue. > > Can someone take a look? > > > > Thanks, > > Piotr > > > > 2017-10-20 07:59:12,485::log_utils.py::__exit__::607::ovirtlago.prefix: > > :DEBUG:: > > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, > > in wrapper > > return func(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line > > 111, in wrapper > > with utils.repo_server_context(args[0]): > > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ > > return self.gen.next() > > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line > > 100, in repo_server_context > > root_dir=prefix.paths.internal_repo(), > > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76, > > in _create_http_server > > generate_request_handler(root_dir), > > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ > > self.server_bind() > > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind > > SocketServer.TCPServer.server_bind(self) > > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind > > self.socket.bind(self.server_address) > > File "/usr/lib64/python2.7/socket.py", line 224, in meth > > return getattr(self._sock,name)(*args) > > > > 2017-10-20 07:59:12,485::cmd.py::do_run::365::root::ERROR::Error > > occured, aborting > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, in > > do_run > > self.cli_plugins[args.ovirtverb].do_run(args) > > File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line > > 184, in do_run > > self._do_run(**vars(args)) > > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in > > wrapper > > return func(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in > > wrapper > > return func(*args, prefix=prefix, **kwargs) > > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 166, > > in do_deploy > > prefix.deploy() > > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, > > in wrapper > > return func(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line > > 111, in wrapper > > with utils.repo_server_context(args[0]): > > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ > > return self.gen.next() > > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line > > 100, in repo_server_context > > root_dir=prefix.paths.internal_repo(), > > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76, > > in _create_http_server > > generate_request_handler(root_dir), > > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ > > self.server_bind() > > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind > > SocketServer.TCPServer.server_bind(self) > > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind > > self.socket.bind(self.server_address) > > File "/usr/lib64/python2.7/socket.py", line 224, in meth > > return getattr(self._sock,name)(*args) > > error: [Errno 98] Address already in use > > _______________________________________________ > > Infra mailing list > > Infra(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > > -- > Eyal edri > MANAGER > RHV DevOps > EMEA VIRTUALIZATION R&D > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100075)

1 0

[JIRA] (OVIRT-1712) OST jobs fails on "address already in use"
by eyal edri (oVirt JIRA) 26 Dec '17

26 Dec '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1712?page=com.atlassian.jira.… ] eyal edri updated OVIRT-1712: ----------------------------- Priority: High (was: Medium) > OST jobs fails on "address already in use" > ------------------------------------------ > > Key: OVIRT-1712 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1712 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: eyal edri > Assignee: Gal Ben Haim > Priority: High > > Evgheni, > Was there any change recently to Lago slaves? > On Fri, Oct 20, 2017 at 11:05 AM, Piotr Kliczewski < > piotr.kliczewski(a)gmail.com> wrote: > > I attempted to run manual OST twice and both failed with below issue. > > Can someone take a look? > > > > Thanks, > > Piotr > > > > 2017-10-20 07:59:12,485::log_utils.py::__exit__::607::ovirtlago.prefix: > > :DEBUG:: > > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, > > in wrapper > > return func(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line > > 111, in wrapper > > with utils.repo_server_context(args[0]): > > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ > > return self.gen.next() > > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line > > 100, in repo_server_context > > root_dir=prefix.paths.internal_repo(), > > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76, > > in _create_http_server > > generate_request_handler(root_dir), > > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ > > self.server_bind() > > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind > > SocketServer.TCPServer.server_bind(self) > > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind > > self.socket.bind(self.server_address) > > File "/usr/lib64/python2.7/socket.py", line 224, in meth > > return getattr(self._sock,name)(*args) > > > > 2017-10-20 07:59:12,485::cmd.py::do_run::365::root::ERROR::Error > > occured, aborting > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, in > > do_run > > self.cli_plugins[args.ovirtverb].do_run(args) > > File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line > > 184, in do_run > > self._do_run(**vars(args)) > > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in > > wrapper > > return func(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in > > wrapper > > return func(*args, prefix=prefix, **kwargs) > > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 166, > > in do_deploy > > prefix.deploy() > > File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, > > in wrapper > > return func(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line > > 111, in wrapper > > with utils.repo_server_context(args[0]): > > File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ > > return self.gen.next() > > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line > > 100, in repo_server_context > > root_dir=prefix.paths.internal_repo(), > > File "/usr/lib/python2.7/site-packages/ovirtlago/utils.py", line 76, > > in _create_http_server > > generate_request_handler(root_dir), > > File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ > > self.server_bind() > > File "/usr/lib64/python2.7/BaseHTTPServer.py", line 108, in server_bind > > SocketServer.TCPServer.server_bind(self) > > File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind > > self.socket.bind(self.server_address) > > File "/usr/lib64/python2.7/socket.py", line 224, in meth > > return getattr(self._sock,name)(*args) > > error: [Errno 98] Address already in use > > _______________________________________________ > > Infra mailing list > > Infra(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > > -- > Eyal edri > MANAGER > RHV DevOps > EMEA VIRTUALIZATION R&D > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100075)

1 0

[JIRA] (OVIRT-1734) Add CI jobs to test slaves on staging engine
by eyal edri (oVirt JIRA) 26 Dec '17

26 Dec '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1734?page=com.atlassian.jira.… ] eyal edri commented on OVIRT-1734: ---------------------------------- I think we now have more jobs running on staging, can we close this? > Add CI jobs to test slaves on staging engine > -------------------------------------------- > > Key: OVIRT-1734 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1734 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Components: oVirt CI, oVirt Infra > Reporter: Barak Korren > Assignee: infra > Labels: staging > > After the staging engine was installed, several VMs were created on it and added as slaves to the staging Jenkins. > We need to add some jobs that will take advantage on those slaves so we can tell if they are working properly. > The should probably just add CI jobs for the 'jenkins' repo. That repo is already configured to run some jobs on the staging Jenkins and is mostly maintained by our team, so and issues there will be visible to us. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100075)

1 0

[JIRA] (OVIRT-1738) configure tuned on jenkins slaves
by eyal edri (oVirt JIRA) 26 Dec '17

26 Dec '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1738?page=com.atlassian.jira.… ] eyal edri updated OVIRT-1738: ----------------------------- Resolution: Duplicate Status: Done (was: To Do) https://ovirt-jira.atlassian.net/browse/OVIRT-1668 > configure tuned on jenkins slaves > --------------------------------- > > Key: OVIRT-1738 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1738 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: oVirt Infra > Reporter: sbonazzo > Assignee: infra > > Just noticed we are not tuning our slaves missing tuned on the systems (at least the one I checked vm0160.workers-phx.ovirt.org ) > According to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ht… > we should gain a bit by setting virtual-guest profile within the slaves. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100075)

1 0

[JIRA] (OVIRT-1668) Load the right tuned profile on CentOS
by eyal edri (oVirt JIRA) 26 Dec '17

26 Dec '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1668?page=com.atlassian.jira.… ] eyal edri reassigned OVIRT-1668: -------------------------------- Assignee: Evgheni Dereveanchin (was: infra) Another option to add to global_setup maybe. > Load the right tuned profile on CentOS > -------------------------------------- > > Key: OVIRT-1668 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1668 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: oVirt CI, oVirt Infra > Reporter: Barak Korren > Assignee: Evgheni Dereveanchin > Priority: High > Labels: jenkins, performance, slaves > > There was a story published this weekend explaining that in CentOS, the {{tuned}} performance profile that is selected by default is probably not right for servers: > http://jperrin.org/centos/boosting-centos-server-performance/ > We need to ensure that we select the right profile for our systems. > We should probably just do this from '{{global_setup.sh}}' -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100075)

1 0

[JIRA] (OVIRT-1814) Re: Jenkins failing on fcraw
by eyal edri (oVirt JIRA) 26 Dec '17

26 Dec '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1814?page=com.atlassian.jira.… ] eyal edri updated OVIRT-1814: ----------------------------- Resolution: Fixed Status: Done (was: To Do) vdsm fcraw jobs were disabled. If at some point maintainers want to enable it, it should be done only after its verified to be working. > Re: Jenkins failing on fcraw > ---------------------------- > > Key: OVIRT-1814 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1814 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: eyal edri > Assignee: infra > > Adding infra-support to track this. > Evgheni, can you have a look to see if its something on infra side? > On Thu, Dec 21, 2017 at 2:10 PM, Milan Zamazal <mzamazal(a)redhat.com> wrote: > > Hi, I experienced CI -1 on https://gerrit.ovirt.org/#/c/85669/1 due to > > failing fcraw test. I can see that it's due to some broken package > > dependencies on that system. I've heard that fcraw should run the tests > > but not vote, however it put -1 score on the patch. Could it be fixed > > please? > > > > Thanks, > > Milan > > _______________________________________________ > > Infra mailing list > > Infra(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > > -- > Eyal edri > MANAGER > RHV DevOps > EMEA VIRTUALIZATION R&D > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100075)

1 0

[JIRA] (OVIRT-1816) Vdsm builds for fc27 and fcraw enabled by mistake?
by eyal edri (oVirt JIRA) 26 Dec '17

26 Dec '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1816?page=com.atlassian.jira.… ] eyal edri updated OVIRT-1816: ----------------------------- Resolution: Fixed Status: Done (was: To Do) jobs disabled for now, please update if/when they should be enabled. Also, if this is something that won't be enabled soon, it might be better to remove it from the YAML > Vdsm builds for fc27 and fcraw enabled by mistake? > -------------------------------------------------- > > Key: OVIRT-1816 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1816 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > We tried to enable fc27 and fcraw builds few weeks ago, but we found > that some packages are still missing (I think the only missing package now > is ioprocess). The jobs were disabled until we resolve this. > For some reason the jobs were activated yesterday, and now vdsm patches > fail again: > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fcraw-x86_64/86/ > Please disable the fc27 and fcraw jobs until we resolve the missing > packages. > Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100075)

1 0