- Infra - oVirt List Archives

[JIRA] (OVIRT-1658) Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
by Evgheni Dereveanchin (oVirt JIRA) 20 Sep '17

20 Sep '17

This is a multi-part message in MIME format... ------------=_1505906876-10461-2701 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit [ https://ovirt-jira.atlassian.net/browse/OVIRT-1658?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-1658: --------------------------------------------- Seen this as well. Logged a CentOS bug and will run the localedef command manually across our systems to avoid this breaking stuff. > Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale) > ---------------------------------------------------------------------- > > Key: OVIRT-1658 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Barak Korren > Assignee: Evgheni Dereveanchin > > Seems there is an issue with locale files in recent CentOS versions, > and we're hitting it in slaves that have been updated recently. > The symptoms are that Jenkins disconnects from the slave and then > refuses to reconnect to it. The agent log in Jenkins shows: > [09/19/17 15:42:56] [SSH] Connection closed. > [09/19/17 15:58:48] [SSH] Opening SSH connection to > vm0002.workers-phx.ovirt.org:22. > [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being > verified. Man-in-the-middle attacks may be possible against this > connection. > [09/19/17 15:58:48] [SSH] Authentication successful. > SSH connection reports a garbage before a command execution. > Check your .bashrc, .profile, and so on to make sure it is quiet. > The received junk text is as follows: > /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot > change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot > change locale (en_US.utf8): No such file or directory > null > [09/19/17 15:58:48] Launch failed - cleaning up connection > [09/19/17 15:58:48] [SSH] Connection closed. > The same locale error messages can also be reproduced on the slave by > running an interactive login from the console or 'su -'. When running > 'locale -a' you can also see the en_US.UTF-8 locale is somehow > missing. > Looking around for this I found the following: > https://github.com/CentOS/sig-cloud-instance-images/issues/71 > I tried downgrading glibc back to the version we had before, but that > did not seem to resolve the issue. Eventually I managed to resolve it > by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave. > I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is > attached to the staging Jenkins, but I've no reason to believe this > won't start impacting production slaves. > We need to research this further and find out if we need to do > something to prevent this issue from surfacing on production slaves. > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted -- This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060) ------------=_1505906876-10461-2701 Content-Type: text/html; charset="UTF-8" Content-Disposition: inline Content-Transfer-Encoding: 7bit <html><body> <pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1658?page=com.atlassian.jira.… ]</pre> <h3>Evgheni Dereveanchin commented on OVIRT-1658:</h3> Seen this as well. Logged a CentOS bug and will run the localedef command manually across our systems to avoid this breaking stuff. <blockquote><h3>Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)</h3> <pre> Key: OVIRT-1658 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Barak Korren Assignee: Evgheni Dereveanchin</pre> Seems there is an issue with locale files in recent CentOS versions, and we're hitting it in slaves that have been updated recently. The symptoms are that Jenkins disconnects from the slave and then refuses to reconnect to it. The agent log in Jenkins shows: [09/19/17 15:42:56] [SSH] Connection closed. [09/19/17 15:58:48] [SSH] Opening SSH connection to vm0002.workers-phx.ovirt.org:22. [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. [09/19/17 15:58:48] [SSH] Authentication successful. SSH connection reports a garbage before a command execution. Check your .bashrc, .profile, and so on to make sure it is quiet. The received junk text is as follows: /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.utf8): No such file or directory null [09/19/17 15:58:48] Launch failed – cleaning up connection [09/19/17 15:58:48] [SSH] Connection closed. The same locale error messages can also be reproduced on the slave by running an interactive login from the console or ‘su -’. When running ‘locale -a’ you can also see the en_US.UTF-8 locale is somehow missing. Looking around for this I found the following: <a href="https://github.com/CentOS/sig-cloud-instance-images/issues/71">https://github.com/CentOS/sig-cloud-instance-images/issues/71</a> I tried downgrading glibc back to the version we had before, but that did not seem to resolve the issue. Eventually I managed to resolve it by running ‘localedef -i en_US -f UTF-8 en_US.UTF-8’ on the slave. I've seen this happen on ‘vm0002.workers-phx.ovirt.org’ which is attached to the staging Jenkins, but I've no reason to believe this won't start impacting production slaves. We need to research this further and find out if we need to do something to prevent this issue from surfacing on production slaves. — Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted</blockquote> — This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060) <img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYLyz…" alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/> </body></html> ------------=_1505906876-10461-2701--

1 0

[CQ]: 82022, 1 (ovirt-engine-sdk-ruby) failed "ovirt-master" system tests
by oVirt Jenkins 20 Sep '17

20 Sep '17

Change 82022,1 (ovirt-engine-sdk-ruby) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/82022/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/

1 0

Subject: [ OST Failure Report ] [ master ] [ 20-09-2017] [ No Test Failed - missing packages in fc25 ]
by Dafna Ron 20 Sep '17

20 Sep '17

This is a multi-part message in MIME format. --------------60D60522EFE450E2C842A076 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, CQ reported a failure in one of the builds and looking at the console I found that we had a failure before the OST started due to: ovirt-hosted-engine-ha_master_build-artifacts-fc25-x86_64 (67) failed building The build failed of ovirt-hosted-engine-ha_master_build-artifacts-fc25-x86_64 failed due to missing packages. * Link to suspected patches: patch reported as failed is https://gerrit.ovirt.org/#/c/81792/ ** Link to Job: *cq job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2732/console* *failed build job: http://jenkins.ovirt.org/job/ovirt-hosted-engine-ha_master_build-artifacts-… Link to all logs: *CQ job logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2732/artifact… *failed build log: http://jenkins.ovirt.org/job/ovirt-hosted-engine-ha_master_build-artifacts-… ** *(Relevant) error snippet from the log: * *<error>* **07:46:21* Getting requirements for ovirt-hosted-engine-ha.spec *07:46:21* --> Already installed : python-2.7.13-2.fc25.x86_64 *07:46:21* --> python2-mock-2.0.0-2.fc25.noarch *07:46:22* --> python2-nose-1.3.7-11.fc25.noarch *07:46:22* --> Already installed : python-devel-2.7.13-2.fc25.x86_64 *07:46:22* --> Already installed : systemd-231-17.fc25.x86_64 *07:46:22* --> vdsm-4.20.3-47.git4801123.fc25.x86_64 *07:46:22* --> vdsm-client-4.20.3-47.git4801123.fc25.noarch *07:46:22* --> vdsm-python-4.20.3-47.git4801123.fc25.noarch *07:46:31* Error: Package: vdsm-4.20.3-47.git4801123.fc25.x86_64 (ovirt-master-snapshot) *07:46:31* Requires: openvswitch >= 2.7.0 *07:46:31* Available: openvswitch-2.5.0-4.fc25.x86_64 (fedora-base-fc25) *07:46:31* openvswitch = 2.5.0-4.fc25 *07:46:31* You could try using --skip-broken to work around the problem *07:46:31* You could try running: rpm -Va --nofiles --nodigest *07:46:31* Took 18 seconds* </error> *Thanks, Dafna --------------60D60522EFE450E2C842A076 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGh0bWw+CiAgPGhlYWQ+CgogICAgPG1ldGEgaHR0cC1lcXVpdj0iY29udGVudC10eXBlIiBj b250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPgogIDwvaGVhZD4KICA8Ym9keSB0 ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4KICAgIDxwPkhpLCA8YnI+CiAgICA8 L3A+CiAgICA8cD5DUSByZXBvcnRlZCBhIGZhaWx1cmUgaW4gb25lIG9mIHRoZSBidWlsZHMg YW5kIGxvb2tpbmcgYXQgdGhlCiAgICAgIGNvbnNvbGUgSSBmb3VuZCB0aGF0IHdlIGhhZCBh IGZhaWx1cmUgYmVmb3JlIHRoZSBPU1Qgc3RhcnRlZCBkdWUKICAgICAgdG86IDxicj4KICAg IDwvcD4KICAgIDxwcmUgY2xhc3M9ImNvbnNvbGUtb3V0cHV0IiBzdHlsZT0iYm94LXNpemlu ZzogYm9yZGVyLWJveDsgd2hpdGUtc3BhY2U6IHByZS13cmFwOyB3b3JkLXdyYXA6IGJyZWFr LXdvcmQ7IG1hcmdpbjogMHB4OyBjb2xvcjogcmdiKDUxLCA1MSwgNTEpOyBmb250LXNpemU6 IDE0cHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWxpZ2F0dXJlczogbm9y bWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogbm9ybWFsOyBs ZXR0ZXItc3BhY2luZzogbm9ybWFsOyBvcnBoYW5zOiAyOyB0ZXh0LWFsaWduOiBzdGFydDsg dGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdpZG93czogMjsgd29y ZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1k ZWNvcmF0aW9uLXN0eWxlOiBpbml0aWFsOyB0ZXh0LWRlY29yYXRpb24tY29sb3I6IGluaXRp YWw7Ij5vdmlydC1ob3N0ZWQtZW5naW5lLWhhX21hc3Rlcl9idWlsZC1hcnRpZmFjdHMtZmMy NS14ODZfNjQgKDY3KSBmYWlsZWQgYnVpbGRpbmcKClRoZSBidWlsZCBmYWlsZWQgb2Ygb3Zp cnQtaG9zdGVkLWVuZ2luZS1oYV9tYXN0ZXJfYnVpbGQtYXJ0aWZhY3RzLWZjMjUteDg2XzY0 IGZhaWxlZCBkdWUgdG8gbWlzc2luZyBwYWNrYWdlcy4gCgo8YiBzdHlsZT0iZm9udC13ZWln aHQ6bm9ybWFsOyIgaWQ9ImRvY3MtaW50ZXJuYWwtZ3VpZC01ODU5YjdhMS05ZWI0LTgyYjct NDgwNS05Njc3MTljNjIxYTkiPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4z ODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdDsiPjxzcGFuIHN0eWxlPSJmb250 LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQt Y29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2Zv bnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246 YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXA7Ij5MaW5rIHRvIHN1c3BlY3RlZCBwYXRj aGVzOiAgcGF0Y2ggcmVwb3J0ZWQgYXMgZmFpbGVkIGlzIDxhIGNsYXNzPSJtb3otdHh0LWxp bmstZnJlZXRleHQiIGhyZWY9Imh0dHBzOi8vZ2Vycml0Lm92aXJ0Lm9yZy8jL2MvODE3OTIv Ij5odHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9jLzgxNzkyLzwvYT48L3NwYW4+PC9wPjwv Yj48YiBzdHlsZT0iZm9udC13ZWlnaHQ6bm9ybWFsOyIgaWQ9ImRvY3MtaW50ZXJuYWwtZ3Vp ZC01ODU5YjdhMS05ZWI0LTgyYjctNDgwNS05Njc3MTljNjIxYTkiPgo8cCBkaXI9Imx0ciIg c3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTow cHQ7Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29s b3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQw MDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlv bjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwOyI+ TGluayB0byBKb2I6IAo8L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWln aHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdDsiPjxzcGFuIHN0eWxl PSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tn cm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9y bWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwt YWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXA7Ij48Yj5jcSBqb2I6IDxhIGNs YXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9qZW5raW5zLm92aXJ0 Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvMjczMi9jb25zb2xl Ij5odHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVl dWUtdGVzdGVyLzI3MzIvY29uc29sZTwvYT48L2I+PC9zcGFuPjwvcD48cCBkaXI9Imx0ciIg c3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTow cHQ7Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29s b3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQw MDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlv bjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwOyI+ PGI+ZmFpbGVkIGJ1aWxkIGpvYjogPGEgY2xhc3M9Im1vei10eHQtbGluay1mcmVldGV4dCIg aHJlZj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1ob3N0ZWQtZW5naW5l LWhhX21hc3Rlcl9idWlsZC1hcnRpZmFjdHMtZmMyNS14ODZfNjQvNjcvIj5odHRwOi8vamVu a2lucy5vdmlydC5vcmcvam9iL292aXJ0LWhvc3RlZC1lbmdpbmUtaGFfbWFzdGVyX2J1aWxk LWFydGlmYWN0cy1mYzI1LXg4Nl82NC82Ny88L2E+PC9iPgo8L3NwYW4+PC9wPgo8cCBkaXI9 Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJv dHRvbTowcHQ7Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJp YWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2Vp Z2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVj b3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13 cmFwOyI+TGluayB0byBhbGwgbG9nczogCjwvc3Bhbj48L3A+PHAgZGlyPSJsdHIiIHN0eWxl PSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0OyI+ PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMw MDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9u dC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9u ZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcDsiPjxiPkNR IGpvYiBsb2dzOiA8YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRw Oi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVz dGVyLzI3MzIvYXJ0aWZhY3QvZXhwb3J0ZWQtYXJ0aWZhY3RzLyI+aHR0cDovL2plbmtpbnMu b3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8yNzMyL2Fy dGlmYWN0L2V4cG9ydGVkLWFydGlmYWN0cy88L2E+PC9iPgo8L3NwYW4+PC9wPjxwIGRpcj0i bHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90 dG9tOjBwdDsiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlh bDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWln aHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNv cmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdy YXA7Ij48Yj5mYWlsZWQgYnVpbGQgbG9nOiA8YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0 ZXh0IiBocmVmPSJodHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LWhvc3RlZC1l bmdpbmUtaGFfbWFzdGVyX2J1aWxkLWFydGlmYWN0cy1mYzI1LXg4Nl82NC82Ny9hcnRpZmFj dC8iPmh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtaG9zdGVkLWVuZ2luZS1o YV9tYXN0ZXJfYnVpbGQtYXJ0aWZhY3RzLWZjMjUteDg2XzY0LzY3L2FydGlmYWN0LzwvYT48 L2I+Cjwvc3Bhbj48L3A+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21h cmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0OyI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6 ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xv cjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12 YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNl bGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcDsiPgo8L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBz dHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBw dDsiPgo8YiBzdHlsZT0iZm9udC13ZWlnaHQ6bm9ybWFsOyIgaWQ9ImRvY3MtaW50ZXJuYWwt Z3VpZC01ODU5YjdhMS05ZWI3LWFmOWYtNTZjNC0wY2M5M2YwZmNiMWIiPjxwIGRpcj0ibHRy IiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9t OjBwdDsiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtj b2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6 NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0 aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXA7 Ij4oUmVsZXZhbnQpIGVycm9yIHNuaXBwZXQgZnJvbSB0aGUgbG9nOiA8L3NwYW4+PC9wPjxw IGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJn aW4tYm90dG9tOjBwdDsiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWls eTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9u dC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4 dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6 cHJlLXdyYXA7Ij4mbHQ7ZXJyb3ImZ3Q7PC9zcGFuPjwvcD48L2I+CjwvcD4KPHNwYW4gY2xh c3M9InRpbWVzdGFtcCIgc3R5bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7Ij48YiBzdHls ZT0iYm94LXNpemluZzogYm9yZGVyLWJveDsiPjxzcGFuIGNsYXNzPSJ0aW1lc3RhbXAiIHN0 eWxlPSJib3gtc2l6aW5nOiBib3JkZXItYm94OyI+PGIgc3R5bGU9ImJveC1zaXppbmc6IGJv cmRlci1ib3g7Ij4wNzo0NjoyMTwvYj4gPC9zcGFuPkdldHRpbmcgcmVxdWlyZW1lbnRzIGZv ciBvdmlydC1ob3N0ZWQtZW5naW5lLWhhLnNwZWMKPHNwYW4gY2xhc3M9InRpbWVzdGFtcCIg c3R5bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7Ij48YiBzdHlsZT0iYm94LXNpemluZzog Ym9yZGVyLWJveDsiPjA3OjQ2OjIxPC9iPiA8L3NwYW4+IC0tJmd0OyBBbHJlYWR5IGluc3Rh bGxlZCA6IHB5dGhvbi0yLjcuMTMtMi5mYzI1Lng4Nl82NAo8c3BhbiBjbGFzcz0idGltZXN0 YW1wIiBzdHlsZT0iYm94LXNpemluZzogYm9yZGVyLWJveDsiPjxiIHN0eWxlPSJib3gtc2l6 aW5nOiBib3JkZXItYm94OyI+MDc6NDY6MjE8L2I+IDwvc3Bhbj4gLS0mZ3Q7IHB5dGhvbjIt bW9jay0yLjAuMC0yLmZjMjUubm9hcmNoCjxzcGFuIGNsYXNzPSJ0aW1lc3RhbXAiIHN0eWxl PSJib3gtc2l6aW5nOiBib3JkZXItYm94OyI+PGIgc3R5bGU9ImJveC1zaXppbmc6IGJvcmRl ci1ib3g7Ij4wNzo0NjoyMjwvYj4gPC9zcGFuPiAtLSZndDsgcHl0aG9uMi1ub3NlLTEuMy43 LTExLmZjMjUubm9hcmNoCjxzcGFuIGNsYXNzPSJ0aW1lc3RhbXAiIHN0eWxlPSJib3gtc2l6 aW5nOiBib3JkZXItYm94OyI+PGIgc3R5bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7Ij4w Nzo0NjoyMjwvYj4gPC9zcGFuPiAtLSZndDsgQWxyZWFkeSBpbnN0YWxsZWQgOiBweXRob24t ZGV2ZWwtMi43LjEzLTIuZmMyNS54ODZfNjQKPHNwYW4gY2xhc3M9InRpbWVzdGFtcCIgc3R5 bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7Ij48YiBzdHlsZT0iYm94LXNpemluZzogYm9y ZGVyLWJveDsiPjA3OjQ2OjIyPC9iPiA8L3NwYW4+IC0tJmd0OyBBbHJlYWR5IGluc3RhbGxl ZCA6IHN5c3RlbWQtMjMxLTE3LmZjMjUueDg2XzY0CjxzcGFuIGNsYXNzPSJ0aW1lc3RhbXAi IHN0eWxlPSJib3gtc2l6aW5nOiBib3JkZXItYm94OyI+PGIgc3R5bGU9ImJveC1zaXppbmc6 IGJvcmRlci1ib3g7Ij4wNzo0NjoyMjwvYj4gPC9zcGFuPiAtLSZndDsgdmRzbS00LjIwLjMt NDcuZ2l0NDgwMTEyMy5mYzI1Lng4Nl82NAo8c3BhbiBjbGFzcz0idGltZXN0YW1wIiBzdHls ZT0iYm94LXNpemluZzogYm9yZGVyLWJveDsiPjxiIHN0eWxlPSJib3gtc2l6aW5nOiBib3Jk ZXItYm94OyI+MDc6NDY6MjI8L2I+IDwvc3Bhbj4gLS0mZ3Q7IHZkc20tY2xpZW50LTQuMjAu My00Ny5naXQ0ODAxMTIzLmZjMjUubm9hcmNoCjxzcGFuIGNsYXNzPSJ0aW1lc3RhbXAiIHN0 eWxlPSJib3gtc2l6aW5nOiBib3JkZXItYm94OyI+PGIgc3R5bGU9ImJveC1zaXppbmc6IGJv cmRlci1ib3g7Ij4wNzo0NjoyMjwvYj4gPC9zcGFuPiAtLSZndDsgdmRzbS1weXRob24tNC4y MC4zLTQ3LmdpdDQ4MDExMjMuZmMyNS5ub2FyY2gKPHNwYW4gY2xhc3M9InRpbWVzdGFtcCIg c3R5bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7Ij48YiBzdHlsZT0iYm94LXNpemluZzog Ym9yZGVyLWJveDsiPjA3OjQ2OjMxPC9iPiA8L3NwYW4+RXJyb3I6IFBhY2thZ2U6IHZkc20t NC4yMC4zLTQ3LmdpdDQ4MDExMjMuZmMyNS54ODZfNjQgKG92aXJ0LW1hc3Rlci1zbmFwc2hv dCkKPHNwYW4gY2xhc3M9InRpbWVzdGFtcCIgc3R5bGU9ImJveC1zaXppbmc6IGJvcmRlci1i b3g7Ij48YiBzdHlsZT0iYm94LXNpemluZzogYm9yZGVyLWJveDsiPjA3OjQ2OjMxPC9iPiA8 L3NwYW4+ICAgICAgICAgICBSZXF1aXJlczogb3BlbnZzd2l0Y2ggJmd0Oz0gMi43LjAKPHNw YW4gY2xhc3M9InRpbWVzdGFtcCIgc3R5bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7Ij48 YiBzdHlsZT0iYm94LXNpemluZzogYm9yZGVyLWJveDsiPjA3OjQ2OjMxPC9iPiA8L3NwYW4+ ICAgICAgICAgICBBdmFpbGFibGU6IG9wZW52c3dpdGNoLTIuNS4wLTQuZmMyNS54ODZfNjQg KGZlZG9yYS1iYXNlLWZjMjUpCjxzcGFuIGNsYXNzPSJ0aW1lc3RhbXAiIHN0eWxlPSJib3gt c2l6aW5nOiBib3JkZXItYm94OyI+PGIgc3R5bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7 Ij4wNzo0NjozMTwvYj4gPC9zcGFuPiAgICAgICAgICAgICAgIG9wZW52c3dpdGNoID0gMi41 LjAtNC5mYzI1CjxzcGFuIGNsYXNzPSJ0aW1lc3RhbXAiIHN0eWxlPSJib3gtc2l6aW5nOiBi b3JkZXItYm94OyI+PGIgc3R5bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7Ij4wNzo0Njoz MTwvYj4gPC9zcGFuPiBZb3UgY291bGQgdHJ5IHVzaW5nIC0tc2tpcC1icm9rZW4gdG8gd29y ayBhcm91bmQgdGhlIHByb2JsZW0KPHNwYW4gY2xhc3M9InRpbWVzdGFtcCIgc3R5bGU9ImJv eC1zaXppbmc6IGJvcmRlci1ib3g7Ij48YiBzdHlsZT0iYm94LXNpemluZzogYm9yZGVyLWJv eDsiPjA3OjQ2OjMxPC9iPiA8L3NwYW4+IFlvdSBjb3VsZCB0cnkgcnVubmluZzogcnBtIC1W YSAtLW5vZmlsZXMgLS1ub2RpZ2VzdAo8c3BhbiBjbGFzcz0idGltZXN0YW1wIiBzdHlsZT0i Ym94LXNpemluZzogYm9yZGVyLWJveDsiPjxiIHN0eWxlPSJib3gtc2l6aW5nOiBib3JkZXIt Ym94OyI+MDc6NDY6MzE8L2I+IDwvc3Bhbj5Ub29rIDE4IHNlY29uZHM8L2I+PC9zcGFuPgoK CjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDtt YXJnaW4tYm90dG9tOjBwdDsiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZh bWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7 Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7 dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3Bh Y2U6cHJlLXdyYXA7Ij4mbHQ7L2Vycm9yJmd0Ozwvc3Bhbj48L3A+CjwvYj5UaGFua3MsIApE YWZuYQrCoDwvcHJlPgogIDwvYm9keT4KPC9odG1sPgo= --------------60D60522EFE450E2C842A076--

2 2

[JIRA] (OVIRT-1658) Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
by Evgheni Dereveanchin (oVirt JIRA) 20 Sep '17

20 Sep '17

This is a multi-part message in MIME format... ------------=_1505900748-14133-1694 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit [ https://ovirt-jira.atlassian.net/browse/OVIRT-1658?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-1658: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale) > ---------------------------------------------------------------------- > > Key: OVIRT-1658 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Barak Korren > Assignee: Evgheni Dereveanchin > > Seems there is an issue with locale files in recent CentOS versions, > and we're hitting it in slaves that have been updated recently. > The symptoms are that Jenkins disconnects from the slave and then > refuses to reconnect to it. The agent log in Jenkins shows: > [09/19/17 15:42:56] [SSH] Connection closed. > [09/19/17 15:58:48] [SSH] Opening SSH connection to > vm0002.workers-phx.ovirt.org:22. > [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being > verified. Man-in-the-middle attacks may be possible against this > connection. > [09/19/17 15:58:48] [SSH] Authentication successful. > SSH connection reports a garbage before a command execution. > Check your .bashrc, .profile, and so on to make sure it is quiet. > The received junk text is as follows: > /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot > change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot > change locale (en_US.utf8): No such file or directory > null > [09/19/17 15:58:48] Launch failed - cleaning up connection > [09/19/17 15:58:48] [SSH] Connection closed. > The same locale error messages can also be reproduced on the slave by > running an interactive login from the console or 'su -'. When running > 'locale -a' you can also see the en_US.UTF-8 locale is somehow > missing. > Looking around for this I found the following: > https://github.com/CentOS/sig-cloud-instance-images/issues/71 > I tried downgrading glibc back to the version we had before, but that > did not seem to resolve the issue. Eventually I managed to resolve it > by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave. > I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is > attached to the staging Jenkins, but I've no reason to believe this > won't start impacting production slaves. > We need to research this further and find out if we need to do > something to prevent this issue from surfacing on production slaves. > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted -- This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060) ------------=_1505900748-14133-1694 Content-Type: text/html; charset="UTF-8" Content-Disposition: inline Content-Transfer-Encoding: 7bit <html><body> <pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1658?page=com.atlassian.jira.… ]</pre> <h3>Evgheni Dereveanchin reassigned OVIRT-1658:</h3> <pre>Assignee: Evgheni Dereveanchin (was: infra)</pre> <blockquote><h3>Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)</h3> <pre> Key: OVIRT-1658 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Barak Korren Assignee: Evgheni Dereveanchin</pre> Seems there is an issue with locale files in recent CentOS versions, and we're hitting it in slaves that have been updated recently. The symptoms are that Jenkins disconnects from the slave and then refuses to reconnect to it. The agent log in Jenkins shows: [09/19/17 15:42:56] [SSH] Connection closed. [09/19/17 15:58:48] [SSH] Opening SSH connection to vm0002.workers-phx.ovirt.org:22. [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. [09/19/17 15:58:48] [SSH] Authentication successful. SSH connection reports a garbage before a command execution. Check your .bashrc, .profile, and so on to make sure it is quiet. The received junk text is as follows: /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.utf8): No such file or directory null [09/19/17 15:58:48] Launch failed – cleaning up connection [09/19/17 15:58:48] [SSH] Connection closed. The same locale error messages can also be reproduced on the slave by running an interactive login from the console or ‘su -’. When running ‘locale -a’ you can also see the en_US.UTF-8 locale is somehow missing. Looking around for this I found the following: <a href="https://github.com/CentOS/sig-cloud-instance-images/issues/71">https://github.com/CentOS/sig-cloud-instance-images/issues/71</a> I tried downgrading glibc back to the version we had before, but that did not seem to resolve the issue. Eventually I managed to resolve it by running ‘localedef -i en_US -f UTF-8 en_US.UTF-8’ on the slave. I've seen this happen on ‘vm0002.workers-phx.ovirt.org’ which is attached to the staging Jenkins, but I've no reason to believe this won't start impacting production slaves. We need to research this further and find out if we need to do something to prevent this issue from surfacing on production slaves. — Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted</blockquote> — This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060) <img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYLyz…" alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/> </body></html> ------------=_1505900748-14133-1694--

1 0

Subject: [ OST Failure Report ] [ master ] [ 19-09-2017E ] [ 004_basic_sanity ]
by Dafna Ron 20 Sep '17

20 Sep '17

This is a multi-part message in MIME format. --------------38C81434D2056F5745CD1AD2 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, Lago reports add_console was the failed action: <testcase classname="004_basic_sanity" name="add_console" time="1.765">...</testcase> In the engine log I can see that both hosts fail to respond and that we have also have errors in ansible for invalid services in fedora. CQ reports a network patch as the probable cause. See further info below ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/81921/* * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifact… (Relevant) error snippet from the log: <error> * **Here is the ansible error: *** * host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ovirt-vmconsole'}) => {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'glusterfs'}) => {"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ctdb'}) => {"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ovirt-storageconsole'}) => {"failed": true, "item": {"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-storageconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'nrpe'}) => {"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ovirt-vmconsole'}) => {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'glusterfs'}) => {"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ctdb'}) => {"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ovirt-storageconsole'}) => {"failed": true, "item": {"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-storageconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'nrpe'}) => {"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} **Here is the engine.log error** *017-09-19 12:28:59,191-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "634fae71-5f8f-49c1-9d5f-ebc24239cfa9", "result": true} 2017-09-19 12:28:59,191-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "634fae71-5f8f-49c1-9d5f-ebc24239cfa9" 2017-09-19 12:28:59,194-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [5438f56c] MESSAGE content-length:80 destination:jms.topic.vdsm_responses content-type:application/json subscription:f7138b74-fed3-4eb2-9fdc-957bc4af8f23 {"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true}^@ 2017-09-19 12:28:59,194-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true} 2017-09-19 12:28:59,194-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b" 2017-09-19 12:28:59,234-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [5438f56c] MESSAGE content-length:8534 destination:jms.topic.vdsm_responses content-type:application/json subscription:279e9c67-7fc4-4ace-8709-2534f4d53241 * *</error>* ** Thanks. Dafna --------------38C81434D2056F5745CD1AD2 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, <div class="collapsible-content" style="color: rgb(0, 0, 0); font-family: "Times New Roman"; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"> <div class="collapsible" id="collapsible1"> <div class="collapsed hidden"> <div class="line">Lago reports add_console was the failed action: <testcase classname="004_basic_sanity" name="add_console" time="1.765">...</testcase> In the engine log I can see that both hosts fail to respond and that we have also have errors in ansible for invalid services in fedora. CQ reports a network patch as the probable cause. See further info below </div> </div> </div> </div> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/81921/">https://gerrit.ovirt.org/#/c/81921/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifact…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifact…</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> Here is the ansible error: host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ovirt-vmconsole'}) => {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'glusterfs'}) => {"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ctdb'}) => {"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ovirt-storageconsole'}) => {"failed": true, "item": {"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-storageconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'nrpe'}) => {"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ovirt-vmconsole'}) => {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'glusterfs'}) => {"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ctdb'}) => {"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ovirt-storageconsole'}) => {"failed": true, "item": {"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-storageconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'nrpe'}) => {"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} Here is the engine.log error 017-09-19 12:28:59,191-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "634fae71-5f8f-49c1-9d5f-ebc24239cfa9", "result": true} 2017-09-19 12:28:59,191-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "634fae71-5f8f-49c1-9d5f-ebc24239cfa9" 2017-09-19 12:28:59,194-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [5438f56c] MESSAGE content-length:80 destination:jms.topic.vdsm_responses content-type:application/json subscription:f7138b74-fed3-4eb2-9fdc-957bc4af8f23 {"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true}^@ 2017-09-19 12:28:59,194-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true} 2017-09-19 12:28:59,194-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b" 2017-09-19 12:28:59,234-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [5438f56c] MESSAGE content-length:8534 destination:jms.topic.vdsm_responses content-type:application/json subscription:279e9c67-7fc4-4ace-8709-2534f4d53241 <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> Thanks. Dafna </body> </html> --------------38C81434D2056F5745CD1AD2--

1 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #797
by jenkins＠jenkins.phx.ovirt.org 20 Sep '17

20 Sep '17

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/changes/13/75913/5:patch --prune > git rev-parse origin/patch^{commit} # timeout=10 > git rev-parse patch^{commit} # timeout=10 Checking out Revision 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f (patch) Commit message: "Exclude big packages from mirrors" > git config core.sparsecheckout # timeout=10 > git checkout -f 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f > git rev-list 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins664955239406877516.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1478, in _commonLoadRepoXML self._revertOldRepoXML() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1323, in _revertOldRepoXML os.rename(old_data['old_local'], old_data['local']) OSError: [Errno 2] No such file or directory Build step 'Execute shell' marked build as failure

1 2

[CQ]: 81792, 4 (ovirt-hosted-engine-ha) failed "ovirt-master" system tests
by oVirt Jenkins 20 Sep '17

20 Sep '17

Change 81792,4 (ovirt-hosted-engine-ha) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/81792/4 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2732/

1 0

[oVirt Jenkins] ovirt-node-ng_ovirt-4.1-snapshot_build-artifacts-el7-x86_64 - Build # 259 - Failure!
by jenkins＠jenkins.phx.ovirt.org 20 Sep '17

20 Sep '17

------=_Part_620_18096676.1505841561014 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.1-snapshot_build-artifac… Build: http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.1-snapshot_build-artifac… Build Number: 259 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #259 [Yuval Turgeman] automation: adding loop devices ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_620_18096676.1505841561014--

1 1

[JIRA] (OVIRT-1658) Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
by sbonazzo (oVirt JIRA) 20 Sep '17

20 Sep '17

This is a multi-part message in MIME format... ------------=_1505882859-1664-2496 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit [ https://ovirt-jira.atlassian.net/browse/OVIRT-1658?page=com.atlassian.jira.… ] sbonazzo commented on OVIRT-1658: --------------------------------- Please open a bug on bugs.centos.org to let them know about the issue Il 19 Set 2017 18:10, "Barak Korren" <bkorren(a)redhat.com> ha scritto: > Seems there is an issue with locale files in recent CentOS versions, > and we're hitting it in slaves that have been updated recently. > > The symptoms are that Jenkins disconnects from the slave and then > refuses to reconnect to it. The agent log in Jenkins shows: > > [09/19/17 15:42:56] [SSH] Connection closed. > > [09/19/17 15:58:48] [SSH] Opening SSH connection to > vm0002.workers-phx.ovirt.org:22. > [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being > verified. Man-in-the-middle attacks may be possible against this > connection. > > [09/19/17 15:58:48] [SSH] Authentication successful. > SSH connection reports a garbage before a command execution. > Check your .bashrc, .profile, and so on to make sure it is quiet. > The received junk text is as follows: > /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot > change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot > change locale (en_US.utf8): No such file or directory > > null > [09/19/17 15:58:48] Launch failed - cleaning up connection > [09/19/17 15:58:48] [SSH] Connection closed. > > > The same locale error messages can also be reproduced on the slave by > running an interactive login from the console or 'su -'. When running > 'locale -a' you can also see the en_US.UTF-8 locale is somehow > missing. > > Looking around for this I found the following: > https://github.com/CentOS/sig-cloud-instance-images/issues/71 > > I tried downgrading glibc back to the version we had before, but that > did not seem to resolve the issue. Eventually I managed to resolve it > by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave. > > I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is > attached to the staging Jenkins, but I've no reason to believe this > won't start impacting production slaves. > > We need to research this further and find out if we need to do > something to prevent this issue from surfacing on production slaves. > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > > Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale) > ---------------------------------------------------------------------- > > Key: OVIRT-1658 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Barak Korren > Assignee: infra > > Seems there is an issue with locale files in recent CentOS versions, > and we're hitting it in slaves that have been updated recently. > The symptoms are that Jenkins disconnects from the slave and then > refuses to reconnect to it. The agent log in Jenkins shows: > [09/19/17 15:42:56] [SSH] Connection closed. > [09/19/17 15:58:48] [SSH] Opening SSH connection to > vm0002.workers-phx.ovirt.org:22. > [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being > verified. Man-in-the-middle attacks may be possible against this > connection. > [09/19/17 15:58:48] [SSH] Authentication successful. > SSH connection reports a garbage before a command execution. > Check your .bashrc, .profile, and so on to make sure it is quiet. > The received junk text is as follows: > /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot > change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot > change locale (en_US.utf8): No such file or directory > null > [09/19/17 15:58:48] Launch failed - cleaning up connection > [09/19/17 15:58:48] [SSH] Connection closed. > The same locale error messages can also be reproduced on the slave by > running an interactive login from the console or 'su -'. When running > 'locale -a' you can also see the en_US.UTF-8 locale is somehow > missing. > Looking around for this I found the following: > https://github.com/CentOS/sig-cloud-instance-images/issues/71 > I tried downgrading glibc back to the version we had before, but that > did not seem to resolve the issue. Eventually I managed to resolve it > by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave. > I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is > attached to the staging Jenkins, but I've no reason to believe this > won't start impacting production slaves. > We need to research this further and find out if we need to do > something to prevent this issue from surfacing on production slaves. > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted -- This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060) ------------=_1505882859-1664-2496 Content-Type: text/html; charset="UTF-8" Content-Disposition: inline Content-Transfer-Encoding: 7bit <html><body> <pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1658?page=com.atlassian.jira.… ]</pre> <h3>sbonazzo commented on OVIRT-1658:</h3> Please open a bug on bugs.centos.org to let them know about the issue Il 19 Set 2017 18:10, “Barak Korren” <bkorren(a)redhat.com> ha scritto: <blockquote>Seems there is an issue with locale files in recent CentOS versions, and we're hitting it in slaves that have been updated recently. The symptoms are that Jenkins disconnects from the slave and then refuses to reconnect to it. The agent log in Jenkins shows: [09/19/17 15:42:56] [SSH] Connection closed. [09/19/17 15:58:48] [SSH] Opening SSH connection to vm0002.workers-phx.ovirt.org:22. [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. [09/19/17 15:58:48] [SSH] Authentication successful. SSH connection reports a garbage before a command execution. Check your .bashrc, .profile, and so on to make sure it is quiet. The received junk text is as follows: /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.utf8): No such file or directory null [09/19/17 15:58:48] Launch failed – cleaning up connection [09/19/17 15:58:48] [SSH] Connection closed. The same locale error messages can also be reproduced on the slave by running an interactive login from the console or ‘su -’. When running ‘locale -a’ you can also see the en_US.UTF-8 locale is somehow missing. Looking around for this I found the following: <a href="https://github.com/CentOS/sig-cloud-instance-images/issues/71">https://github.com/CentOS/sig-cloud-instance-images/issues/71</a> I tried downgrading glibc back to the version we had before, but that did not seem to resolve the issue. Eventually I managed to resolve it by running ‘localedef -i en_US -f UTF-8 en_US.UTF-8’ on the slave. I've seen this happen on ‘vm0002.workers-phx.ovirt.org’ which is attached to the staging Jenkins, but I've no reason to believe this won't start impacting production slaves. We need to research this further and find out if we need to do something to prevent this issue from surfacing on production slaves. — Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _____________________________________________ Infra mailing list Infra(a)ovirt.org <a href="http://lists.ovirt.org/mailman/listinfo/infra">http://lists.ovirt.org/mailman/listinfo/infra</a> <h3>Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)</h3> <pre> Key: OVIRT-1658 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Barak Korren Assignee: infra</pre> Seems there is an issue with locale files in recent CentOS versions, and we're hitting it in slaves that have been updated recently. The symptoms are that Jenkins disconnects from the slave and then refuses to reconnect to it. The agent log in Jenkins shows: [09/19/17 15:42:56] [SSH] Connection closed. [09/19/17 15:58:48] [SSH] Opening SSH connection to vm0002.workers-phx.ovirt.org:22. [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. [09/19/17 15:58:48] [SSH] Authentication successful. SSH connection reports a garbage before a command execution. Check your .bashrc, .profile, and so on to make sure it is quiet. The received junk text is as follows: /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.utf8): No such file or directory null [09/19/17 15:58:48] Launch failed – cleaning up connection [09/19/17 15:58:48] [SSH] Connection closed. The same locale error messages can also be reproduced on the slave by running an interactive login from the console or ‘su -’. When running ‘locale -a’ you can also see the en_US.UTF-8 locale is somehow missing. Looking around for this I found the following: <a href="https://github.com/CentOS/sig-cloud-instance-images/issues/71">https://github.com/CentOS/sig-cloud-instance-images/issues/71</a> I tried downgrading glibc back to the version we had before, but that did not seem to resolve the issue. Eventually I managed to resolve it by running ‘localedef -i en_US -f UTF-8 en_US.UTF-8’ on the slave. I've seen this happen on ‘vm0002.workers-phx.ovirt.org’ which is attached to the staging Jenkins, but I've no reason to believe this won't start impacting production slaves. We need to research this further and find out if we need to do something to prevent this issue from surfacing on production slaves. — Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted</blockquote> — This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060) <img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYLyz…" alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/> </body></html> ------------=_1505882859-1664-2496--

1 0

Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
by Barak Korren 20 Sep '17

20 Sep '17

Seems there is an issue with locale files in recent CentOS versions, and we're hitting it in slaves that have been updated recently. The symptoms are that Jenkins disconnects from the slave and then refuses to reconnect to it. The agent log in Jenkins shows: [09/19/17 15:42:56] [SSH] Connection closed. [09/19/17 15:58:48] [SSH] Opening SSH connection to vm0002.workers-phx.ovirt.org:22. [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. [09/19/17 15:58:48] [SSH] Authentication successful. SSH connection reports a garbage before a command execution. Check your .bashrc, .profile, and so on to make sure it is quiet. The received junk text is as follows: /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.utf8): No such file or directory null [09/19/17 15:58:48] Launch failed - cleaning up connection [09/19/17 15:58:48] [SSH] Connection closed. The same locale error messages can also be reproduced on the slave by running an interactive login from the console or 'su -'. When running 'locale -a' you can also see the en_US.UTF-8 locale is somehow missing. Looking around for this I found the following: https://github.com/CentOS/sig-cloud-instance-images/issues/71 I tried downgrading glibc back to the version we had before, but that did not seem to resolve the issue. Eventually I managed to resolve it by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave. I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is attached to the staging Jenkins, but I've no reason to believe this won't start impacting production slaves. We need to research this further and find out if we need to do something to prevent this issue from surfacing on production slaves. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

2 1