<html><body>
<h3>Barak Korren created OVIRT-1658:</h3>
<pre> Summary: Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
Key: OVIRT-1658
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: Barak Korren
Assignee: infra</pre>
<p>Seems there is an issue with locale files in recent CentOS versions, and we're hitting it in slaves that have been updated recently.</p>
<p>The symptoms are that Jenkins disconnects from the slave and then refuses to reconnect to it. The agent log in Jenkins shows:</p>
<p>[09/19/17 15:42:56] [SSH] Connection closed.</p>
<p>[09/19/17 15:58:48] [SSH] Opening SSH connection to vm0002.workers-phx.ovirt.org:22. [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.</p>
<p>[09/19/17 15:58:48] [SSH] Authentication successful. SSH connection reports a garbage before a command execution. Check your .bashrc, .profile, and so on to make sure it is quiet. The received junk text is as follows: /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.utf8): No such file or directory</p>
<p>null [09/19/17 15:58:48] Launch failed – cleaning up connection [09/19/17 15:58:48] [SSH] Connection closed.</p>
<p>The same locale error messages can also be reproduced on the slave by running an interactive login from the console or ‘su -’. When running ‘locale -a’ you can also see the en_US.UTF-8 locale is somehow missing.</p>
<p>Looking around for this I found the following: <a href="https://github.com/CentOS/sig-cloud-instance-images/issues/71">https://github.com/CentOS/sig-cloud-instance-images/issues/71</a></p>
<p>I tried downgrading glibc back to the version we had before, but that did not seem to resolve the issue. Eventually I managed to resolve it by running ‘localedef -i en_US -f UTF-8 en_US.UTF-8’ on the slave.</p>
<p>I've seen this happen on ‘vm0002.workers-phx.ovirt.org’ which is attached to the staging Jenkins, but I've no reason to believe this won't start impacting production slaves.</p>
<p>We need to research this further and find out if we need to do something to prevent this issue from surfacing on production slaves.</p>
<p>— Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted</p>
<p>— This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060)</p>
<img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYLyzYS3r1Q939T2sgz0-2FFacMlvTDbko5e9z30X7BuCeuaT5y2WdX3AfO2EmgvP6tdttkM0nFQLGKZCMxVGOsHScYpRfIUPjvwrT55Crxr2ugK0m6Hwb77fa0H9QMsg-2B-2FNnV9OYRQ9cUS2KT60gR-2FAuWIPFZ86dgoeAtqsoCaydDWQrzTlWvMK5ZubICtX30uf75QG7qZZP0G61S4fHE-3D" alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/>
</body></html>