September 2017 - Infra - oVirt List Archives

Subject: [ OST Failure Report ] [ master ] [ 19-09-2017E ] [ 004_basic_sanity ]
by Dafna Ron 20 Sep '17

20 Sep '17

This is a multi-part message in MIME format. --------------38C81434D2056F5745CD1AD2 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, Lago reports add_console was the failed action: <testcase classname="004_basic_sanity" name="add_console" time="1.765">...</testcase> In the engine log I can see that both hosts fail to respond and that we have also have errors in ansible for invalid services in fedora. CQ reports a network patch as the probable cause. See further info below ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/81921/* * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifact… (Relevant) error snippet from the log: <error> * **Here is the ansible error: *** * host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ovirt-vmconsole'}) => {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'glusterfs'}) => {"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ctdb'}) => {"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ovirt-storageconsole'}) => {"failed": true, "item": {"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-storageconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'nrpe'}) => {"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ovirt-vmconsole'}) => {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'glusterfs'}) => {"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ctdb'}) => {"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ovirt-storageconsole'}) => {"failed": true, "item": {"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-storageconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'nrpe'}) => {"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} **Here is the engine.log error** *017-09-19 12:28:59,191-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "634fae71-5f8f-49c1-9d5f-ebc24239cfa9", "result": true} 2017-09-19 12:28:59,191-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "634fae71-5f8f-49c1-9d5f-ebc24239cfa9" 2017-09-19 12:28:59,194-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [5438f56c] MESSAGE content-length:80 destination:jms.topic.vdsm_responses content-type:application/json subscription:f7138b74-fed3-4eb2-9fdc-957bc4af8f23 {"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true}^@ 2017-09-19 12:28:59,194-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true} 2017-09-19 12:28:59,194-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b" 2017-09-19 12:28:59,234-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [5438f56c] MESSAGE content-length:8534 destination:jms.topic.vdsm_responses content-type:application/json subscription:279e9c67-7fc4-4ace-8709-2534f4d53241 * *</error>* ** Thanks. Dafna --------------38C81434D2056F5745CD1AD2 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, <div class="collapsible-content" style="color: rgb(0, 0, 0); font-family: "Times New Roman"; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"> <div class="collapsible" id="collapsible1"> <div class="collapsed hidden"> <div class="line">Lago reports add_console was the failed action: <testcase classname="004_basic_sanity" name="add_console" time="1.765">...</testcase> In the engine log I can see that both hosts fail to respond and that we have also have errors in ansible for invalid services in fedora. CQ reports a network patch as the probable cause. See further info below </div> </div> </div> </div> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/81921/">https://gerrit.ovirt.org/#/c/81921/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifact…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifact…</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> Here is the ansible error: host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ovirt-vmconsole'}) => {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'glusterfs'}) => {"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ctdb'}) => {"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ovirt-storageconsole'}) => {"failed": true, "item": {"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-storageconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'nrpe'}) => {"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ovirt-vmconsole'}) => {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'glusterfs'}) => {"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ctdb'}) => {"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ovirt-storageconsole'}) => {"failed": true, "item": {"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-storageconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'nrpe'}) => {"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} Here is the engine.log error 017-09-19 12:28:59,191-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "634fae71-5f8f-49c1-9d5f-ebc24239cfa9", "result": true} 2017-09-19 12:28:59,191-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "634fae71-5f8f-49c1-9d5f-ebc24239cfa9" 2017-09-19 12:28:59,194-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [5438f56c] MESSAGE content-length:80 destination:jms.topic.vdsm_responses content-type:application/json subscription:f7138b74-fed3-4eb2-9fdc-957bc4af8f23 {"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true}^@ 2017-09-19 12:28:59,194-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true} 2017-09-19 12:28:59,194-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b" 2017-09-19 12:28:59,234-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [5438f56c] MESSAGE content-length:8534 destination:jms.topic.vdsm_responses content-type:application/json subscription:279e9c67-7fc4-4ace-8709-2534f4d53241 <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> Thanks. Dafna </body> </html> --------------38C81434D2056F5745CD1AD2--

1 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #797
by jenkins＠jenkins.phx.ovirt.org 20 Sep '17

20 Sep '17

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/changes/13/75913/5:patch --prune > git rev-parse origin/patch^{commit} # timeout=10 > git rev-parse patch^{commit} # timeout=10 Checking out Revision 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f (patch) Commit message: "Exclude big packages from mirrors" > git config core.sparsecheckout # timeout=10 > git checkout -f 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f > git rev-list 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins664955239406877516.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1478, in _commonLoadRepoXML self._revertOldRepoXML() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1323, in _revertOldRepoXML os.rename(old_data['old_local'], old_data['local']) OSError: [Errno 2] No such file or directory Build step 'Execute shell' marked build as failure

1 2

[CQ]: 81792, 4 (ovirt-hosted-engine-ha) failed "ovirt-master" system tests
by oVirt Jenkins 20 Sep '17

20 Sep '17

Change 81792,4 (ovirt-hosted-engine-ha) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/81792/4 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2732/

1 0

[oVirt Jenkins] ovirt-node-ng_ovirt-4.1-snapshot_build-artifacts-el7-x86_64 - Build # 259 - Failure!
by jenkins＠jenkins.phx.ovirt.org 20 Sep '17

20 Sep '17

------=_Part_620_18096676.1505841561014 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.1-snapshot_build-artifac… Build: http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.1-snapshot_build-artifac… Build Number: 259 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #259 [Yuval Turgeman] automation: adding loop devices ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_620_18096676.1505841561014--

1 1

[JIRA] (OVIRT-1658) Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
by sbonazzo (oVirt JIRA) 20 Sep '17

20 Sep '17

This is a multi-part message in MIME format... ------------=_1505882859-1664-2496 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit [ https://ovirt-jira.atlassian.net/browse/OVIRT-1658?page=com.atlassian.jira.… ] sbonazzo commented on OVIRT-1658: --------------------------------- Please open a bug on bugs.centos.org to let them know about the issue Il 19 Set 2017 18:10, "Barak Korren" <bkorren(a)redhat.com> ha scritto: > Seems there is an issue with locale files in recent CentOS versions, > and we're hitting it in slaves that have been updated recently. > > The symptoms are that Jenkins disconnects from the slave and then > refuses to reconnect to it. The agent log in Jenkins shows: > > [09/19/17 15:42:56] [SSH] Connection closed. > > [09/19/17 15:58:48] [SSH] Opening SSH connection to > vm0002.workers-phx.ovirt.org:22. > [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being > verified. Man-in-the-middle attacks may be possible against this > connection. > > [09/19/17 15:58:48] [SSH] Authentication successful. > SSH connection reports a garbage before a command execution. > Check your .bashrc, .profile, and so on to make sure it is quiet. > The received junk text is as follows: > /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot > change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot > change locale (en_US.utf8): No such file or directory > > null > [09/19/17 15:58:48] Launch failed - cleaning up connection > [09/19/17 15:58:48] [SSH] Connection closed. > > > The same locale error messages can also be reproduced on the slave by > running an interactive login from the console or 'su -'. When running > 'locale -a' you can also see the en_US.UTF-8 locale is somehow > missing. > > Looking around for this I found the following: > https://github.com/CentOS/sig-cloud-instance-images/issues/71 > > I tried downgrading glibc back to the version we had before, but that > did not seem to resolve the issue. Eventually I managed to resolve it > by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave. > > I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is > attached to the staging Jenkins, but I've no reason to believe this > won't start impacting production slaves. > > We need to research this further and find out if we need to do > something to prevent this issue from surfacing on production slaves. > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > > Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale) > ---------------------------------------------------------------------- > > Key: OVIRT-1658 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Barak Korren > Assignee: infra > > Seems there is an issue with locale files in recent CentOS versions, > and we're hitting it in slaves that have been updated recently. > The symptoms are that Jenkins disconnects from the slave and then > refuses to reconnect to it. The agent log in Jenkins shows: > [09/19/17 15:42:56] [SSH] Connection closed. > [09/19/17 15:58:48] [SSH] Opening SSH connection to > vm0002.workers-phx.ovirt.org:22. > [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being > verified. Man-in-the-middle attacks may be possible against this > connection. > [09/19/17 15:58:48] [SSH] Authentication successful. > SSH connection reports a garbage before a command execution. > Check your .bashrc, .profile, and so on to make sure it is quiet. > The received junk text is as follows: > /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot > change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: > cannot change locale (en_US.utf8): No such file or directory > /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot > change locale (en_US.utf8): No such file or directory > null > [09/19/17 15:58:48] Launch failed - cleaning up connection > [09/19/17 15:58:48] [SSH] Connection closed. > The same locale error messages can also be reproduced on the slave by > running an interactive login from the console or 'su -'. When running > 'locale -a' you can also see the en_US.UTF-8 locale is somehow > missing. > Looking around for this I found the following: > https://github.com/CentOS/sig-cloud-instance-images/issues/71 > I tried downgrading glibc back to the version we had before, but that > did not seem to resolve the issue. Eventually I managed to resolve it > by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave. > I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is > attached to the staging Jenkins, but I've no reason to believe this > won't start impacting production slaves. > We need to research this further and find out if we need to do > something to prevent this issue from surfacing on production slaves. > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted -- This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060) ------------=_1505882859-1664-2496 Content-Type: text/html; charset="UTF-8" Content-Disposition: inline Content-Transfer-Encoding: 7bit <html><body> <pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1658?page=com.atlassian.jira.… ]</pre> <h3>sbonazzo commented on OVIRT-1658:</h3> Please open a bug on bugs.centos.org to let them know about the issue Il 19 Set 2017 18:10, “Barak Korren” <bkorren(a)redhat.com> ha scritto: <blockquote>Seems there is an issue with locale files in recent CentOS versions, and we're hitting it in slaves that have been updated recently. The symptoms are that Jenkins disconnects from the slave and then refuses to reconnect to it. The agent log in Jenkins shows: [09/19/17 15:42:56] [SSH] Connection closed. [09/19/17 15:58:48] [SSH] Opening SSH connection to vm0002.workers-phx.ovirt.org:22. [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. [09/19/17 15:58:48] [SSH] Authentication successful. SSH connection reports a garbage before a command execution. Check your .bashrc, .profile, and so on to make sure it is quiet. The received junk text is as follows: /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.utf8): No such file or directory null [09/19/17 15:58:48] Launch failed – cleaning up connection [09/19/17 15:58:48] [SSH] Connection closed. The same locale error messages can also be reproduced on the slave by running an interactive login from the console or ‘su -’. When running ‘locale -a’ you can also see the en_US.UTF-8 locale is somehow missing. Looking around for this I found the following: <a href="https://github.com/CentOS/sig-cloud-instance-images/issues/71">https://github.com/CentOS/sig-cloud-instance-images/issues/71</a> I tried downgrading glibc back to the version we had before, but that did not seem to resolve the issue. Eventually I managed to resolve it by running ‘localedef -i en_US -f UTF-8 en_US.UTF-8’ on the slave. I've seen this happen on ‘vm0002.workers-phx.ovirt.org’ which is attached to the staging Jenkins, but I've no reason to believe this won't start impacting production slaves. We need to research this further and find out if we need to do something to prevent this issue from surfacing on production slaves. — Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _____________________________________________ Infra mailing list Infra(a)ovirt.org <a href="http://lists.ovirt.org/mailman/listinfo/infra">http://lists.ovirt.org/mailman/listinfo/infra</a> <h3>Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)</h3> <pre> Key: OVIRT-1658 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Barak Korren Assignee: infra</pre> Seems there is an issue with locale files in recent CentOS versions, and we're hitting it in slaves that have been updated recently. The symptoms are that Jenkins disconnects from the slave and then refuses to reconnect to it. The agent log in Jenkins shows: [09/19/17 15:42:56] [SSH] Connection closed. [09/19/17 15:58:48] [SSH] Opening SSH connection to vm0002.workers-phx.ovirt.org:22. [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. [09/19/17 15:58:48] [SSH] Authentication successful. SSH connection reports a garbage before a command execution. Check your .bashrc, .profile, and so on to make sure it is quiet. The received junk text is as follows: /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.utf8): No such file or directory null [09/19/17 15:58:48] Launch failed – cleaning up connection [09/19/17 15:58:48] [SSH] Connection closed. The same locale error messages can also be reproduced on the slave by running an interactive login from the console or ‘su -’. When running ‘locale -a’ you can also see the en_US.UTF-8 locale is somehow missing. Looking around for this I found the following: <a href="https://github.com/CentOS/sig-cloud-instance-images/issues/71">https://github.com/CentOS/sig-cloud-instance-images/issues/71</a> I tried downgrading glibc back to the version we had before, but that did not seem to resolve the issue. Eventually I managed to resolve it by running ‘localedef -i en_US -f UTF-8 en_US.UTF-8’ on the slave. I've seen this happen on ‘vm0002.workers-phx.ovirt.org’ which is attached to the staging Jenkins, but I've no reason to believe this won't start impacting production slaves. We need to research this further and find out if we need to do something to prevent this issue from surfacing on production slaves. — Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted</blockquote> — This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060) <img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYLyz…" alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/> </body></html> ------------=_1505882859-1664-2496--

1 0

Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
by Barak Korren 20 Sep '17

20 Sep '17

Seems there is an issue with locale files in recent CentOS versions, and we're hitting it in slaves that have been updated recently. The symptoms are that Jenkins disconnects from the slave and then refuses to reconnect to it. The agent log in Jenkins shows: [09/19/17 15:42:56] [SSH] Connection closed. [09/19/17 15:58:48] [SSH] Opening SSH connection to vm0002.workers-phx.ovirt.org:22. [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. [09/19/17 15:58:48] [SSH] Authentication successful. SSH connection reports a garbage before a command execution. Check your .bashrc, .profile, and so on to make sure it is quiet. The received junk text is as follows: /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.utf8): No such file or directory null [09/19/17 15:58:48] Launch failed - cleaning up connection [09/19/17 15:58:48] [SSH] Connection closed. The same locale error messages can also be reproduced on the slave by running an interactive login from the console or 'su -'. When running 'locale -a' you can also see the en_US.UTF-8 locale is somehow missing. Looking around for this I found the following: https://github.com/CentOS/sig-cloud-instance-images/issues/71 I tried downgrading glibc back to the version we had before, but that did not seem to resolve the issue. Eventually I managed to resolve it by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave. I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is attached to the staging Jenkins, but I've no reason to believe this won't start impacting production slaves. We need to research this further and find out if we need to do something to prevent this issue from surfacing on production slaves. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

2 1

oVirt infra daily report - unstable production jobs - 449
by jenkins＠jenkins.phx.ovirt.org 19 Sep '17

19 Sep '17

------=_Part_622_688795696.1505862006356 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/449//artifact/exported-a… Cheers, Jenkins ------=_Part_622_688795696.1505862006356 Content-Type: text/html; charset=us-ascii; name=upstream_report.html Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=upstream_report.html Content-ID: <upstream_report.html> <!DOCTYPE html><head><style type="text/css"> table.gridtable { border-collapse: collapse; table-layout:fixed; width:1600px; font-family: monospace; font-size:13px; } .head { font-size:20px; font-family: arial; } .sub { font-size:18px; background-color:#e5e5e5; font-family: arial; } pre { font-family: monospace; display: inline; white-space: pre-wrap; white-space: -moz-pre-wrap !important; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word; } </style> </head> <body> <table class="gridtable" border=2> <tr><th colspan=2 class=head> RHEVM CI Jenkins Daily Report - 19/09/2017 </th></tr><tr><th colspan=2 class=sub> <a href="http://jenkins.ovirt.org/">00 Unstable Critical</a> </th></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/">system-sync_mirrors-centos-updates-el7-x86_64</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> ------=_Part_622_688795696.1505862006356--

1 0

[CQ]: 81921,3 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 19 Sep '17

19 Sep '17

Change 81921,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/81921/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/

1 0

[JIRA] (OVIRT-1658) Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
by Barak Korren (oVirt JIRA) 19 Sep '17

19 Sep '17

Barak Korren created OVIRT-1658: ----------------------------------- Summary: Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale) Key: OVIRT-1658 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Barak Korren Assignee: infra Seems there is an issue with locale files in recent CentOS versions, and we're hitting it in slaves that have been updated recently. The symptoms are that Jenkins disconnects from the slave and then refuses to reconnect to it. The agent log in Jenkins shows: [09/19/17 15:42:56] [SSH] Connection closed. [09/19/17 15:58:48] [SSH] Opening SSH connection to vm0002.workers-phx.ovirt.org:22. [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. [09/19/17 15:58:48] [SSH] Authentication successful. SSH connection reports a garbage before a command execution. Check your .bashrc, .profile, and so on to make sure it is quiet. The received junk text is as follows: /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.utf8): No such file or directory null [09/19/17 15:58:48] Launch failed - cleaning up connection [09/19/17 15:58:48] [SSH] Connection closed. The same locale error messages can also be reproduced on the slave by running an interactive login from the console or 'su -'. When running 'locale -a' you can also see the en_US.UTF-8 locale is somehow missing. Looking around for this I found the following: https://github.com/CentOS/sig-cloud-instance-images/issues/71 I tried downgrading glibc back to the version we had before, but that did not seem to resolve the issue. Eventually I managed to resolve it by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave. I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is attached to the staging Jenkins, but I've no reason to believe this won't start impacting production slaves. We need to research this further and find out if we need to do something to prevent this issue from surfacing on production slaves. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted -- This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060)

1 0

[JIRA] (OVIRT-1657) Unity CI whitelist mechanism for GitHub and Gerrit
by Barak Korren (oVirt JIRA) 19 Sep '17

19 Sep '17

This is a multi-part message in MIME format... ------------=_1505835477-31751-486 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit [ https://ovirt-jira.atlassian.net/browse/OVIRT-1657?page=com.atlassian.jira.… ] Barak Korren reassigned OVIRT-1657: ----------------------------------- Assignee: Barak Korren (was: infra) > Unity CI whitelist mechanism for GitHub and Gerrit > -------------------------------------------------- > > Key: OVIRT-1657 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1657 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: oVirt CI > Reporter: Barak Korren > Assignee: Barak Korren > > Right now we have two separate CI mechanisms: > # A mechanism based on the '{{jenkins-whitelist}}' repository that is used by CI jobs for Gerrit projects > # A mechanism that is part of the 'GitHub pull request builder' plugin that is used for GitHub based projects. > The '{{jenkins-whitelist}}' has a few shortcomings: > * Its impossible for maintainers to manually override it > * Maintainers hate to send patches to an unrelated repo > * Its impossible to have a per-project whitelist > The GitHub style whitelist has a few shortcomings as well: > * Whitelist members that were added via comments on PRs are stored in the job configuration and can vanish if jobs are updated via JJB > * The GHPRB plugin forces us to create jobs in a very specific way (One job per project) > * The whitelist can prevent GitHub events without specific people behind them from triggering the jobs - For example jobs are not triggered when commits are added to an existing PR because such events are sent by the 'null' user whihc typically is not in the whitelist... (This is probably a bug with the plugin). -- This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060) ------------=_1505835477-31751-486 Content-Type: text/html; charset="UTF-8" Content-Disposition: inline Content-Transfer-Encoding: 7bit <html><body> <pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1657?page=com.atlassian.jira.… ]</pre> <h3>Barak Korren reassigned OVIRT-1657:</h3> <pre>Assignee: Barak Korren (was: infra)</pre> <blockquote><h3>Unity CI whitelist mechanism for GitHub and Gerrit</h3> <pre> Key: OVIRT-1657 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1657 Project: oVirt - virtualization made easy Issue Type: Improvement Components: oVirt CI Reporter: Barak Korren Assignee: Barak Korren</pre> Right now we have two separate CI mechanisms: # A mechanism based on the ‘{{jenkins-whitelist}}’ repository that is used by CI jobs for Gerrit projects # A mechanism that is part of the ‘GitHub pull request builder’ plugin that is used for GitHub based projects. The ‘{{jenkins-whitelist}}’ has a few shortcomings: <ul><li>Its impossible for maintainers to manually override it</li> <li>Maintainers hate to send patches to an unrelated repo</li> <li>Its impossible to have a per-project whitelist</li></ul> The GitHub style whitelist has a few shortcomings as well: <ul><li>Whitelist members that were added via comments on PRs are stored in the job configuration and can vanish if jobs are updated via JJB</li> <li>The GHPRB plugin forces us to create jobs in a very specific way (One job per project)</li> <li>The whitelist can prevent GitHub events without specific people behind them from triggering the jobs – For example jobs are not triggered when commits are added to an existing PR because such events are sent by the ‘null’ user whihc typically is not in the whitelist… (This is probably a bug with the plugin).</li></ul></blockquote> — This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060) <img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYLyz…" alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/> </body></html> ------------=_1505835477-31751-486--

1 0