Subject: [ OST Failure Report ] [ master ] [ 19-09-2017E ] [ 004_basic_sanity ]
by Dafna Ron
This is a multi-part message in MIME format.
--------------38C81434D2056F5745CD1AD2
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Hi,
Lago reports add_console was the failed action:
<testcase classname="004_basic_sanity" name="add_console" time="1.765">...</testcase>
In the engine log I can see that both hosts fail to respond and that we
have also have errors in ansible for invalid services in fedora.
CQ reports a network patch as the probable cause.
See further info below
**
*Link to suspected patches: https://gerrit.ovirt.org/#/c/81921/*
*
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifa...
(Relevant) error snippet from the log:
<error>
*
**Here is the ansible error: ***
*
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'ovirt-vmconsole'})
=> {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg":
"ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception:
INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services
are defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'glusterfs'}) =>
{"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR:
Exception caught: org.fedoraproject.FirewallD1.Exception:
INVALID_SERVICE: 'glusterfs' not among existing services Services are
defined by port/tcp relationship and named as they are in /etc/services
(on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'ctdb'}) =>
{"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service':
u'ovirt-storageconsole'}) => {"failed": true, "item": {"service":
"ovirt-storageconsole"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-storageconsole' not among existing services Services are defined
by port/tcp relationship and named as they are in /etc/services (on most
systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'nrpe'}) =>
{"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'ovirt-vmconsole'})
=> {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg":
"ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception:
INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services
are defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'glusterfs'}) =>
{"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR:
Exception caught: org.fedoraproject.FirewallD1.Exception:
INVALID_SERVICE: 'glusterfs' not among existing services Services are
defined by port/tcp relationship and named as they are in /etc/services
(on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'ctdb'}) =>
{"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service':
u'ovirt-storageconsole'}) => {"failed": true, "item": {"service":
"ovirt-storageconsole"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-storageconsole' not among existing services Services are defined
by port/tcp relationship and named as they are in /etc/services (on most
systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'nrpe'}) =>
{"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most systems)"}
**Here is the engine.log error**
*017-09-19 12:28:59,191-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker)
[] Message received: {"jsonrpc": "2.0", "id":
"634fae71-5f8f-49c1-9d5f-ebc24239cfa9", "result": true}
2017-09-19 12:28:59,191-04 ERROR
[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not
able to update response for "634fae71-5f8f-49c1-9d5f-ebc24239cfa9"
2017-09-19 12:28:59,194-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp
Reactor) [5438f56c] MESSAGE
content-length:80
destination:jms.topic.vdsm_responses
content-type:application/json
subscription:f7138b74-fed3-4eb2-9fdc-957bc4af8f23
{"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b",
"result": true}^@
2017-09-19 12:28:59,194-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker)
[] Message received: {"jsonrpc": "2.0", "id":
"b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true}
2017-09-19 12:28:59,194-04 ERROR
[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not
able to update response for "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b"
2017-09-19 12:28:59,234-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp
Reactor) [5438f56c] MESSAGE
content-length:8534
destination:jms.topic.vdsm_responses
content-type:application/json
subscription:279e9c67-7fc4-4ace-8709-2534f4d53241
*
*</error>*
**
Thanks.
Dafna
--------------38C81434D2056F5745CD1AD2
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi, </p>
<div class="collapsible-content" style="color: rgb(0, 0, 0);
font-family: "Times New Roman"; font-size: medium;
font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: normal; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration-style: initial; text-decoration-color: initial;">
<div class="collapsible" id="collapsible1">
<div class="collapsed hidden">
<div class="line"><span class="html-tag">Lago reports
add_console was the failed action: <br>
<br>
</span><span class="html-tag"><span class="html-tag"><testcase<span
class="html-attribute"><span> </span><span
class="html-attribute-name">classname</span>="<span
class="html-attribute-value">004_basic_sanity</span>"</span><span
class="html-attribute"><span> </span><span
class="html-attribute-name">name</span>="<span
class="html-attribute-value">add_console</span>"</span><span
class="html-attribute"><span> </span><span
class="html-attribute-name">time</span>="<span
class="html-attribute-value">1.765</span>"</span>></span><span
class="text">...</span><span class="html-tag"></testcase><br>
</span> <br>
In the engine log I can see that both hosts fail to
respond and that we have also have errors in ansible for
invalid services in fedora. <br>
<br>
CQ reports a network patch as the probable cause. <br>
<br>
See further info below<br>
<br>
<br class="Apple-interchange-newline">
</span></div>
</div>
</div>
</div>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/81921/">https://gerrit.ovirt.org/#/c/81921/</a></span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/</a></span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifa...">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifa...</a></span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: </span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p>
</b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"><b>Here
is the ansible error: </b></b><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"><br>
</b></p>
<p><span style="font-weight:normal;">host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service':
u'ovirt-vmconsole'}) => {"failed": true, "item": {"service":
"ovirt-vmconsole"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-vmconsole' not among existing services Services are
defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service':
u'glusterfs'}) => {"failed": true, "item": {"service":
"glusterfs"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'glusterfs' not among existing services Services are defined by
port/tcp relationship and named as they are in /etc/services (on
most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'ctdb'})
=> {"failed": true, "item": {"service": "ctdb"}, "msg":
"ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most
systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service':
u'ovirt-storageconsole'}) => {"failed": true, "item":
{"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-storageconsole' not among existing services Services are
defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'nrpe'})
=> {"failed": true, "item": {"service": "nrpe"}, "msg":
"ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most
systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service':
u'ovirt-vmconsole'}) => {"failed": true, "item": {"service":
"ovirt-vmconsole"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-vmconsole' not among existing services Services are
defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service':
u'glusterfs'}) => {"failed": true, "item": {"service":
"glusterfs"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'glusterfs' not among existing services Services are defined by
port/tcp relationship and named as they are in /etc/services (on
most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'ctdb'})
=> {"failed": true, "item": {"service": "ctdb"}, "msg":
"ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most
systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service':
u'ovirt-storageconsole'}) => {"failed": true, "item":
{"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-storageconsole' not among existing services Services are
defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'nrpe'})
=> {"failed": true, "item": {"service": "nrpe"}, "msg":
"ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most
systems)"}<br>
<br>
</span></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"><b>Here
is the engine.log error</b></b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271">017-09-19
12:28:59,191-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker]
(ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id":
"634fae71-5f8f-49c1-9d5f-ebc24239cfa9", "result": true}<br>
2017-09-19 12:28:59,191-04 ERROR
[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker)
[] Not able to update response for
"634fae71-5f8f-49c1-9d5f-ebc24239cfa9"<br>
2017-09-19 12:28:59,194-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL
Stomp Reactor) [5438f56c] MESSAGE<br>
content-length:80<br>
destination:jms.topic.vdsm_responses<br>
content-type:application/json<br>
subscription:f7138b74-fed3-4eb2-9fdc-957bc4af8f23<br>
<br>
{"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b",
"result": true}^@<br>
2017-09-19 12:28:59,194-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker]
(ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id":
"b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true}<br>
2017-09-19 12:28:59,194-04 ERROR
[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker)
[] Not able to update response for
"b4842d9b-38f1-4fd4-a4de-a6b42f25d10b"<br>
2017-09-19 12:28:59,234-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL
Stomp Reactor) [5438f56c] MESSAGE<br>
content-length:8534<br>
destination:jms.topic.vdsm_responses<br>
content-type:application/json<br>
subscription:279e9c67-7fc4-4ace-8709-2534f4d53241<br>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p>
</b></p>
<p><br>
</p>
<p>Thanks. <br>
</p>
<p>Dafna</p>
<p><br class="Apple-interchange-newline">
</p>
</body>
</html>
--------------38C81434D2056F5745CD1AD2--
7 years, 2 months
Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #797
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/changes/13/75913/5:patch --prune
> git rev-parse origin/patch^{commit} # timeout=10
> git rev-parse patch^{commit} # timeout=10
Checking out Revision 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f (patch)
Commit message: "Exclude big packages from mirrors"
> git config core.sparsecheckout # timeout=10
> git checkout -f 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f
> git rev-list 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f # timeout=10
[system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins664955239406877516.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1478, in _commonLoadRepoXML
self._revertOldRepoXML()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1323, in _revertOldRepoXML
os.rename(old_data['old_local'], old_data['local'])
OSError: [Errno 2] No such file or directory
Build step 'Execute shell' marked build as failure
7 years, 2 months
[JIRA] (OVIRT-1658) Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
by sbonazzo (oVirt JIRA)
This is a multi-part message in MIME format...
------------=_1505882859-1664-2496
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1658?page=com.atlassian.jir... ]
sbonazzo commented on OVIRT-1658:
---------------------------------
Please open a bug on bugs.centos.org to let them know about the issue
Il 19 Set 2017 18:10, "Barak Korren" <bkorren(a)redhat.com> ha scritto:
> Seems there is an issue with locale files in recent CentOS versions,
> and we're hitting it in slaves that have been updated recently.
>
> The symptoms are that Jenkins disconnects from the slave and then
> refuses to reconnect to it. The agent log in Jenkins shows:
>
> [09/19/17 15:42:56] [SSH] Connection closed.
>
> [09/19/17 15:58:48] [SSH] Opening SSH connection to
> vm0002.workers-phx.ovirt.org:22.
> [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being
> verified. Man-in-the-middle attacks may be possible against this
> connection.
>
> [09/19/17 15:58:48] [SSH] Authentication successful.
> SSH connection reports a garbage before a command execution.
> Check your .bashrc, .profile, and so on to make sure it is quiet.
> The received junk text is as follows:
> /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot
> change locale (en_US.utf8): No such file or directory
> /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE:
> cannot change locale (en_US.utf8): No such file or directory
> /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES:
> cannot change locale (en_US.utf8): No such file or directory
> /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC:
> cannot change locale (en_US.utf8): No such file or directory
> /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot
> change locale (en_US.utf8): No such file or directory
>
> null
> [09/19/17 15:58:48] Launch failed - cleaning up connection
> [09/19/17 15:58:48] [SSH] Connection closed.
>
>
> The same locale error messages can also be reproduced on the slave by
> running an interactive login from the console or 'su -'. When running
> 'locale -a' you can also see the en_US.UTF-8 locale is somehow
> missing.
>
> Looking around for this I found the following:
> https://github.com/CentOS/sig-cloud-instance-images/issues/71
>
> I tried downgrading glibc back to the version we had before, but that
> did not seem to resolve the issue. Eventually I managed to resolve it
> by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave.
>
> I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is
> attached to the staging Jenkins, but I've no reason to believe this
> won't start impacting production slaves.
>
> We need to research this further and find out if we need to do
> something to prevent this issue from surfacing on production slaves.
>
> --
> Barak Korren
> RHV DevOps team , RHCE, RHCi
> Red Hat EMEA
> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
> Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
> ----------------------------------------------------------------------
>
> Key: OVIRT-1658
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Barak Korren
> Assignee: infra
>
> Seems there is an issue with locale files in recent CentOS versions,
> and we're hitting it in slaves that have been updated recently.
> The symptoms are that Jenkins disconnects from the slave and then
> refuses to reconnect to it. The agent log in Jenkins shows:
> [09/19/17 15:42:56] [SSH] Connection closed.
> [09/19/17 15:58:48] [SSH] Opening SSH connection to
> vm0002.workers-phx.ovirt.org:22.
> [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being
> verified. Man-in-the-middle attacks may be possible against this
> connection.
> [09/19/17 15:58:48] [SSH] Authentication successful.
> SSH connection reports a garbage before a command execution.
> Check your .bashrc, .profile, and so on to make sure it is quiet.
> The received junk text is as follows:
> /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot
> change locale (en_US.utf8): No such file or directory
> /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE:
> cannot change locale (en_US.utf8): No such file or directory
> /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES:
> cannot change locale (en_US.utf8): No such file or directory
> /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC:
> cannot change locale (en_US.utf8): No such file or directory
> /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot
> change locale (en_US.utf8): No such file or directory
> null
> [09/19/17 15:58:48] Launch failed - cleaning up connection
> [09/19/17 15:58:48] [SSH] Connection closed.
> The same locale error messages can also be reproduced on the slave by
> running an interactive login from the console or 'su -'. When running
> 'locale -a' you can also see the en_US.UTF-8 locale is somehow
> missing.
> Looking around for this I found the following:
> https://github.com/CentOS/sig-cloud-instance-images/issues/71
> I tried downgrading glibc back to the version we had before, but that
> did not seem to resolve the issue. Eventually I managed to resolve it
> by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave.
> I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is
> attached to the staging Jenkins, but I've no reason to believe this
> won't start impacting production slaves.
> We need to research this further and find out if we need to do
> something to prevent this issue from surfacing on production slaves.
> --
> Barak Korren
> RHV DevOps team , RHCE, RHCi
> Red Hat EMEA
> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
--
This message was sent by Atlassian {0}
(v1001.0.0-SNAPSHOT#100060)
------------=_1505882859-1664-2496
Content-Type: text/html; charset="UTF-8"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
<html><body>
<pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1658?page=com.atlassian.jir... ]</pre>
<h3>sbonazzo commented on OVIRT-1658:</h3>
<p>Please open a bug on bugs.centos.org to let them know about the issue</p>
<p>Il 19 Set 2017 18:10, “Barak Korren” <bkorren(a)redhat.com> ha scritto:</p>
<blockquote><p>Seems there is an issue with locale files in recent CentOS versions, and we're hitting it in slaves that have been updated recently.</p>
<p>The symptoms are that Jenkins disconnects from the slave and then refuses to reconnect to it. The agent log in Jenkins shows:</p>
<p>[09/19/17 15:42:56] [SSH] Connection closed.</p>
<p>[09/19/17 15:58:48] [SSH] Opening SSH connection to vm0002.workers-phx.ovirt.org:22. [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.</p>
<p>[09/19/17 15:58:48] [SSH] Authentication successful. SSH connection reports a garbage before a command execution. Check your .bashrc, .profile, and so on to make sure it is quiet. The received junk text is as follows: /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.utf8): No such file or directory</p>
<p>null [09/19/17 15:58:48] Launch failed – cleaning up connection [09/19/17 15:58:48] [SSH] Connection closed.</p>
<p>The same locale error messages can also be reproduced on the slave by running an interactive login from the console or ‘su -’. When running ‘locale -a’ you can also see the en_US.UTF-8 locale is somehow missing.</p>
<p>Looking around for this I found the following: <a href="https://github.com/CentOS/sig-cloud-instance-images/issues/71">https://github.com/CentOS/sig-cloud-instance-images/issues/71</a></p>
<p>I tried downgrading glibc back to the version we had before, but that did not seem to resolve the issue. Eventually I managed to resolve it by running ‘localedef -i en_US -f UTF-8 en_US.UTF-8’ on the slave.</p>
<p>I've seen this happen on ‘vm0002.workers-phx.ovirt.org’ which is attached to the staging Jenkins, but I've no reason to believe this won't start impacting production slaves.</p>
<p>We need to research this further and find out if we need to do something to prevent this issue from surfacing on production slaves.</p>
<p>— Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted <em>____________________________________________</em>_ Infra mailing list Infra(a)ovirt.org <a href="http://lists.ovirt.org/mailman/listinfo/infra">http://lists.ovirt.org/mailman/listinfo/infra</a></p>
<h3>Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)</h3>
<pre> Key: OVIRT-1658
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: Barak Korren
Assignee: infra</pre>
<p>Seems there is an issue with locale files in recent CentOS versions, and we're hitting it in slaves that have been updated recently. The symptoms are that Jenkins disconnects from the slave and then refuses to reconnect to it. The agent log in Jenkins shows: [09/19/17 15:42:56] [SSH] Connection closed. [09/19/17 15:58:48] [SSH] Opening SSH connection to vm0002.workers-phx.ovirt.org:22. [09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. [09/19/17 15:58:48] [SSH] Authentication successful. SSH connection reports a garbage before a command execution. Check your .bashrc, .profile, and so on to make sure it is quiet. The received junk text is as follows: /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.utf8): No such file or directory /etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.utf8): No such file or directory null [09/19/17 15:58:48] Launch failed – cleaning up connection [09/19/17 15:58:48] [SSH] Connection closed. The same locale error messages can also be reproduced on the slave by running an interactive login from the console or ‘su -’. When running ‘locale -a’ you can also see the en_US.UTF-8 locale is somehow missing. Looking around for this I found the following: <a href="https://github.com/CentOS/sig-cloud-instance-images/issues/71">https://github.com/CentOS/sig-cloud-instance-images/issues/71</a> I tried downgrading glibc back to the version we had before, but that did not seem to resolve the issue. Eventually I managed to resolve it by running ‘localedef -i en_US -f UTF-8 en_US.UTF-8’ on the slave. I've seen this happen on ‘vm0002.workers-phx.ovirt.org’ which is attached to the staging Jenkins, but I've no reason to believe this won't start impacting production slaves. We need to research this further and find out if we need to do something to prevent this issue from surfacing on production slaves. — Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted</p></blockquote>
<p>— This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060)</p>
<img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYL..." alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/>
</body></html>
------------=_1505882859-1664-2496--
7 years, 2 months
Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
by Barak Korren
Seems there is an issue with locale files in recent CentOS versions,
and we're hitting it in slaves that have been updated recently.
The symptoms are that Jenkins disconnects from the slave and then
refuses to reconnect to it. The agent log in Jenkins shows:
[09/19/17 15:42:56] [SSH] Connection closed.
[09/19/17 15:58:48] [SSH] Opening SSH connection to
vm0002.workers-phx.ovirt.org:22.
[09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being
verified. Man-in-the-middle attacks may be possible against this
connection.
[09/19/17 15:58:48] [SSH] Authentication successful.
SSH connection reports a garbage before a command execution.
Check your .bashrc, .profile, and so on to make sure it is quiet.
The received junk text is as follows:
/etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot
change locale (en_US.utf8): No such file or directory
/etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE:
cannot change locale (en_US.utf8): No such file or directory
/etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES:
cannot change locale (en_US.utf8): No such file or directory
/etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC:
cannot change locale (en_US.utf8): No such file or directory
/etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot
change locale (en_US.utf8): No such file or directory
null
[09/19/17 15:58:48] Launch failed - cleaning up connection
[09/19/17 15:58:48] [SSH] Connection closed.
The same locale error messages can also be reproduced on the slave by
running an interactive login from the console or 'su -'. When running
'locale -a' you can also see the en_US.UTF-8 locale is somehow
missing.
Looking around for this I found the following:
https://github.com/CentOS/sig-cloud-instance-images/issues/71
I tried downgrading glibc back to the version we had before, but that
did not seem to resolve the issue. Eventually I managed to resolve it
by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave.
I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is
attached to the staging Jenkins, but I've no reason to believe this
won't start impacting production slaves.
We need to research this further and find out if we need to do
something to prevent this issue from surfacing on production slaves.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
7 years, 2 months
oVirt infra daily report - unstable production jobs - 449
by jenkins＠jenkins.phx.ovirt.org
------=_Part_622_688795696.1505862006356
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Good morning!
Attached is the HTML page with the jenkins status report. You can see it also here:
- http://jenkins.ovirt.org/job/system_jenkins-report/449//artifact/exported...
Cheers,
Jenkins
------=_Part_622_688795696.1505862006356
Content-Type: text/html; charset=us-ascii; name=upstream_report.html
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=upstream_report.html
Content-ID: <upstream_report.html>
<!DOCTYPE html><head><style type="text/css">
table.gridtable {
border-collapse: collapse;
table-layout:fixed;
width:1600px;
font-family: monospace;
font-size:13px;
}
.head {
font-size:20px;
font-family: arial;
}
.sub {
font-size:18px;
background-color:#e5e5e5;
font-family: arial;
}
pre {
font-family: monospace;
display: inline;
white-space: pre-wrap;
white-space: -moz-pre-wrap !important;
white-space: -pre-wrap;
white-space: -o-pre-wrap;
word-wrap: break-word;
}
</style>
</head>
<body>
<table class="gridtable" border=2>
<tr><th colspan=2 class=head>
RHEVM CI Jenkins Daily Report - 19/09/2017
</th></tr><tr><th colspan=2 class=sub>
<font color="blue"><a href="http://jenkins.ovirt.org/">00 Unstable Critical</a></font>
</th></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/">system-sync_mirrors-centos-updates-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
------=_Part_622_688795696.1505862006356--
7 years, 2 months
[JIRA] (OVIRT-1658) Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
by Barak Korren (oVirt JIRA)
Barak Korren created OVIRT-1658:
-----------------------------------
Summary: Potential issue coming to CentOS7 slaves (glibc, missing UTF-8 locale)
Key: OVIRT-1658
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1658
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: Barak Korren
Assignee: infra
Seems there is an issue with locale files in recent CentOS versions,
and we're hitting it in slaves that have been updated recently.
The symptoms are that Jenkins disconnects from the slave and then
refuses to reconnect to it. The agent log in Jenkins shows:
[09/19/17 15:42:56] [SSH] Connection closed.
[09/19/17 15:58:48] [SSH] Opening SSH connection to
vm0002.workers-phx.ovirt.org:22.
[09/19/17 15:58:48] [SSH] WARNING: SSH Host Keys are not being
verified. Man-in-the-middle attacks may be possible against this
connection.
[09/19/17 15:58:48] [SSH] Authentication successful.
SSH connection reports a garbage before a command execution.
Check your .bashrc, .profile, and so on to make sure it is quiet.
The received junk text is as follows:
/etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot
change locale (en_US.utf8): No such file or directory
/etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE:
cannot change locale (en_US.utf8): No such file or directory
/etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES:
cannot change locale (en_US.utf8): No such file or directory
/etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC:
cannot change locale (en_US.utf8): No such file or directory
/etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot
change locale (en_US.utf8): No such file or directory
null
[09/19/17 15:58:48] Launch failed - cleaning up connection
[09/19/17 15:58:48] [SSH] Connection closed.
The same locale error messages can also be reproduced on the slave by
running an interactive login from the console or 'su -'. When running
'locale -a' you can also see the en_US.UTF-8 locale is somehow
missing.
Looking around for this I found the following:
https://github.com/CentOS/sig-cloud-instance-images/issues/71
I tried downgrading glibc back to the version we had before, but that
did not seem to resolve the issue. Eventually I managed to resolve it
by running 'localedef -i en_US -f UTF-8 en_US.UTF-8' on the slave.
I've seen this happen on 'vm0002.workers-phx.ovirt.org' which is
attached to the staging Jenkins, but I've no reason to believe this
won't start impacting production slaves.
We need to research this further and find out if we need to do
something to prevent this issue from surfacing on production slaves.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
--
This message was sent by Atlassian {0}
(v1001.0.0-SNAPSHOT#100060)
7 years, 2 months
[JIRA] (OVIRT-1657) Unity CI whitelist mechanism for GitHub and Gerrit
by Barak Korren (oVirt JIRA)
This is a multi-part message in MIME format...
------------=_1505835477-31751-486
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1657?page=com.atlassian.jir... ]
Barak Korren reassigned OVIRT-1657:
-----------------------------------
Assignee: Barak Korren (was: infra)
> Unity CI whitelist mechanism for GitHub and Gerrit
> --------------------------------------------------
>
> Key: OVIRT-1657
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1657
> Project: oVirt - virtualization made easy
> Issue Type: Improvement
> Components: oVirt CI
> Reporter: Barak Korren
> Assignee: Barak Korren
>
> Right now we have two separate CI mechanisms:
> # A mechanism based on the '{{jenkins-whitelist}}' repository that is used by CI jobs for Gerrit projects
> # A mechanism that is part of the 'GitHub pull request builder' plugin that is used for GitHub based projects.
> The '{{jenkins-whitelist}}' has a few shortcomings:
> * Its impossible for maintainers to manually override it
> * Maintainers hate to send patches to an unrelated repo
> * Its impossible to have a per-project whitelist
> The GitHub style whitelist has a few shortcomings as well:
> * Whitelist members that were added via comments on PRs are stored in the job configuration and can vanish if jobs are updated via JJB
> * The GHPRB plugin forces us to create jobs in a very specific way (One job per project)
> * The whitelist can prevent GitHub events without specific people behind them from triggering the jobs - For example jobs are not triggered when commits are added to an existing PR because such events are sent by the 'null' user whihc typically is not in the whitelist... (This is probably a bug with the plugin).
--
This message was sent by Atlassian {0}
(v1001.0.0-SNAPSHOT#100060)
------------=_1505835477-31751-486
Content-Type: text/html; charset="UTF-8"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
<html><body>
<pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1657?page=com.atlassian.jir... ]</pre>
<h3>Barak Korren reassigned OVIRT-1657:</h3>
<pre>Assignee: Barak Korren (was: infra)</pre>
<blockquote><h3>Unity CI whitelist mechanism for GitHub and Gerrit</h3>
<pre> Key: OVIRT-1657
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1657
Project: oVirt - virtualization made easy
Issue Type: Improvement
Components: oVirt CI
Reporter: Barak Korren
Assignee: Barak Korren</pre>
<p>Right now we have two separate CI mechanisms: # A mechanism based on the ‘{{jenkins-whitelist}}’ repository that is used by CI jobs for Gerrit projects # A mechanism that is part of the ‘GitHub pull request builder’ plugin that is used for GitHub based projects. The ‘{{jenkins-whitelist}}’ has a few shortcomings:</p>
<ul><li><p>Its impossible for maintainers to manually override it</p></li>
<li><p>Maintainers hate to send patches to an unrelated repo</p></li>
<li><p>Its impossible to have a per-project whitelist</p></li></ul>
<p>The GitHub style whitelist has a few shortcomings as well:</p>
<ul><li><p>Whitelist members that were added via comments on PRs are stored in the job configuration and can vanish if jobs are updated via JJB</p></li>
<li><p>The GHPRB plugin forces us to create jobs in a very specific way (One job per project)</p></li>
<li><p>The whitelist can prevent GitHub events without specific people behind them from triggering the jobs – For example jobs are not triggered when commits are added to an existing PR because such events are sent by the ‘null’ user whihc typically is not in the whitelist… (This is probably a bug with the plugin).</p></li></ul></blockquote>
<p>— This message was sent by Atlassian {0} (v1001.0.0-SNAPSHOT#100060)</p>
<img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYL..." alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/>
</body></html>
------------=_1505835477-31751-486--
7 years, 2 months
