[ OST Failure Report ] [ oVirt master ] [ 28-09-2017 ] [ 004_basic_sanity.disk_operations ]
by Dafna Ron
This is a multi-part message in MIME format.
--------------D32DAC9929FA84F802C5B88B
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Hi,
We had a failure in OST for basic disk operation.
from what I can see in the log, there is an issue deleting a snapshot
image and the issue is related to the live storage migration.
However, a second pair of eyes would help since I can see engine
reporting the error on failed delete while vdsm was not actually
reporting a failed delete, at least not as ERROR.
Also I don't think the patch that is reported as the cause was actually
the cause.
I greped the image from the log: http://pastebin.test.redhat.com/520150
*
Link to suspected patches:
*
https://gerrit.ovirt.org/#/c/82293/2
*
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/
(Relevant) error snippet from the log:
<error>
2017-09-27 16:51:02,760-04 ERROR
[org.ovirt.engine.core.bll.DestroyImageCheckCommand]
(EE-ManagedThreadFactory-commandCoordinator-Thread-6)
[8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not
removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe]
</error>
*
--------------D32DAC9929FA84F802C5B88B
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi, <br>
</p>
<p>We had a failure in OST for basic disk operation. <br>
</p>
<p>from what I can see in the log, there is an issue deleting a
snapshot image and the issue is related to the live storage
migration. <br>
</p>
<p>However, a second pair of eyes would help since I can see engine
reporting the error on failed delete while vdsm was not actually
reporting a failed delete, at least not as ERROR. <br>
</p>
<p>Also I don't think the patch that is reported as the cause was
actually the cause. <br>
</p>
<p>I greped the image from the log:
<a class="moz-txt-link-freetext" href="http://pastebin.test.redhat.com/520150">http://pastebin.test.redhat.com/520150</a><br>
</p>
<pre wrap="">
<b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: </span></p></b><pre wrap=""><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82293/2">https://gerrit.ovirt.org/#/c/82293/2</a></pre><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792">
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906</a></span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/</a></span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p>
2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d24d-33b9-43f0-
9ab4-15f0334e2ebe]
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p>
</b></pre>
</body>
</html>
--------------D32DAC9929FA84F802C5B88B--
7 years, 1 month
Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #829
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/changes/13/75913/5:patch --prune
> git rev-parse origin/patch^{commit} # timeout=10
> git rev-parse patch^{commit} # timeout=10
Checking out Revision 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f (patch)
Commit message: "Exclude big packages from mirrors"
> git config core.sparsecheckout # timeout=10
> git checkout -f 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f
> git rev-list 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f # timeout=10
[system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins5464824030326316176.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1478, in _commonLoadRepoXML
self._revertOldRepoXML()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1323, in _revertOldRepoXML
os.rename(old_data['old_local'], old_data['local'])
OSError: [Errno 2] No such file or directory
Build step 'Execute shell' marked build as failure
7 years, 1 month
[ OST Failure Report ] [ oVirt master ] [ 28-09-2017 ] [ 003_00_metrics_bootstrap.configure_metrics ]
by Dafna Ron
This is a multi-part message in MIME format.
--------------CCBA598F31714CCB8129AF90
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Hi,
We had failure in OST last night which seems to be related to two
fluentd configuration patchs.
I found that the error is actually in the deployment and can be found in
the lago log.
I am not sure it's related but I am also seeing unexpected errors in the
vdsm log for getDeviceList: http://pastebin.test.redhat.com/520514
We apparently calling getDeviceList + checkStatus=true with no guid.
**
*Link to suspected patches: *
*
**https://gerrit.ovirt.org/#/c/82329/ * *
**
*https://gerrit.ovirt.org/#/c/81551/*
Link to Job:
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2932/*
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2933/*
Link to all logs:
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2932/artifact/**
**http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2933/artifact/**
***
*
*(Relevant) error snippet from the log: *
*
<error>
TASK [ovirt_fluentd/client : Install non-ssl client]
***************************
fatal: [lago-basic-suite-master-host-1]: FAILED! => {"changed": false,
"failed": true, "msg": "AnsibleUndefinedVariable: 'fluentd_num_threads'
is undefined"}
fatal: [lago-basic-suite-master-host-0]: FAILED! => {"changed": false,
"failed": true, "msg": "AnsibleUndefinedVariable: 'fluentd_num_threads'
is undefined"}
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
"AnsibleUndefinedVariable: 'fluentd_num_threads' is undefined"}
</error>
*
--------------CCBA598F31714CCB8129AF90
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi, <br>
</p>
<p>We had failure in OST last night which seems to be related to two
fluentd configuration patchs. <br>
</p>
<p>I found that the error is actually in the deployment and can be
found in the lago log. <br>
</p>
<p>I am not sure it's related but I am also seeing unexpected errors
in the vdsm log for getDeviceList:
<a class="moz-txt-link-freetext" href="http://pastebin.test.redhat.com/520514">http://pastebin.test.redhat.com/520514</a></p>
<p>We apparently calling getDeviceList + checkStatus=true with no
guid. <br>
</p>
<p><br>
</p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-cd16-1d43-c8eb-3860974f6d9c">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches:
</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><b><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><b><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82329/">https://gerrit.ovirt.org/#/c/82329/</a> </b>
</span></b></p>
</b><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-cd16-1d43-c8eb-3860974f6d9c">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><b><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/81551/">https://gerrit.ovirt.org/#/c/81551/</a></b>
</span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job:</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><b><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2932/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2932/</a></b></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><b><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2933/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2933/</a></b>
</span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs:</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p>
<b><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2932/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2932/artifact/</a></b></b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-cd16-1d43-c8eb-3860974f6d9c"><b><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2933/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2933/artifact/</a></b></b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-cd16-1d43-c8eb-3860974f6d9c"><b></b><br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: </span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p>
<br>
TASK [ovirt_fluentd/client : Install non-ssl client]
***************************<br>
fatal: [lago-basic-suite-master-host-1]: FAILED! =>
{"changed": false, "failed": true, "msg":
"AnsibleUndefinedVariable: 'fluentd_num_threads' is undefined"}<br>
fatal: [lago-basic-suite-master-host-0]: FAILED! =>
{"changed": false, "failed": true, "msg":
"AnsibleUndefinedVariable: 'fluentd_num_threads' is undefined"}<br>
fatal: [localhost]: FAILED! => {"changed": false, "failed":
true, "msg": "AnsibleUndefinedVariable: 'fluentd_num_threads' is
undefined"}<br>
<br>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p>
</b><br class="Apple-interchange-newline">
</p>
</body>
</html>
--------------CCBA598F31714CCB8129AF90--
7 years, 1 month
Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #827
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/changes/13/75913/5:patch --prune
> git rev-parse origin/patch^{commit} # timeout=10
> git rev-parse patch^{commit} # timeout=10
Checking out Revision 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f (patch)
Commit message: "Exclude big packages from mirrors"
> git config core.sparsecheckout # timeout=10
> git checkout -f 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f
> git rev-list 4b0fe3e0c9fba26cdbaafe2b29fddd3411225d6f # timeout=10
[system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins6223189949086079841.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML
if self._latestRepoXML(local):
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1443, in _latestRepoXML
repomd = self.metalink_data.repomd
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 916, in <lambda>
metalink_data = property(fget=lambda self: self._getMetalink(),
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 912, in _getMetalink
self._metalink = metalink.MetaLinkRepoMD(self.metalink_filename)
File "/usr/lib/python2.7/site-packages/yum/metalink.py", line 189, in __init__
raise MetaLinkRepoErrorParseFail, "File %s is not XML" % filename
yum.metalink.MetaLinkRepoErrorParseFail: File /home/jenkins/mirrors_cache/fedora-updates-fc25/metalink.xml is not XML
Build step 'Execute shell' marked build as failure
7 years, 1 month
oVirt infra daily report - unstable production jobs - 458
by jenkins＠jenkins.phx.ovirt.org
------=_Part_836_1112201525.1506639606004
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Good morning!
Attached is the HTML page with the jenkins status report. You can see it also here:
- http://jenkins.ovirt.org/job/system_jenkins-report/458//artifact/exported...
Cheers,
Jenkins
------=_Part_836_1112201525.1506639606004
Content-Type: text/html; charset=us-ascii; name=upstream_report.html
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=upstream_report.html
Content-ID: <upstream_report.html>
<!DOCTYPE html><head><style type="text/css">
table.gridtable {
border-collapse: collapse;
table-layout:fixed;
width:1600px;
font-family: monospace;
font-size:13px;
}
.head {
font-size:20px;
font-family: arial;
}
.sub {
font-size:18px;
background-color:#e5e5e5;
font-family: arial;
}
pre {
font-family: monospace;
display: inline;
white-space: pre-wrap;
white-space: -moz-pre-wrap !important;
white-space: -pre-wrap;
white-space: -o-pre-wrap;
word-wrap: break-word;
}
</style>
</head>
<body>
<table class="gridtable" border=2>
<tr><th colspan=2 class=head>
RHEVM CI Jenkins Daily Report - 28/09/2017
</th></tr><tr><th colspan=2 class=sub>
<font color="blue"><a href="http://jenkins.ovirt.org/">00 Unstable Critical</a></font>
</th></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.1/">ovirt-system-tests_he-basic-suite-4.1</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
------=_Part_836_1112201525.1506639606004--
7 years, 1 month
