[JIRA] (OVIRT-682) Improve CI logging
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-682?page=com.atlassian.jira... ]
Barak Korren edited comment on OVIRT-682 at 2/26/17 11:58 AM:
--------------------------------------------------------------
We can probably improve the user experience by putting a better message in Gerrit, something like:
{quote}
You can see standard-CI script output log +here(1)+
For individual test results please +click here(2)+, or use the +test results analyser(3)+
Additional built artifacts can be found +here(4)+
For full, unfiltered, job output +click here(5)+
To re-trigger the job +click here(6)+ or type *\@ci check*
{quote}
Where:
* (1) points to the {{check-patch.sh}}, {check-merged.sh}}, etc. script output.
* If any JUNIT XML files were found (If not the line shouldnt be shown):
** (2) points to the test results page
** (3) point to the test results analyser
* If any artifacts that are not mock logs or JUNIT XML files wee found (4) points to them. If <= 3 artifacts found we may even include direct links (This will be useful for things like {{differences.html}} in the Jenkins repo.
* (5) Will point to the good-old job console output screen
* If the job failed, (6) will point to the re-trigger button
We will need to check if its possible to embed links in Gerrit comments. If no we will unfortunately have to use ugly full URLs.
was (Author: bkorren(a)redhat.com):
We can probably improve the user experience by putting a better message in Gerrit, something like:
{quote}
You can see standard-CI script output log +here(1)+
For individual test results please +click here(2)+, or use the +test results analyser(3)+
Additional built artifacts can be found +here(4)+
For full, unfiltered, job output +click here(5)+
{quote}
Where:
* (1) points to the {{check-patch.sh}}, {check-merged.sh}}, etc. script output.
* If any JUNIT XML files were found (If not the line shouldnt be shown):
** (2) points to the test results page
** (3) point to the test results analyser
* If any artifacts that are not mock logs or JUNIT XML files wee found (4) points to them. If <= 3 artifacts found we may even include direct links (This will be useful for things like {{differences.html}} in the Jenkins repo.
* (5) Will point to the good-old job console output screen
We will need to check if its possible to embed links in Gerrit comments. If no we will unfortunately have to use ugly full URLs.
> Improve CI logging
> ------------------
>
> Key: OVIRT-682
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-682
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: eyal edri [Administrator]
> Assignee: infra
>
> On Fri, Aug 12, 2016 at 8:08 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
> > Hi all,
> >
> > Lately vdsm builds are failing in the install packages stage with
> > this unhelpful error:
> >
> > 13:59:42 INFO: installing package(s): autoconf automake gdb ...
> > 13:59:53 ERROR: Command failed. See logs for output.
> >
> > I downloaded the logs.tgz file from
> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-
> > x86_64/1154/artifact/exported-artifacts/logs.tgz
> >
> > And indeed in /./vdsm/logs/mocker-fedora-24-x86_64.fc24.install_packages/
> > root.log
> > I found found this error:
> >
> > DEBUG util.py:421: Error: Package:
> > python-ioprocess-0.17.0-1.201608111609.gitbd272f2.fc24.noarch
> > (ovirt-snapshot)
> > DEBUG util.py:421: Requires: ioprocess =
> > 0.17.0-1.201608111609.gitbd272f2.fc24
> > DEBUG util.py:421: Available:
> > ioprocess-0.15.1-1.fc24.x86_64 (fedora)
> > DEBUG util.py:421: ioprocess = 0.15.1-1.fc24
> > DEBUG util.py:421: Available:
> > ioprocess-0.16.1-1.fc24.x86_64 (updates)
> > DEBUG util.py:421: ioprocess = 0.16.1-1.fc24
> > DEBUG util.py:421: Available:
> > ioprocess-0.17.0-1.201607121058.gitbd272f2.fc24.x86_64
> > (ovirt-snapshot)
> > DEBUG util.py:421: ioprocess =
> > 0.17.0-1.201607121058.gitbd272f2.fc24
> > DEBUG util.py:421: Available:
> > ioprocess-0.17.0-1.201608111129.gitbd272f2.fc24.x86_64
> > (ovirt-snapshot)
> > DEBUG util.py:421: ioprocess =
> > 0.17.0-1.201608111129.gitbd272f2.fc24
> >
> > So we have 2 issues here:
> >
> > 1. We need *all* errors in the console
> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-
> > x86_64/1154/console
> Not sure its possible or easy to do, these errors usually come from mock
> and are not relevant to the code running on the console.
> I've cc'd infra-support and updated topic to track it and check our options.
> Going foward, we might want to install a log collector like logstash and
> then searching errors there should be much easier.
> >
> >
> > 2. Someone need to fix the ovirt-snapshot repository - it should have
> > the missing ioprocess
> > package.
> > Maybe the project xml is not correct?
> >
> > Nir
> >
> --
> Eyal Edri
> Associate Manager
> RHV DevOps
> EMEA ENG Virtualization R&D
> Red Hat Israel
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
This message was sent by Atlassian JIRA
(v1000.784.2#100032)
8 years, 2 months
[JIRA] (OVIRT-682) Improve CI logging
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-682?page=com.atlassian.jira... ]
Barak Korren commented on OVIRT-682:
------------------------------------
We can probably improve the user experience by putting a better message in Gerrit, something like:
{quote}
You can see standard-CI script output log +here(1)+
For individual test results please +click here(2)+, or use the +test results analyser(3)+
Additional built artifacts can be found +here(4)+
For full, unfiltered, job output +click here(5)+
{quote}
Where:
* (1) points to the {{check-patch.sh}}, {check-merged.sh}}, etc. script output.
* If any JUNIT XML files were found (If not the line shouldnt be shown):
** (2) points to the test results page
** (3) point to the test results analyser
* If any artifacts that are not mock logs or JUNIT XML files wee found (4) points to them. If <= 3 artifacts found we may even include direct links (This will be useful for things like {{differences.html}} in the Jenkins repo.
* (5) Will point to the good-old job console output screen
We will need to check if its possible to embed links in Gerrit comments. If no we will unfortunately have to use ugly full URLs.
> Improve CI logging
> ------------------
>
> Key: OVIRT-682
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-682
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: eyal edri [Administrator]
> Assignee: infra
>
> On Fri, Aug 12, 2016 at 8:08 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
> > Hi all,
> >
> > Lately vdsm builds are failing in the install packages stage with
> > this unhelpful error:
> >
> > 13:59:42 INFO: installing package(s): autoconf automake gdb ...
> > 13:59:53 ERROR: Command failed. See logs for output.
> >
> > I downloaded the logs.tgz file from
> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-
> > x86_64/1154/artifact/exported-artifacts/logs.tgz
> >
> > And indeed in /./vdsm/logs/mocker-fedora-24-x86_64.fc24.install_packages/
> > root.log
> > I found found this error:
> >
> > DEBUG util.py:421: Error: Package:
> > python-ioprocess-0.17.0-1.201608111609.gitbd272f2.fc24.noarch
> > (ovirt-snapshot)
> > DEBUG util.py:421: Requires: ioprocess =
> > 0.17.0-1.201608111609.gitbd272f2.fc24
> > DEBUG util.py:421: Available:
> > ioprocess-0.15.1-1.fc24.x86_64 (fedora)
> > DEBUG util.py:421: ioprocess = 0.15.1-1.fc24
> > DEBUG util.py:421: Available:
> > ioprocess-0.16.1-1.fc24.x86_64 (updates)
> > DEBUG util.py:421: ioprocess = 0.16.1-1.fc24
> > DEBUG util.py:421: Available:
> > ioprocess-0.17.0-1.201607121058.gitbd272f2.fc24.x86_64
> > (ovirt-snapshot)
> > DEBUG util.py:421: ioprocess =
> > 0.17.0-1.201607121058.gitbd272f2.fc24
> > DEBUG util.py:421: Available:
> > ioprocess-0.17.0-1.201608111129.gitbd272f2.fc24.x86_64
> > (ovirt-snapshot)
> > DEBUG util.py:421: ioprocess =
> > 0.17.0-1.201608111129.gitbd272f2.fc24
> >
> > So we have 2 issues here:
> >
> > 1. We need *all* errors in the console
> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-
> > x86_64/1154/console
> Not sure its possible or easy to do, these errors usually come from mock
> and are not relevant to the code running on the console.
> I've cc'd infra-support and updated topic to track it and check our options.
> Going foward, we might want to install a log collector like logstash and
> then searching errors there should be much easier.
> >
> >
> > 2. Someone need to fix the ovirt-snapshot repository - it should have
> > the missing ioprocess
> > package.
> > Maybe the project xml is not correct?
> >
> > Nir
> >
> --
> Eyal Edri
> Associate Manager
> RHV DevOps
> EMEA ENG Virtualization R&D
> Red Hat Israel
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
This message was sent by Atlassian JIRA
(v1000.784.2#100032)
8 years, 2 months
[JIRA] (OVIRT-892) Allow for optional bind mounts in Standard-CI
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-892?page=com.atlassian.jira... ]
Barak Korren reassigned OVIRT-892:
----------------------------------
Assignee: Barak Korren (was: infra)
> Allow for optional bind mounts in Standard-CI
> ---------------------------------------------
>
> Key: OVIRT-892
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-892
> Project: oVirt - virtualization made easy
> Issue Type: Improvement
> Reporter: Barak Korren
> Assignee: Barak Korren
> Labels: containers, docker, standard-ci
>
> Standard-CI as supported a *.mounts file for a while. The way it works is that it fails if something it tries to mount does not exist.
> We recently changed it (in OVIRT-751) so that if something that it tries to mount does not exist, it create an empty directory and mounts it instead.
> We want to change the behaviour to make it more controllable so that we can specify in the mount file what to do if the thing we want to mount does not exist (And also what it should be, directory, file or a device), there are a few options:
> # Fail - what it used to do
> # Create what it needs - what it does now
> # Just ignore and not mound
> Option #3 is useful for situation where you mount sockets in order to connect to services on the host (Like Docker or Libvirt). If the bound mount is optional - we could gracefully handle the situation where the service is not running on the host by installing and running it from within the Standard-CI script.
--
This message was sent by Atlassian JIRA
(v1000.784.2#100032)
8 years, 2 months
[JIRA] (OVIRT-892) Allow for optional bind mounts in Standard-CI
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-892?page=com.atlassian.jira... ]
Barak Korren commented on OVIRT-892:
------------------------------------
Turns out Docker deamon is not willing to run from inside mock, so option #3 becomes useless.
To fail as in option #1 is probably useful only when someone is creating brand-new CI scripts, so maybe no that important.
We implemented a [patch|https://gerrit.ovirt.org/#/c/72991/] that detects if we're trying to mount a socket file instead of a directory and make sure that works. This solves the issue of mounting the Docket socket.
> Allow for optional bind mounts in Standard-CI
> ---------------------------------------------
>
> Key: OVIRT-892
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-892
> Project: oVirt - virtualization made easy
> Issue Type: Improvement
> Reporter: Barak Korren
> Assignee: infra
> Labels: containers, docker, standard-ci
>
> Standard-CI as supported a *.mounts file for a while. The way it works is that it fails if something it tries to mount does not exist.
> We recently changed it (in OVIRT-751) so that if something that it tries to mount does not exist, it create an empty directory and mounts it instead.
> We want to change the behaviour to make it more controllable so that we can specify in the mount file what to do if the thing we want to mount does not exist (And also what it should be, directory, file or a device), there are a few options:
> # Fail - what it used to do
> # Create what it needs - what it does now
> # Just ignore and not mound
> Option #3 is useful for situation where you mount sockets in order to connect to services on the host (Like Docker or Libvirt). If the bound mount is optional - we could gracefully handle the situation where the service is not running on the host by installing and running it from within the Standard-CI script.
--
This message was sent by Atlassian JIRA
(v1000.784.2#100032)
8 years, 2 months
Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #174
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6b2f531b5f503b5673c35dbdffd8836e302b4df5 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 6b2f531b5f503b5673c35dbdffd8836e302b4df5
> git rev-list 6b2f531b5f503b5673c35dbdffd8836e302b4df5 # timeout=10
[system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/hudson8200751681655898103.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 687, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 727, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1451, in _commonLoadRepoXML
if self._latestRepoXML(local):
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1420, in _latestRepoXML
oxml = self._saveOldRepoXML(local)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1278, in _saveOldRepoXML
shutil.copy2(local, old_local)
File "/usr/lib64/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
File "/usr/lib64/python2.7/shutil.py", line 98, in copystat
os.utime(dst, (st.st_atime, st.st_mtime))
OSError: [Errno 2] No such file or directory: '/home/jenkins/mirrors_cache/fedora-base-fc25/repomd.xml.old.tmp'
Build step 'Execute shell' marked build as failure
8 years, 2 months
