February 2017 - Infra - oVirt List Archives

[JIRA] (OVIRT-1157) Re-factor the standard-CI triggers file
by Barak Korren (oVirt JIRA) 26 Feb '17

26 Feb '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1157?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1157: -------------------------------- Labels: standard-ci (was: ) > Re-factor the standard-CI triggers file > --------------------------------------- > > Key: OVIRT-1157 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1157 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: Jenkins > Reporter: Barak Korren > Assignee: infra > Labels: standard-ci > > The {{/jobs/confs/yaml/triggers/standard.yaml}} file contains A LOT of needless code duplication. We need to re-factor it with templates and YAML inheritance. -- This message was sent by Atlassian JIRA (v1000.784.2#100032)

1 0

[JIRA] (OVIRT-682) Improve CI logging
by Barak Korren (oVirt JIRA) 26 Feb '17

26 Feb '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-682?page=com.atlassian.jira.p… ] Barak Korren edited comment on OVIRT-682 at 2/26/17 11:58 AM: -------------------------------------------------------------- We can probably improve the user experience by putting a better message in Gerrit, something like: {quote} You can see standard-CI script output log +here(1)+ For individual test results please +click here(2)+, or use the +test results analyser(3)+ Additional built artifacts can be found +here(4)+ For full, unfiltered, job output +click here(5)+ To re-trigger the job +click here(6)+ or type *\@ci check* {quote} Where: * (1) points to the {{check-patch.sh}}, {check-merged.sh}}, etc. script output. * If any JUNIT XML files were found (If not the line shouldnt be shown): ** (2) points to the test results page ** (3) point to the test results analyser * If any artifacts that are not mock logs or JUNIT XML files wee found (4) points to them. If <= 3 artifacts found we may even include direct links (This will be useful for things like {{differences.html}} in the Jenkins repo. * (5) Will point to the good-old job console output screen * If the job failed, (6) will point to the re-trigger button We will need to check if its possible to embed links in Gerrit comments. If no we will unfortunately have to use ugly full URLs. was (Author: bkorren(a)redhat.com) We can probably improve the user experience by putting a better message in Gerrit, something like: {quote} You can see standard-CI script output log +here(1)+ For individual test results please +click here(2)+, or use the +test results analyser(3)+ Additional built artifacts can be found +here(4)+ For full, unfiltered, job output +click here(5)+ {quote} Where: * (1) points to the {{check-patch.sh}}, {check-merged.sh}}, etc. script output. * If any JUNIT XML files were found (If not the line shouldnt be shown): ** (2) points to the test results page ** (3) point to the test results analyser * If any artifacts that are not mock logs or JUNIT XML files wee found (4) points to them. If <= 3 artifacts found we may even include direct links (This will be useful for things like {{differences.html}} in the Jenkins repo. * (5) Will point to the good-old job console output screen We will need to check if its possible to embed links in Gerrit comments. If no we will unfortunately have to use ugly full URLs. > Improve CI logging > ------------------ > > Key: OVIRT-682 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-682 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: eyal edri [Administrator] > Assignee: infra > > On Fri, Aug 12, 2016 at 8:08 PM, Nir Soffer <nsoffer(a)redhat.com> wrote: > > Hi all, > > > > Lately vdsm builds are failing in the install packages stage with > > this unhelpful error: > > > > 13:59:42 INFO: installing package(s): autoconf automake gdb ... > > 13:59:53 ERROR: Command failed. See logs for output. > > > > I downloaded the logs.tgz file from > > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- > > x86_64/1154/artifact/exported-artifacts/logs.tgz > > > > And indeed in /./vdsm/logs/mocker-fedora-24-x86_64.fc24.install_packages/ > > root.log > > I found found this error: > > > > DEBUG util.py:421: Error: Package: > > python-ioprocess-0.17.0-1.201608111609.gitbd272f2.fc24.noarch > > (ovirt-snapshot) > > DEBUG util.py:421: Requires: ioprocess = > > 0.17.0-1.201608111609.gitbd272f2.fc24 > > DEBUG util.py:421: Available: > > ioprocess-0.15.1-1.fc24.x86_64 (fedora) > > DEBUG util.py:421: ioprocess = 0.15.1-1.fc24 > > DEBUG util.py:421: Available: > > ioprocess-0.16.1-1.fc24.x86_64 (updates) > > DEBUG util.py:421: ioprocess = 0.16.1-1.fc24 > > DEBUG util.py:421: Available: > > ioprocess-0.17.0-1.201607121058.gitbd272f2.fc24.x86_64 > > (ovirt-snapshot) > > DEBUG util.py:421: ioprocess = > > 0.17.0-1.201607121058.gitbd272f2.fc24 > > DEBUG util.py:421: Available: > > ioprocess-0.17.0-1.201608111129.gitbd272f2.fc24.x86_64 > > (ovirt-snapshot) > > DEBUG util.py:421: ioprocess = > > 0.17.0-1.201608111129.gitbd272f2.fc24 > > > > So we have 2 issues here: > > > > 1. We need *all* errors in the console > > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- > > x86_64/1154/console > Not sure its possible or easy to do, these errors usually come from mock > and are not relevant to the code running on the console. > I've cc'd infra-support and updated topic to track it and check our options. > Going foward, we might want to install a log collector like logstash and > then searching errors there should be much easier. > > > > > > 2. Someone need to fix the ovirt-snapshot repository - it should have > > the missing ioprocess > > package. > > Maybe the project xml is not correct? > > > > Nir > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) -- This message was sent by Atlassian JIRA (v1000.784.2#100032)

1 0

[JIRA] (OVIRT-682) Improve CI logging
by Barak Korren (oVirt JIRA) 26 Feb '17

26 Feb '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-682?page=com.atlassian.jira.p… ] Barak Korren commented on OVIRT-682: ------------------------------------ We can probably improve the user experience by putting a better message in Gerrit, something like: {quote} You can see standard-CI script output log +here(1)+ For individual test results please +click here(2)+, or use the +test results analyser(3)+ Additional built artifacts can be found +here(4)+ For full, unfiltered, job output +click here(5)+ {quote} Where: * (1) points to the {{check-patch.sh}}, {check-merged.sh}}, etc. script output. * If any JUNIT XML files were found (If not the line shouldnt be shown): ** (2) points to the test results page ** (3) point to the test results analyser * If any artifacts that are not mock logs or JUNIT XML files wee found (4) points to them. If <= 3 artifacts found we may even include direct links (This will be useful for things like {{differences.html}} in the Jenkins repo. * (5) Will point to the good-old job console output screen We will need to check if its possible to embed links in Gerrit comments. If no we will unfortunately have to use ugly full URLs. > Improve CI logging > ------------------ > > Key: OVIRT-682 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-682 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: eyal edri [Administrator] > Assignee: infra > > On Fri, Aug 12, 2016 at 8:08 PM, Nir Soffer <nsoffer(a)redhat.com> wrote: > > Hi all, > > > > Lately vdsm builds are failing in the install packages stage with > > this unhelpful error: > > > > 13:59:42 INFO: installing package(s): autoconf automake gdb ... > > 13:59:53 ERROR: Command failed. See logs for output. > > > > I downloaded the logs.tgz file from > > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- > > x86_64/1154/artifact/exported-artifacts/logs.tgz > > > > And indeed in /./vdsm/logs/mocker-fedora-24-x86_64.fc24.install_packages/ > > root.log > > I found found this error: > > > > DEBUG util.py:421: Error: Package: > > python-ioprocess-0.17.0-1.201608111609.gitbd272f2.fc24.noarch > > (ovirt-snapshot) > > DEBUG util.py:421: Requires: ioprocess = > > 0.17.0-1.201608111609.gitbd272f2.fc24 > > DEBUG util.py:421: Available: > > ioprocess-0.15.1-1.fc24.x86_64 (fedora) > > DEBUG util.py:421: ioprocess = 0.15.1-1.fc24 > > DEBUG util.py:421: Available: > > ioprocess-0.16.1-1.fc24.x86_64 (updates) > > DEBUG util.py:421: ioprocess = 0.16.1-1.fc24 > > DEBUG util.py:421: Available: > > ioprocess-0.17.0-1.201607121058.gitbd272f2.fc24.x86_64 > > (ovirt-snapshot) > > DEBUG util.py:421: ioprocess = > > 0.17.0-1.201607121058.gitbd272f2.fc24 > > DEBUG util.py:421: Available: > > ioprocess-0.17.0-1.201608111129.gitbd272f2.fc24.x86_64 > > (ovirt-snapshot) > > DEBUG util.py:421: ioprocess = > > 0.17.0-1.201608111129.gitbd272f2.fc24 > > > > So we have 2 issues here: > > > > 1. We need *all* errors in the console > > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- > > x86_64/1154/console > Not sure its possible or easy to do, these errors usually come from mock > and are not relevant to the code running on the console. > I've cc'd infra-support and updated topic to track it and check our options. > Going foward, we might want to install a log collector like logstash and > then searching errors there should be much easier. > > > > > > 2. Someone need to fix the ovirt-snapshot repository - it should have > > the missing ioprocess > > package. > > Maybe the project xml is not correct? > > > > Nir > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) -- This message was sent by Atlassian JIRA (v1000.784.2#100032)

1 0

Clone job failing although all criteria is met
by Tal Nisan 26 Feb '17

26 Feb '17

Trying to clone this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1418280 With the new clone job fails on a missing flag although all flags exist: https://rhev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/bugzilla_clone_bug…

1 0

[JIRA] (OVIRT-892) Allow for optional bind mounts in Standard-CI
by Barak Korren (oVirt JIRA) 26 Feb '17

26 Feb '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-892?page=com.atlassian.jira.p… ] Barak Korren reassigned OVIRT-892: ---------------------------------- Assignee: Barak Korren (was: infra) > Allow for optional bind mounts in Standard-CI > --------------------------------------------- > > Key: OVIRT-892 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-892 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Barak Korren > Assignee: Barak Korren > Labels: containers, docker, standard-ci > > Standard-CI as supported a *.mounts file for a while. The way it works is that it fails if something it tries to mount does not exist. > We recently changed it (in OVIRT-751) so that if something that it tries to mount does not exist, it create an empty directory and mounts it instead. > We want to change the behaviour to make it more controllable so that we can specify in the mount file what to do if the thing we want to mount does not exist (And also what it should be, directory, file or a device), there are a few options: > # Fail - what it used to do > # Create what it needs - what it does now > # Just ignore and not mound > Option #3 is useful for situation where you mount sockets in order to connect to services on the host (Like Docker or Libvirt). If the bound mount is optional - we could gracefully handle the situation where the service is not running on the host by installing and running it from within the Standard-CI script. -- This message was sent by Atlassian JIRA (v1000.784.2#100032)

1 0

[JIRA] (OVIRT-892) Allow for optional bind mounts in Standard-CI
by Barak Korren (oVirt JIRA) 26 Feb '17

26 Feb '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-892?page=com.atlassian.jira.p… ] Barak Korren commented on OVIRT-892: ------------------------------------ Turns out Docker deamon is not willing to run from inside mock, so option #3 becomes useless. To fail as in option #1 is probably useful only when someone is creating brand-new CI scripts, so maybe no that important. We implemented a [patch|https://gerrit.ovirt.org/#/c/72991/] that detects if we're trying to mount a socket file instead of a directory and make sure that works. This solves the issue of mounting the Docket socket. > Allow for optional bind mounts in Standard-CI > --------------------------------------------- > > Key: OVIRT-892 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-892 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Barak Korren > Assignee: infra > Labels: containers, docker, standard-ci > > Standard-CI as supported a *.mounts file for a while. The way it works is that it fails if something it tries to mount does not exist. > We recently changed it (in OVIRT-751) so that if something that it tries to mount does not exist, it create an empty directory and mounts it instead. > We want to change the behaviour to make it more controllable so that we can specify in the mount file what to do if the thing we want to mount does not exist (And also what it should be, directory, file or a device), there are a few options: > # Fail - what it used to do > # Create what it needs - what it does now > # Just ignore and not mound > Option #3 is useful for situation where you mount sockets in order to connect to services on the host (Like Docker or Libvirt). If the bound mount is optional - we could gracefully handle the situation where the service is not running on the host by installing and running it from within the Standard-CI script. -- This message was sent by Atlassian JIRA (v1000.784.2#100032)

1 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #174
by jenkins＠jenkins.phx.ovirt.org 26 Feb '17

26 Feb '17

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 6b2f531b5f503b5673c35dbdffd8836e302b4df5 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 6b2f531b5f503b5673c35dbdffd8836e302b4df5 > git rev-list 6b2f531b5f503b5673c35dbdffd8836e302b4df5 # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/hudson8200751681655898103.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 687, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 727, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1451, in _commonLoadRepoXML if self._latestRepoXML(local): File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1420, in _latestRepoXML oxml = self._saveOldRepoXML(local) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1278, in _saveOldRepoXML shutil.copy2(local, old_local) File "/usr/lib64/python2.7/shutil.py", line 131, in copy2 copystat(src, dst) File "/usr/lib64/python2.7/shutil.py", line 98, in copystat os.utime(dst, (st.st_atime, st.st_mtime)) OSError: [Errno 2] No such file or directory: '/home/jenkins/mirrors_cache/fedora-base-fc25/repomd.xml.old.tmp' Build step 'Execute shell' marked build as failure

1 1

[JIRA] (OVIRT-1195) Unexplained failure in CI job
by Barak Korren (oVirt JIRA) 26 Feb '17

26 Feb '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1195?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-1195: ------------------------------------- I think this can also be because we're not blocking outside repos in the VMs any more. We probably need to go back to doing that, or implement [Lago issue #426|https://github.com/lago-project/lago/issues/426] to allow injecting the mirrors from the {{reposync-config.conf}} file in to the VMs > Unexplained failure in CI job > ----------------------------- > > Key: OVIRT-1195 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1195 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: Jenkins > Reporter: Yaniv Kaul > Assignee: infra > Priority: High > > See http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_… > I'm not sure what the failure is. All I see is: > 18:50:23 Start: yum install > 18:50:37 ERROR: Command failed. See logs for output. -- This message was sent by Atlassian JIRA (v1000.784.2#100032)

1 0

[JIRA] (OVIRT-1195) Unexplained failure in CI job
by eyal edri [Administrator] (oVirt JIRA) 26 Feb '17

26 Feb '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1195?page=com.atlassian.jira.… ] eyal edri [Administrator] commented on OVIRT-1195: -------------------------------------------------- We still used the old snapshot repo, I changed it to tested: https://gerrit.ovirt.org/73093 Once this patch passes CI, I'll merge it and hopefully it should solve any missing RPMs from the snapshot repo. > Unexplained failure in CI job > ----------------------------- > > Key: OVIRT-1195 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1195 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: Jenkins > Reporter: Yaniv Kaul > Assignee: infra > Priority: High > > See http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_… > I'm not sure what the failure is. All I see is: > 18:50:23 Start: yum install > 18:50:37 ERROR: Command failed. See logs for output. -- This message was sent by Atlassian JIRA (v1000.784.2#100032)

1 0

[JIRA] (OVIRT-1195) Unexplained failure in CI job
by Gil Shinar (oVirt JIRA) 26 Feb '17

26 Feb '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1195?page=com.atlassian.jira.… ] Gil Shinar commented on OVIRT-1195: ----------------------------------- You should look in the logs that are being archived in the build main page [1] *DEBUG util.py:435: No Presto metadata available for centos-updates-el7 DEBUG util.py:435: http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/x86_64/python-…: [Errno 14] curl#5 - "Could not resolve proxy: proxy01.phx.ovirt.org" DEBUG util.py:435: Trying other mirror. DEBUG util.py:435: Error downloading packages: DEBUG util.py:435: python-ovirt-engine-sdk4-4.2.0-0.a0.20170220gita7305aa.el7.centos.x86_64: [Errno 256] No more mirrors to try. DEBUG util.py:573: Child return code was: 1* Looks like a centos mirrors issue [1] http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_… > Unexplained failure in CI job > ----------------------------- > > Key: OVIRT-1195 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1195 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: Jenkins > Reporter: Yaniv Kaul > Assignee: infra > Priority: High > > See http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_… > I'm not sure what the failure is. All I see is: > 18:50:23 Start: yum install > 18:50:37 ERROR: Command failed. See logs for output. -- This message was sent by Atlassian JIRA (v1000.784.2#100032)

1 0