- Infra - oVirt List Archives

Build failed in Jenkins: system-sync_mirrors-centos-qemu-ev-release-el7-x86_64 #2354
by jenkins＠jenkins.phx.ovirt.org 22 Aug '19

22 Aug '19

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-qemu-ev-release-el7…> ------------------------------------------ Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-qemu-ev-release-el7…> No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision cfeb68c000d32e1c9843a2f8650b7b9a65e01a8e (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f cfeb68c000d32e1c9843a2f8650b7b9a65e01a8e Commit message: "Disable use_bootstrap_container for Fedora rawhide" > git rev-list --no-walk cfeb68c000d32e1c9843a2f8650b7b9a65e01a8e # timeout=10 [system-sync_mirrors-centos-qemu-ev-release-el7-x86_64] $ /bin/bash -xe /tmp/jenkins6854356434980691008.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-qemu-ev-release-el7 x86_64 jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror centos-qemu-ev-release-el7 x86_64 jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror centos-qemu-ev-release-el7 x86_64 jenkins/data/mirrors-reposync.conf + local repo_name=centos-qemu-ev-release-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs centos-qemu-ev-release-el7 yum + local repo_name=centos-qemu-ev-release-el7 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/centos-qemu-ev-release-el7 /var/www/html/repos/yum/centos-qemu-ev-release-el7/base + check_yum_sync_needed centos-qemu-ev-release-el7 x86_64 jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=centos-qemu-ev-release-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/centos-qemu-ev-release-el7 ++ IFS=, ++ echo x86_64 + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync centos-qemu-ev-release-el7 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=centos-qemu-ev-release-el7 ++ local repo_arch=x86_64 ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=centos-qemu-ev-release-el7 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/centos-qemu-ev-release-el7/base --norepopath --newest-only --urls --quiet Traceback (most recent call last): File "/usr/bin/reposync", line 373, in <module> main() File "/usr/bin/reposync", line 185, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1477, in _commonLoadRepoXML if self._latestRepoXML(local): File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1455, in _latestRepoXML repomd = self.metalink_data.repomd File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 921, in <lambda> metalink_data = property(fget=lambda self: self._getMetalink(), File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 917, in _getMetalink self._metalink = metalink.MetaLinkRepoMD(self.metalink_filename) File "/usr/lib/python2.7/site-packages/yum/metalink.py", line 189, in __init__ raise MetaLinkRepoErrorParseFail, "File %s is not XML" % filename yum.metalink.MetaLinkRepoErrorParseFail: File /home/jenkins/mirrors_cache/fedora-base-fc29/metalink.xml is not XML + reposync_out='Could not get metalink https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64 error was 16: error opening file from https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64, IOError: [Errno 2] No such file or directory: '\''/home/jenkins/mirrors_cache/fedora-base-fc29/metalink.xml.tmp'\''' Build step 'Execute shell' marked build as failure

1 1

[JIRA] (OVIRT-2781) https://artifactory.ovirt.org/ has invalid ssl certificate for HSTS
by Sandro Bonazzola (oVirt JIRA) 22 Aug '19

22 Aug '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2781?page=com.atlassian.jira.… ] Sandro Bonazzola reassigned OVIRT-2781: --------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > https://artifactory.ovirt.org/ has invalid ssl certificate for HSTS > ------------------------------------------------------------------- > > Key: OVIRT-2781 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2781 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Sandro Bonazzola > Assignee: Evgheni Dereveanchin > > https://artifactory.ovirt.org/ has invalid ssl certificate for HSTS, > expired on June, 26th 2019 > Please fix ssl certificate. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100107)

1 0

[JIRA] (OVIRT-2781) https://artifactory.ovirt.org/ has invalid ssl certificate for HSTS
by Sandro Bonazzola (oVirt JIRA) 22 Aug '19

22 Aug '19

Sandro Bonazzola created OVIRT-2781: --------------------------------------- Summary: https://artifactory.ovirt.org/ has invalid ssl certificate for HSTS Key: OVIRT-2781 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2781 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Sandro Bonazzola Assignee: infra https://artifactory.ovirt.org/ has invalid ssl certificate for HSTS, expired on June, 26th 2019 Please fix ssl certificate. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100107)

1 0

Relaunching the website migration project
by Marc Dequènes (Duck) 22 Aug '19

22 Aug '19

Quack, ### A pinch of history As a short introduction for those who do not know me as I rarely post here: I'm part of Red Hat OSPO Comminfra Team (previously OSPO was called OSAS) and currently hosting and taking care of the website. This topic has been discussed many times: neither people writing the site neither us for the maintenance are happy with the current state of affair. But we had not real plan and picking a random tool which might lead to similar problems and without any way to support it properly was not very appealing. Moreover we tried migrating to Middleman 4, the obvious path to modernization, but this happened to be more complicated than expected and this new version had terrible performance problems. As you now version 3 is already slow, and that's quite an understatement. In the meanwhile a few things have changed, and people taking care of the website too, so let's talk about the current situation. (very short version of the history around this) ### Why now? What has changed: * several projects moved to Jekyll, a software we were already investigating at the time and is being used by Google Pages (they created it in fact), and they were very happy with the result; we believe it has the necessary features, it is well maintained because Google uses it heavily, and it's very fast * we've tried Jekyll ourselves, we're happy too, and we do have the Ansible playbooks and tooling to deploy it now * over time the website had some waves of cleanup (still needs quite some love though) and recently the blog moved to WordPress to get comment support and a more friendly interface (see OVIRT-2652) which allowed more cleanup (see #2030) With the blog separated and a lot of custom Ruby code removed, the tooling ready, I believe we can now work on migrating the content. We may realize Jekyll is not the right tool, but people seemed to like the idea at the time and current experience seem to indicate it should improve things and be maintainable. The goal here is to experiment and switch to production only if we're happy with it. ### Early work I have started a branch called 'jekyll_migration' to put my work on it. This is very early work (I just started), I already hit various difficulties, and I can't commit 100% of my time on it so it will require some time. Several of my changes happened to not be really specific to the migration, and the current site would benefit from these fixes/cleanups/simplifications… thus I'll extracts these changes and create separate PRs for master. If you wish to help, then you can contact me directly or reply to this thread. You may also create PRs to this topic branch, but please do not push anything directly. Regards. \_o<

5 9

[JIRA] (OVIRT-2780) Fwd: [ovirt-devel] jenkins CI failing to fetch repo
by Eyal Edri (oVirt JIRA) 22 Aug '19

22 Aug '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2780?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2780: ---------------------------------- possible fix for extending the timeout of pipeline jobs: [https://gerrit.ovirt.org/#/c/102774/|https://gerrit.ovirt.org/#/c/102774/] > Fwd: [ovirt-devel] jenkins CI failing to fetch repo > --------------------------------------------------- > > Key: OVIRT-2780 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2780 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Ehud Yonasi > Assignee: infra > > ---------- Forwarded message --------- > From: Miguel Duarte de Mora Barroso <mdbarroso(a)redhat.com> > Date: Thu, Aug 22, 2019 at 11:02 AM > Subject: [ovirt-devel] jenkins CI failing to fetch repo > To: oVirt devel <devel(a)ovirt.org>, oVirt Jenkins <jenkins(a)ovirt.org> > Hi, > Since yesterday I can't get ovirt-provider-ovn builds to run, each > manifesting the error below: > error: index-pack died of signal 15 > 09:27:35 fatal: index-pack failed > 09:27:35 > 09:27:35 at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042) > 09:27:35 at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761) > 09:27:35 at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72) > 09:27:35 at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442) > 09:27:35 at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:655) > 09:27:35 at > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) > 09:27:35 at > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) > 09:27:35 at hudson.remoting.UserRequest.perform(UserRequest.java:212) > 09:27:35 at hudson.remoting.UserRequest.perform(UserRequest.java:54) > 09:27:35 at hudson.remoting.Request$2.run(Request.java:369) > 09:27:35 at > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > 09:27:35 at java.util.concurrent.FutureTask.run(FutureTask.java:266) > 09:27:35 at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 09:27:35 at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 09:27:35 at java.lang.Thread.run(Thread.java:748) > 09:27:35 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: > Remote call to vm0038.workers-phx.ovirt.org > 09:27:35 at > hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) > 09:27:35 at > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) > 09:27:35 at hudson.remoting.Channel.call(Channel.java:957) > 09:27:35 at > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146) > 09:27:35 at sun.reflect.GeneratedMethodAccessor793.invoke(Unknown Source) > 09:27:35 at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > 09:27:35 at java.lang.reflect.Method.invoke(Method.java:498) > 09:27:35 at > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132) > 09:27:35 at com.sun.proxy.$Proxy118.execute(Unknown Source) > 09:27:35 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152) > 09:27:35 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) > 09:27:35 at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) > 09:27:35 at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) > 09:27:35 at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) > 09:27:35 at > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) > 09:27:35 at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > 09:27:35 ... 4 more > It can be seen on: > - > https://jenkins.ovirt.org/job/ovirt-provider-ovn_standard-check-patch/2476/… > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/XOYJOK4AXT7AK… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100107)

1 0

Re: [ovirt-devel] Re: jenkins CI failing to fetch repo
by Eyal Edri 22 Aug '19

22 Aug '19

It looks like we're still suffering from the network issues that are affecting out PHX DC, this is still being investigated by IT. For now I've sent a patch [1] to increase the timeout on pipeline jobs to 20 min, so git clones will work even if the network is slow. [1] https://gerrit.ovirt.org/#/c/102774/ On Thu, Aug 22, 2019 at 11:41 AM Miguel Duarte de Mora Barroso < mdbarroso(a)redhat.com> wrote: > On Thu, Aug 22, 2019 at 10:01 AM Miguel Duarte de Mora Barroso > <mdbarroso(a)redhat.com> wrote: > > > > Hi, > > > > Since yesterday I can't get ovirt-provider-ovn builds to run, each > > manifesting the error below: > > More info: this seems to only happen on the fedora29 target. > > > > > error: index-pack died of signal 15 > > 09:27:35 fatal: index-pack failed > > 09:27:35 > > 09:27:35 at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042) > > 09:27:35 at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761) > > 09:27:35 at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72) > > 09:27:35 at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442) > > 09:27:35 at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:655) > > 09:27:35 at > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) > > 09:27:35 at > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) > > 09:27:35 at hudson.remoting.UserRequest.perform(UserRequest.java:212) > > 09:27:35 at hudson.remoting.UserRequest.perform(UserRequest.java:54) > > 09:27:35 at hudson.remoting.Request$2.run(Request.java:369) > > 09:27:35 at > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > > 09:27:35 at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > 09:27:35 at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > 09:27:35 at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > 09:27:35 at java.lang.Thread.run(Thread.java:748) > > 09:27:35 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: > > Remote call to vm0038.workers-phx.ovirt.org > > 09:27:35 at > hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) > > 09:27:35 at > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) > > 09:27:35 at hudson.remoting.Channel.call(Channel.java:957) > > 09:27:35 at > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146) > > 09:27:35 at sun.reflect.GeneratedMethodAccessor793.invoke(Unknown > Source) > > 09:27:35 at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > 09:27:35 at java.lang.reflect.Method.invoke(Method.java:498) > > 09:27:35 at > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132) > > 09:27:35 at com.sun.proxy.$Proxy118.execute(Unknown Source) > > 09:27:35 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152) > > 09:27:35 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) > > 09:27:35 at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) > > 09:27:35 at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) > > 09:27:35 at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) > > 09:27:35 at > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) > > 09:27:35 at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > 09:27:35 ... 4 more > > > > It can be seen on: > > - > https://jenkins.ovirt.org/job/ovirt-provider-ovn_standard-check-patch/2476/… > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/2XK4PBKQVPIPH… > -- Eyal edri He / Him / His MANAGER CONTINUOUS PRODUCTIZATION SYSTEM ENGINEERING Red Hat <https://www.redhat.com/> <https://red.ht/sig> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel)

1 0

[JIRA] (OVIRT-2779) New ovirt project
by Emil Natan (oVirt JIRA) 22 Aug '19

22 Aug '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2779?page=com.atlassian.jira.… ] Emil Natan reassigned OVIRT-2779: --------------------------------- Assignee: Emil Natan (was: infra) > New ovirt project > ----------------- > > Key: OVIRT-2779 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2779 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Yuval Turgeman > Assignee: Emil Natan > > Hi, > Can you please create a new project in gerrit: ovirt-node-coreos ? > Thanks ! -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100107)

1 0

[CQ]: 102461, 15 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 22 Aug '19

22 Aug '19

A system test invoked by the "ovirt-master" change queue including change 102461,15 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 102733,2 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 102733,2 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/102461/15 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/102733/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/15473/

1 0

[JIRA] (OVIRT-2780) Fwd: [ovirt-devel] jenkins CI failing to fetch repo
by Ehud Yonasi (oVirt JIRA) 22 Aug '19

22 Aug '19

Ehud Yonasi created OVIRT-2780: ---------------------------------- Summary: Fwd: [ovirt-devel] jenkins CI failing to fetch repo Key: OVIRT-2780 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2780 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Ehud Yonasi Assignee: infra ---------- Forwarded message --------- From: Miguel Duarte de Mora Barroso <mdbarroso(a)redhat.com> Date: Thu, Aug 22, 2019 at 11:02 AM Subject: [ovirt-devel] jenkins CI failing to fetch repo To: oVirt devel <devel(a)ovirt.org>, oVirt Jenkins <jenkins(a)ovirt.org> Hi, Since yesterday I can't get ovirt-provider-ovn builds to run, each manifesting the error below: error: index-pack died of signal 15 09:27:35 fatal: index-pack failed 09:27:35 09:27:35 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042) 09:27:35 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761) 09:27:35 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72) 09:27:35 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442) 09:27:35 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:655) 09:27:35 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) 09:27:35 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) 09:27:35 at hudson.remoting.UserRequest.perform(UserRequest.java:212) 09:27:35 at hudson.remoting.UserRequest.perform(UserRequest.java:54) 09:27:35 at hudson.remoting.Request$2.run(Request.java:369) 09:27:35 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 09:27:35 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 09:27:35 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 09:27:35 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 09:27:35 at java.lang.Thread.run(Thread.java:748) 09:27:35 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to vm0038.workers-phx.ovirt.org 09:27:35 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) 09:27:35 at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) 09:27:35 at hudson.remoting.Channel.call(Channel.java:957) 09:27:35 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146) 09:27:35 at sun.reflect.GeneratedMethodAccessor793.invoke(Unknown Source) 09:27:35 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 09:27:35 at java.lang.reflect.Method.invoke(Method.java:498) 09:27:35 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132) 09:27:35 at com.sun.proxy.$Proxy118.execute(Unknown Source) 09:27:35 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152) 09:27:35 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) 09:27:35 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) 09:27:35 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) 09:27:35 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) 09:27:35 at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) 09:27:35 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 09:27:35 ... 4 more It can be seen on: - https://jenkins.ovirt.org/job/ovirt-provider-ovn_standard-check-patch/2476/… _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/XOYJOK4AXT7AK… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100107)

1 0

Sizing load on oVirt CI's "loader" containers
by Barak Korren 22 Aug '19

22 Aug '19

Our STDCI V2 system make extensive use of the so-called 'loader' PODs except for very special cases, every job we have allocates and uses such a POD at least once. Those PODs are used for among other things: 1. Loading Jenkins pipeline groovy code 2. Loading the scripts from our `jenkins` repo 3. Running the STDCI V2 logic to parse the YAML files and figure out what to run on which resources 4. Rendering the STDCI V2 graphical report The PODs are configured to require 500Mib of ram and run on the zone:ci, type:vm hosts. This means they end up running on one of the following VMs: name memory shift-n04.phx.ovirt.org 16264540Ki shift-n05.phx.ovirt.org 16264540Ki shift-n06.phx.ovirt.org 16264528Ki So if we make the simple calculation of how many such pods can run on 16Gib vms, we come up with the theoretical result of 96, but Running a query like the following on one of those hosts reveals that we share those hosts with many other containers: oc get --all-namespaces pods --field-selector=spec.nodeName= shift-n04.phx.ovirt.org,status.phase==Running I suspect allocation of the loader container is starting to be a bottleneck. I think we might have to either increase the amount of RAM the VMs have, or make the loader containers require less RAM. But we need to be able to measure some things better to make a decision. Do we have ongoing metrics for: - What does the RAM utilization on the relevant VMs looks like - How much ram is actually used inside the loader containers WDYT? -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 0