[JIRA] (OVIRT-1756) configure jenkins.ovirt.org to G1 garbage collector
by rgolanbb (oVirt JIRA)
This is a multi-part message in MIME format...
------------=_1510506038-15316-115
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
rgolanbb created OVIRT-1756:
-------------------------------
Summary: configure jenkins.ovirt.org to G1 garbage collector
Key: OVIRT-1756
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1756
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: rgolanbb
Assignee: infra
I noticed jenkins process cpu consumption going over 100% and stalling the
web handlers. The io wait is not a problem nor mem.
What I suspect is going on is tons of GC and GC pressure given that 12GB
heap and fairly nice amount of users and requests.
What we can do is to configure the gc logging to see if that is really GC
pauses really and to move to using the G1 garbage collector.
See this post from cloudbees on the move to G1 collector
https://www.cloudbees.com/blog/joining-big-leagues-tuning-jenkins-gc-resp...
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100070)
------------=_1510506038-15316-115
Content-Type: text/html; charset="UTF-8"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
<html><body>
<h3>rgolanbb created OVIRT-1756:</h3>
<pre> Summary: configure jenkins.ovirt.org to G1 garbage collector
Key: OVIRT-1756
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1756
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: rgolanbb
Assignee: infra</pre>
<p>I noticed jenkins process cpu consumption going over 100% and stalling the web handlers. The io wait is not a problem nor mem.</p>
<p>What I suspect is going on is tons of GC and GC pressure given that 12GB heap and fairly nice amount of users and requests.</p>
<p>What we can do is to configure the gc logging to see if that is really GC pauses really and to move to using the G1 garbage collector.</p>
<p>See this post from cloudbees on the move to G1 collector <a href="https://www.cloudbees.com/blog/joining-big-leagues-tuning-jenkins-gc-resp...">https://www.cloudbees.com/blog/joining-big-leagues-tuning-jenkins-gc-resp...</a></p>
<p>— This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100070)</p>
<img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYL..." alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/>
</body></html>
------------=_1510506038-15316-115--
7 years
Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #977
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision af75c3610da7b918706f8766c69be3bee3734792 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f af75c3610da7b918706f8766c69be3bee3734792
Commit message: "ovirt-engine: Add FC26 build"
> git rev-list af75c3610da7b918706f8766c69be3bee3734792 # timeout=10
[system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins3411637261384965401.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1456, in _commonLoadRepoXML
local = self.cachedir + '/repomd.xml'
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 772, in <lambda>
cachedir = property(lambda self: self._dirGetAttr('cachedir'))
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 755, in _dirGetAttr
self.dirSetup()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 734, in dirSetup
self._dirSetupMkdir_p(dir)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 711, in _dirSetupMkdir_p
raise Errors.RepoError, msg
yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7'
Build step 'Execute shell' marked build as failure
7 years
[JIRA] (OVIRT-1751) Re: Delivery Status Notification (Failure)
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1751?page=com.atlassian.jir... ]
Barak Korren updated OVIRT-1751:
--------------------------------
Resolution: Won't Fix
Status: Done (was: To Do)
> Re: Delivery Status Notification (Failure)
> ------------------------------------------
>
> Key: OVIRT-1751
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1751
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Nir Soffer
> Assignee: infra
> Attachments: icon.png
>
>
> Mail from jenkins(a)ovirt.org should use a reply-to header so replies go the
> proper
> mailing list (e.g. infra(a)ovirt.org), or if we don't have anything that make
> sense,
> noreply(a)ovirt.org?
> On Sat, Nov 11, 2017 at 2:02 AM Mail Delivery Subsystem <
> mailer-daemon(a)googlemail.com> wrote:
> > [image: icon.png]
> > Address not found
> > Your message wasn't delivered to *jenkins(a)ovirt.org* because the address
> > couldn't be found. Check for typos or unnecessary spaces and try again.
> > The response from the remote server was:
> >
> > 550 5.1.1 <jenkins(a)ovirt.org>: Recipient address rejected: User unknown
> > in local recipient table
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: Nir Soffer <nsoffer(a)redhat.com>
> > To: oVirt Jenkins <jenkins(a)ovirt.org>
> > Cc: infra(a)ovirt.org, Dan Kenigsberg <danken(a)redhat.com>
> > Bcc:
> > Date: Sat, 11 Nov 2017 00:02:32 +0000
> > Subject: Re: [CQ]: 83732,5 (vdsm) failed "ovirt-master" system tests
> > This looks like lago issue, not related to the vdsm patch:
> >
> > *00:09:11.669* [basic-suit] * Running reposync: ERROR (in 0:03:00)*00:09:11.672* [basic-suit] # Syncing remote repos locally (this might take some time): ERROR (in 0:03:00)
> >
> > <b style=
> > ----- Message truncated -----
> >
> >
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100070)
7 years
[JIRA] (OVIRT-1751) Re: Delivery Status Notification (Failure)
by Barak Korren (oVirt JIRA)
This is a multi-part message in MIME format...
------------=_1510500527-26808-118
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1751?page=com.atlassian.jir... ]
Barak Korren edited comment on OVIRT-1751 at 11/12/17 3:28 PM:
---------------------------------------------------------------
[~nsoffer(a)redhat.com] you as I suspected, that message is not for you and you should not reply to it. If real issues are found, you will get an orderly OST failure report on Devel.
was (Author: bkorren(a)redhat.com):
[~nsoffer(a)redhat.com] you as a suspected, that message is not for you and you should not reply to it, if real issues are found, you will get an orderly OST failure report on Devel.
> Re: Delivery Status Notification (Failure)
> ------------------------------------------
>
> Key: OVIRT-1751
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1751
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Nir Soffer
> Assignee: infra
> Attachments: icon.png
>
>
> Mail from jenkins(a)ovirt.org should use a reply-to header so replies go the
> proper
> mailing list (e.g. infra(a)ovirt.org), or if we don't have anything that make
> sense,
> noreply(a)ovirt.org?
> On Sat, Nov 11, 2017 at 2:02 AM Mail Delivery Subsystem <
> mailer-daemon(a)googlemail.com> wrote:
> > [image: icon.png]
> > Address not found
> > Your message wasn't delivered to *jenkins(a)ovirt.org* because the address
> > couldn't be found. Check for typos or unnecessary spaces and try again.
> > The response from the remote server was:
> >
> > 550 5.1.1 <jenkins(a)ovirt.org>: Recipient address rejected: User unknown
> > in local recipient table
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: Nir Soffer <nsoffer(a)redhat.com>
> > To: oVirt Jenkins <jenkins(a)ovirt.org>
> > Cc: infra(a)ovirt.org, Dan Kenigsberg <danken(a)redhat.com>
> > Bcc:
> > Date: Sat, 11 Nov 2017 00:02:32 +0000
> > Subject: Re: [CQ]: 83732,5 (vdsm) failed "ovirt-master" system tests
> > This looks like lago issue, not related to the vdsm patch:
> >
> > *00:09:11.669* [basic-suit] * Running reposync: ERROR (in 0:03:00)*00:09:11.672* [basic-suit] # Syncing remote repos locally (this might take some time): ERROR (in 0:03:00)
> >
> > <b style=
> > ----- Message truncated -----
> >
> >
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100070)
------------=_1510500527-26808-118
Content-Type: text/html; charset="UTF-8"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
<html><body>
<pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1751?page=com.atlassian.jir... ]</pre>
<h3>Barak Korren edited comment on OVIRT-1751 at 11/12/17 3:28 PM:</h3>
<p>[~nsoffer(a)redhat.com] you as I suspected, that message is not for you and you should not reply to it. If real issues are found, you will get an orderly OST failure report on Devel.</p>
<p>was (Author: bkorren(a)redhat.com): [~nsoffer(a)redhat.com] you as a suspected, that message is not for you and you should not reply to it, if real issues are found, you will get an orderly OST failure report on Devel.</p>
<blockquote><h3>Re: Delivery Status Notification (Failure)</h3>
<pre> Key: OVIRT-1751
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1751
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra
Attachments: icon.png</pre>
<p>Mail from jenkins(a)ovirt.org should use a reply-to header so replies go the proper mailing list (e.g. infra(a)ovirt.org), or if we don't have anything that make sense, noreply(a)ovirt.org? On Sat, Nov 11, 2017 at 2:02 AM Mail Delivery Subsystem < mailer-daemon(a)googlemail.com> wrote:</p>
<blockquote><p>[image: icon.png] Address not found Your message wasn't delivered to <strong>jenkins(a)ovirt.org</strong> because the address couldn't be found. Check for typos or unnecessary spaces and try again. The response from the remote server was:</p>
<p>550 5.1.1 <jenkins(a)ovirt.org>: Recipient address rejected: User unknown in local recipient table</p>
<p>---------- Forwarded message ---------- From: Nir Soffer <nsoffer(a)redhat.com> To: oVirt Jenkins <jenkins(a)ovirt.org> Cc: infra(a)ovirt.org, Dan Kenigsberg <danken(a)redhat.com> Bcc: Date: Sat, 11 Nov 2017 00:02:32 +0000 Subject: Re: [CQ]: 83732,5 (vdsm) failed “ovirt-master” system tests This looks like lago issue, not related to the vdsm patch:</p>
<p><strong>00:09:11.669</strong> [basic-suit] * Running reposync: ERROR (in 0:03:00)*00:09:11.672* [basic-suit] # Syncing remote repos locally (this might take some time): ERROR (in 0:03:00)</p>
<h3><b style=</h3></blockquote></blockquote>
<p>— This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100070)</p>
<img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYL..." alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/>
</body></html>
------------=_1510500527-26808-118--
7 years
[JIRA] (OVIRT-1751) Re: Delivery Status Notification (Failure)
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1751?page=com.atlassian.jir... ]
Barak Korren commented on OVIRT-1751:
-------------------------------------
[~nsoffer(a)redhat.com] you as a suspected, that message is not for you and you should not reply to it, if real issues are found, you will get an orderly OST failure report on Devel.
> Re: Delivery Status Notification (Failure)
> ------------------------------------------
>
> Key: OVIRT-1751
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1751
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Nir Soffer
> Assignee: infra
> Attachments: icon.png
>
>
> Mail from jenkins(a)ovirt.org should use a reply-to header so replies go the
> proper
> mailing list (e.g. infra(a)ovirt.org), or if we don't have anything that make
> sense,
> noreply(a)ovirt.org?
> On Sat, Nov 11, 2017 at 2:02 AM Mail Delivery Subsystem <
> mailer-daemon(a)googlemail.com> wrote:
> > [image: icon.png]
> > Address not found
> > Your message wasn't delivered to *jenkins(a)ovirt.org* because the address
> > couldn't be found. Check for typos or unnecessary spaces and try again.
> > The response from the remote server was:
> >
> > 550 5.1.1 <jenkins(a)ovirt.org>: Recipient address rejected: User unknown
> > in local recipient table
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: Nir Soffer <nsoffer(a)redhat.com>
> > To: oVirt Jenkins <jenkins(a)ovirt.org>
> > Cc: infra(a)ovirt.org, Dan Kenigsberg <danken(a)redhat.com>
> > Bcc:
> > Date: Sat, 11 Nov 2017 00:02:32 +0000
> > Subject: Re: [CQ]: 83732,5 (vdsm) failed "ovirt-master" system tests
> > This looks like lago issue, not related to the vdsm patch:
> >
> > *00:09:11.669* [basic-suit] * Running reposync: ERROR (in 0:03:00)*00:09:11.672* [basic-suit] # Syncing remote repos locally (this might take some time): ERROR (in 0:03:00)
> >
> > <b style=
> > ----- Message truncated -----
> >
> >
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100070)
7 years
