1. daily hangs since upgrade to gerrit 2.3.
2. asked eedri to change versions of jenkins gerrit plugin per
3. instance seems busy all the time cpu wise.
4. changed instance size to bigger to rule out load issues.
this is costly, so will track load to see if can go back to previous
5. git clone/pull stopped working post reboot complaining on repo
6. fixed by changing ulimit
7. need to continue tracking for hangs
8. need to consider weekly git repack cron job
9. in a happier note - Gal's patches to improve gerrit emails were
merged yesterday to gerrit upstream!
more details below.
looking at gerrit behavior/hangs since the upgrade to gerrit 2.3 on
sunday, it seems the host is busy cpu wise.
just to rule out heavier load of users/jenkins coinciding with the
upgrade, i started by changing the server config to a bigger machine.
however, post the reboot git pull/clone stopped working for
git://gerrit.ovirt.org/ovirt-engine and vdsm
checking /var/log/messages I saw a git corruption message:
git-daemon: fatal: object e1866a3a1aaefdc5a0537b83b193ac1065d938e6
other repo's worked.
however, git pull/clone for http://gerrit.ovirt.org/ovirt-engine did work!
(small sigh of relief)
more important, git fsck did not complain on corruption
(bigger sigh of relief)
after a bit of research, this is related to limit on number of file
descriptors, where git-daemon is more limited by default than jgit is.
better explanation here:
(and if this didn't happen just because i rebooted/upgraded, i would
have probably remembered earlier we already fixed this some time back on
another gerrit a few months back).
On 06/04/2012 07:19 PM, Adam Litke wrote:
> We are currently in the process of integrating MOM with vdsm. Today the MOM
> code is hosted on github and the mailing lists are on Google Groups. Since the
> bulk of MOM development activity surrounds oVirt I would like to ask if oVirt
> would be willing to host it in gerrit and provide some mailing lists. Doing
> this will help interlock the development of these two projects. If there is
> affirmative agreement, I would request the following be created:
> Gerrit project / repo: mom
> - Please add aglitke<agl(a)us.ibm.com> as the project maintainer/admin
since mom was defined as ovirt incubation, don't see any issue with this.
if you need help pushing the initial git - ping me.
> Mailing lists:
> - mom-devel: Developement discussions for MOM
> - mom-patches: To be used for gerrit status notifications (ala vdsm-patches)
mailing list I think karsten needs to add (and you need to login with
mom-patches to register it to gerrit for mom patches to be sent to it
like a regular user.
> Thanks for your consideration.
Since we're starting to run & build on fedora 17,
Itamar added a new EC2 vm that runs fedora 17, so i've moved the ovirt-engine rpm build to it.
mike, do you need the the f16 for ovirt-node jobs?
If we can migrate all the jobs to use the f17 vm, i believe we can remove the f16 to save costs.
i'm also trying to add a new physical host as a slave, hopefully it will be ready today or tomorrow.
I'm planning to upgrade gerrit to version 2.3 on Sunday.
will send an email before and after the downtime.
plan is to upgrade to "clean" version of 2.3, and after we see
everything is working ok, upgrade it with the patch for better emails.