Full disk on resources.ovirt.org

[Eyal Edri]

----- Original Message -----
> From: "Alexander Rydekull" <rydekull at gmail.com>
> To: "Eyal Edri" <eedri at redhat.com>
> Cc: "Karsten 'quaid' Wade" <kwade at redhat.com>, "infra" <infra at ovirt.org>
> Sent: Wednesday, May 8, 2013 2:50:10 PM
> Subject: Re: Full disk on resources.ovirt.org
> 
> Hrm, all your "pastebin.test.redhat.com" is inaccessible by me.
> 
> 

sorry, i meant to use pastebin.com (sort of a habbit to use the internal $company one).

http://pastebin.com/k8Fa0Qye
http://pastebin.com/LECCHZ8e

> 
> 
> On Wed, May 8, 2013 at 12:04 PM, Eyal Edri <eedri at redhat.com> wrote:
> 
> > ----- Original Message -----
> > > From: "Alexander Rydekull" <rydekull at gmail.com>
> > > To: "Karsten 'quaid' Wade" <kwade at redhat.com>
> > > Cc: "infra" <infra at ovirt.org>
> > > Sent: Wednesday, May 8, 2013 11:44:31 AM
> > > Subject: Re: Full disk on resources.ovirt.org
> > >
> > > Okay, so I'm quite sick of these "full disk on linode01" issues.
> > >
> > > So, I propose the following solution:
> > >
> > > - A maintainer of jenkins publish-rpms-nightly disable the copy of
> > several
> > > versions of the each RPM (preferably only the latest, do we really need
> > > more, its a nightly build afterall)
> > > - A maintainer of jenkins publish-rpms-nightly TRY to make a link to the
> > > files instead of copying them.
> > >
> > > - Until this is done, we can clear some disk (3 GB+) by doing something
> > like
> > > the following:
> > > # cd /var/www/html/releases/nightly
> > > # for yfile in $(for xfile in $(for file in $(find . -type f -name
> > "*.rpm") ;
> > > do echo $file | cut -d"." -f 1-2 ; done | sort -u) ; do ls ${xfile}* |
> > sort
> > > -n | head -n-2 ; done) ; do echo rm $yfile ; done
> > >
> > > This will effectively remove every RPM except the last two revisions of
> > each
> > > package. (Well, obviously you need to remove that very last "echo" in the
> > > line, just putting it there not to ruins someones afternoon).
> > >
> > > - What I propose other then this is just putting the backups somewhere
> > else.
> > > We should have sufficient space for this on another host.
> > > If not, give ma thumbs up and I'll host the backups somewhere else for us
> > > until we've gotten it in order.
> > >
> > > Any thoughts?
> >
> > 1. Anyone with access to resources.ovirt.org can login and review the
> > current scripts that handle the history (found at /etc/cron.d),
> >    no need to wait for a specific maintainer to be free to handle it.
> >
> > 2. since this is becoming a recurrent issue i put here the cleaner script
> > (which i didn't wrote, so not sure if it's working properly)
> >
> > [1] http://pastebin.test.redhat.com/140876
> >
> > you're welcome to review it/fix it/change it.
> >
> > 3. as for the cleanup [2] of temp dir @ /home/jenkins/ovirt-nightly/*
> > which didn't delete the temp dir, it's fixed now.
> >
> > [2] http://pastebin.test.redhat.com/140879
> >
> > i believe there was a bug in the 'move script' ,the output - it has an
> > error on .tar.gz - http://pastebin.test.redhat.com/140881
> >
> >
> > I propose the following:
> >
> > 1. upload these scripts to review on gerrit upstream -
> > gerrit.ovirt.org/jenkins (or another git infra repo ?)
> > 2. i would move these scripts to run from jenkins rather than cron to
> > increase visibility and prompt errors
> > (since we won't allow root login from jenkins to resources, we'll need to
> > run it with jenkins user and sudo).
> > 3. move all backups from that vm to another phsical host we have
> > (currently we only have alterway01/02).
> >
> >
> >
> >
> > >
> > >
> > >
> > > On Wed, May 8, 2013 at 7:18 AM, Karsten 'quaid' Wade < kwade at redhat.com>
> > > wrote:
> > >
> > >
> > >
> > > On 05/01/2013 11:51 PM, Eyal Edri wrote:
> > >
> > > >>
> > > >> What does /jenkins do? I find lots of RPMs in sub-directories that are
> > > >> apparently dailies for the last month. Can we sweep them out to get
> > some
> > > >> breathing room?
> > > >
> > > > this is a temp place for publishing nightlies, but i think it may not
> > be
> > > > deleted probperly.
> > > > i added a remove function to the move cron script
> > 'move_jenkins_nightly' to
> > > > delete all content after it's published.
> > > > (deleting current data released 13GB).
> > >
> > > Looks like we have the same problem again - that directory structure is
> > > filled with many versions of similar files, and the disk is full again.
> > > I freed up enough space to keep Mailman running for now.
> > >
> > > --
> > > Karsten 'quaid' Wade
> > > http://TheOpenSourceWay.org .^\ http://community.redhat.com
> > > @quaid ( identi.ca/twitter/IRC ) \v' gpg: AD0E0C41
> > >
> > >
> > > _______________________________________________
> > > Infra mailing list
> > > Infra at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/infra
> > >
> > >
> > >
> > >
> > > --
> > > /Alexander Rydekull
> > >
> > > _______________________________________________
> > > Infra mailing list
> > > Infra at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/infra
> > >
> >
> 
> 
> 
> --
> /Alexander Rydekull
>