Full disk on resources.ovirt.org

[Eyal Edri]

----- Original Message -----
> From: "Alexander Rydekull" <rydekull at gmail.com>
> To: "Karsten 'quaid' Wade" <kwade at redhat.com>
> Cc: "infra" <infra at ovirt.org>
> Sent: Wednesday, May 8, 2013 11:44:31 AM
> Subject: Re: Full disk on resources.ovirt.org
> 
> Okay, so I'm quite sick of these "full disk on linode01" issues.
> 
> So, I propose the following solution:
> 
> - A maintainer of jenkins publish-rpms-nightly disable the copy of several
> versions of the each RPM (preferably only the latest, do we really need
> more, its a nightly build afterall)
> - A maintainer of jenkins publish-rpms-nightly TRY to make a link to the
> files instead of copying them.
> 
> - Until this is done, we can clear some disk (3 GB+) by doing something like
> the following:
> # cd /var/www/html/releases/nightly
> # for yfile in $(for xfile in $(for file in $(find . -type f -name "*.rpm") ;
> do echo $file | cut -d"." -f 1-2 ; done | sort -u) ; do ls ${xfile}* | sort
> -n | head -n-2 ; done) ; do echo rm $yfile ; done
> 
> This will effectively remove every RPM except the last two revisions of each
> package. (Well, obviously you need to remove that very last "echo" in the
> line, just putting it there not to ruins someones afternoon).
> 
> - What I propose other then this is just putting the backups somewhere else.
> We should have sufficient space for this on another host.
> If not, give ma thumbs up and I'll host the backups somewhere else for us
> until we've gotten it in order.
> 
> Any thoughts?

1. Anyone with access to resources.ovirt.org can login and review the current scripts that handle the history (found at /etc/cron.d),
   no need to wait for a specific maintainer to be free to handle it. 

2. since this is becoming a recurrent issue i put here the cleaner script (which i didn't wrote, so not sure if it's working properly)

[1] http://pastebin.test.redhat.com/140876

you're welcome to review it/fix it/change it.

3. as for the cleanup [2] of temp dir @ /home/jenkins/ovirt-nightly/* 
which didn't delete the temp dir, it's fixed now.

[2] http://pastebin.test.redhat.com/140879

i believe there was a bug in the 'move script' ,the output - it has an error on .tar.gz - http://pastebin.test.redhat.com/140881


I propose the following:

1. upload these scripts to review on gerrit upstream - gerrit.ovirt.org/jenkins (or another git infra repo ?)
2. i would move these scripts to run from jenkins rather than cron to increase visibility and prompt errors 
(since we won't allow root login from jenkins to resources, we'll need to run it with jenkins user and sudo).
3. move all backups from that vm to another phsical host we have (currently we only have alterway01/02).




> 
> 
> 
> On Wed, May 8, 2013 at 7:18 AM, Karsten 'quaid' Wade < kwade at redhat.com >
> wrote:
> 
> 
> 
> On 05/01/2013 11:51 PM, Eyal Edri wrote:
> 
> >> 
> >> What does /jenkins do? I find lots of RPMs in sub-directories that are
> >> apparently dailies for the last month. Can we sweep them out to get some
> >> breathing room?
> > 
> > this is a temp place for publishing nightlies, but i think it may not be
> > deleted probperly.
> > i added a remove function to the move cron script 'move_jenkins_nightly' to
> > delete all content after it's published.
> > (deleting current data released 13GB).
> 
> Looks like we have the same problem again - that directory structure is
> filled with many versions of similar files, and the disk is full again.
> I freed up enough space to keep Mailman running for now.
> 
> --
> Karsten 'quaid' Wade
> http://TheOpenSourceWay.org .^\ http://community.redhat.com
> @quaid ( identi.ca/twitter/IRC ) \v' gpg: AD0E0C41
> 
> 
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
> 
> 
> 
> 
> --
> /Alexander Rydekull
> 
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>