Re: Full disk on resources.ovirt.org

From: "Alexander Rydekull" <rydekull(a)gmail.com&gt; To: "Eyal Edri" <eedri(a)redhat.com&gt; Cc: "Karsten 'quaid' Wade" <kwade(a)redhat.com&gt;, "infra" <infra(a)ovirt.org&gt; Sent: Wednesday, May 8, 2013 2:50:10 PM Subject: Re: Full disk on resources.ovirt.org Hrm, all your "pastebin.test.redhat.com" is inaccessible by me.

On Wed, May 8, 2013 at 12:04 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > ----- Original Message ----- > > From: "Alexander Rydekull" <rydekull(a)gmail.com&gt; > > To: "Karsten 'quaid' Wade" <kwade(a)redhat.com&gt; > > Cc: "infra" <infra(a)ovirt.org&gt; > > Sent: Wednesday, May 8, 2013 11:44:31 AM > > Subject: Re: Full disk on resources.ovirt.org > > > > Okay, so I'm quite sick of these "full disk on linode01" issues. > > > > So, I propose the following solution: > > > > - A maintainer of jenkins publish-rpms-nightly disable the copy of > several > > versions of the each RPM (preferably only the latest, do we really need > > more, its a nightly build afterall) > > - A maintainer of jenkins publish-rpms-nightly TRY to make a link to the > > files instead of copying them. > > > > - Until this is done, we can clear some disk (3 GB+) by doing something > like > > the following: > > # cd /var/www/html/releases/nightly > > # for yfile in $(for xfile in $(for file in $(find . -type f -name > "*.rpm") ; > > do echo $file | cut -d"." -f 1-2 ; done | sort -u) ; do ls ${xfile}* | > sort > > -n | head -n-2 ; done) ; do echo rm $yfile ; done > > > > This will effectively remove every RPM except the last two revisions of > each > > package. (Well, obviously you need to remove that very last "echo" in the > > line, just putting it there not to ruins someones afternoon). > > > > - What I propose other then this is just putting the backups somewhere > else. > > We should have sufficient space for this on another host. > > If not, give ma thumbs up and I'll host the backups somewhere else for us > > until we've gotten it in order. > > > > Any thoughts? > > 1. Anyone with access to resources.ovirt.org can login and review the > current scripts that handle the history (found at /etc/cron.d), > no need to wait for a specific maintainer to be free to handle it. > > 2. since this is becoming a recurrent issue i put here the cleaner script > (which i didn't wrote, so not sure if it's working properly) > > [1] http://pastebin.test.redhat.com/140876 > > you're welcome to review it/fix it/change it. > > 3. as for the cleanup [2] of temp dir @ /home/jenkins/ovirt-nightly/* > which didn't delete the temp dir, it's fixed now. > > [2] http://pastebin.test.redhat.com/140879 > > i believe there was a bug in the 'move script' ,the output - it has an > error on .tar.gz - http://pastebin.test.redhat.com/140881 > > > I propose the following: > > 1. upload these scripts to review on gerrit upstream - > gerrit.ovirt.org/jenkins (or another git infra repo ?) > 2. i would move these scripts to run from jenkins rather than cron to > increase visibility and prompt errors > (since we won't allow root login from jenkins to resources, we'll need to > run it with jenkins user and sudo). > 3. move all backups from that vm to another phsical host we have > (currently we only have alterway01/02). > > > > > > > > > > > > On Wed, May 8, 2013 at 7:18 AM, Karsten 'quaid' Wade < kwade(a)redhat.com&gt; > > wrote: > > > > > > > > On 05/01/2013 11:51 PM, Eyal Edri wrote: > > > > >> > > >> What does /jenkins do? I find lots of RPMs in sub-directories that are > > >> apparently dailies for the last month. Can we sweep them out to get > some > > >> breathing room? > > > > > > this is a temp place for publishing nightlies, but i think it may not > be > > > deleted probperly. > > > i added a remove function to the move cron script > 'move_jenkins_nightly' to > > > delete all content after it's published. > > > (deleting current data released 13GB). > > > > Looks like we have the same problem again - that directory structure is > > filled with many versions of similar files, and the disk is full again. > > I freed up enough space to keep Mailman running for now. > > > > -- > > Karsten 'quaid' Wade > > http://TheOpenSourceWay.org .^\ http://community.redhat.com > > @quaid ( identi.ca/twitter/IRC ) \v' gpg: AD0E0C41 > > > > > > _______________________________________________ > > Infra mailing list > > Infra(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > > > > > -- > > /Alexander Rydekull > > > > _______________________________________________ > > Infra mailing list > > Infra(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > -- /Alexander Rydekull