From: "Karsten 'quaid' Wade" <kwade(a)redhat.com&gt; To: "infra" <infra(a)ovirt.org&gt; Sent: Thursday, May 2, 2013 7:32:38 AM Subject: Full disk on resources.ovirt.org Just a week after I added 12 more GB to resources.ovirt.org, it's full again. Tonight looking for what is taking up that space, I'm not entirely sure. It looks as if there is more in /home than in /var, and that I think is more backups than anything. What does /jenkins do? I find lots of RPMs in sub-directories that are apparently dailies for the last month. Can we sweep them out to get some breathing room?

Is there any chance we can move backup of Gerrit and Jenkins off this host? Like, immediately?

I swept up some Gerrit database dumps from last week to get 53M back so I can send this email out. I notice that space is starting to fill up again. If I can't figure out what to kill that is filling the disk, I'll have to reboot. - Karsten -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 _______________________________________________ Infra mailing list Infra(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

> > What does /jenkins do? I find lots of RPMs in sub-directories that are=

> apparently dailies for the last month. Can we sweep them out to get so=

> breathing room? =20 this is a temp place for publishing nightlies, but i think it may not b=

i added a remove function to the move cron script 'move_jenkins_nightly=

(deleting current data released 13GB).=20

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iD8DBQFRieA22ZIOBq0ODEERAqNXAJ42fFkEuWQf7se+c/NGqC8VI06iXwCgqilz sgQnUCt3cQGxsyQagbS2CdI= =HDtl -----END PGP SIGNATURE-----

From: "Alexander Rydekull" <rydekull(a)gmail.com&gt; To: "Karsten 'quaid' Wade" <kwade(a)redhat.com&gt; Cc: "infra" <infra(a)ovirt.org&gt; Sent: Wednesday, May 8, 2013 11:44:31 AM Subject: Re: Full disk on resources.ovirt.org Okay, so I'm quite sick of these "full disk on linode01" issues. So, I propose the following solution: - A maintainer of jenkins publish-rpms-nightly disable the copy of several versions of the each RPM (preferably only the latest, do we really need more, its a nightly build afterall) - A maintainer of jenkins publish-rpms-nightly TRY to make a link to the files instead of copying them. - Until this is done, we can clear some disk (3 GB+) by doing something like the following: # cd /var/www/html/releases/nightly # for yfile in $(for xfile in $(for file in $(find . -type f -name "*.rpm") ; do echo $file | cut -d"." -f 1-2 ; done | sort -u) ; do ls ${xfile}* | sort -n | head -n-2 ; done) ; do echo rm $yfile ; done This will effectively remove every RPM except the last two revisions of each package. (Well, obviously you need to remove that very last "echo" in the line, just putting it there not to ruins someones afternoon). - What I propose other then this is just putting the backups somewhere else. We should have sufficient space for this on another host. If not, give ma thumbs up and I'll host the backups somewhere else for us until we've gotten it in order. Any thoughts?

On Wed, May 8, 2013 at 7:18 AM, Karsten 'quaid' Wade < kwade(a)redhat.com > wrote: On 05/01/2013 11:51 PM, Eyal Edri wrote: >> >> What does /jenkins do? I find lots of RPMs in sub-directories that are >> apparently dailies for the last month. Can we sweep them out to get some >> breathing room? > > this is a temp place for publishing nightlies, but i think it may not be > deleted probperly. > i added a remove function to the move cron script 'move_jenkins_nightly' to > delete all content after it's published. > (deleting current data released 13GB). Looks like we have the same problem again - that directory structure is filled with many versions of similar files, and the disk is full again. I freed up enough space to keep Mailman running for now. -- Karsten 'quaid' Wade http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid ( identi.ca/twitter/IRC ) \v' gpg: AD0E0C41 _______________________________________________ Infra mailing list Infra(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- /Alexander Rydekull _______________________________________________ Infra mailing list Infra(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

From: "Alexander Rydekull" <rydekull(a)gmail.com&gt; To: "Eyal Edri" <eedri(a)redhat.com&gt; Cc: "Karsten 'quaid' Wade" <kwade(a)redhat.com&gt;, "infra" <infra(a)ovirt.org&gt; Sent: Wednesday, May 8, 2013 2:50:10 PM Subject: Re: Full disk on resources.ovirt.org Hrm, all your "pastebin.test.redhat.com" is inaccessible by me.

On Wed, May 8, 2013 at 12:04 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > ----- Original Message ----- > > From: "Alexander Rydekull" <rydekull(a)gmail.com&gt; > > To: "Karsten 'quaid' Wade" <kwade(a)redhat.com&gt; > > Cc: "infra" <infra(a)ovirt.org&gt; > > Sent: Wednesday, May 8, 2013 11:44:31 AM > > Subject: Re: Full disk on resources.ovirt.org > > > > Okay, so I'm quite sick of these "full disk on linode01" issues. > > > > So, I propose the following solution: > > > > - A maintainer of jenkins publish-rpms-nightly disable the copy of > several > > versions of the each RPM (preferably only the latest, do we really need > > more, its a nightly build afterall) > > - A maintainer of jenkins publish-rpms-nightly TRY to make a link to the > > files instead of copying them. > > > > - Until this is done, we can clear some disk (3 GB+) by doing something > like > > the following: > > # cd /var/www/html/releases/nightly > > # for yfile in $(for xfile in $(for file in $(find . -type f -name > "*.rpm") ; > > do echo $file | cut -d"." -f 1-2 ; done | sort -u) ; do ls ${xfile}* | > sort > > -n | head -n-2 ; done) ; do echo rm $yfile ; done > > > > This will effectively remove every RPM except the last two revisions of > each > > package. (Well, obviously you need to remove that very last "echo" in the > > line, just putting it there not to ruins someones afternoon). > > > > - What I propose other then this is just putting the backups somewhere > else. > > We should have sufficient space for this on another host. > > If not, give ma thumbs up and I'll host the backups somewhere else for us > > until we've gotten it in order. > > > > Any thoughts? > > 1. Anyone with access to resources.ovirt.org can login and review the > current scripts that handle the history (found at /etc/cron.d), > no need to wait for a specific maintainer to be free to handle it. > > 2. since this is becoming a recurrent issue i put here the cleaner script > (which i didn't wrote, so not sure if it's working properly) > > [1] http://pastebin.test.redhat.com/140876 > > you're welcome to review it/fix it/change it. > > 3. as for the cleanup [2] of temp dir @ /home/jenkins/ovirt-nightly/* > which didn't delete the temp dir, it's fixed now. > > [2] http://pastebin.test.redhat.com/140879 > > i believe there was a bug in the 'move script' ,the output - it has an > error on .tar.gz - http://pastebin.test.redhat.com/140881 > > > I propose the following: > > 1. upload these scripts to review on gerrit upstream - > gerrit.ovirt.org/jenkins (or another git infra repo ?) > 2. i would move these scripts to run from jenkins rather than cron to > increase visibility and prompt errors > (since we won't allow root login from jenkins to resources, we'll need to > run it with jenkins user and sudo). > 3. move all backups from that vm to another phsical host we have > (currently we only have alterway01/02). > > > > > > > > > > > > On Wed, May 8, 2013 at 7:18 AM, Karsten 'quaid' Wade < kwade(a)redhat.com&gt; > > wrote: > > > > > > > > On 05/01/2013 11:51 PM, Eyal Edri wrote: > > > > >> > > >> What does /jenkins do? I find lots of RPMs in sub-directories that are > > >> apparently dailies for the last month. Can we sweep them out to get > some > > >> breathing room? > > > > > > this is a temp place for publishing nightlies, but i think it may not > be > > > deleted probperly. > > > i added a remove function to the move cron script > 'move_jenkins_nightly' to > > > delete all content after it's published. > > > (deleting current data released 13GB). > > > > Looks like we have the same problem again - that directory structure is > > filled with many versions of similar files, and the disk is full again. > > I freed up enough space to keep Mailman running for now. > > > > -- > > Karsten 'quaid' Wade > > http://TheOpenSourceWay.org .^\ http://community.redhat.com > > @quaid ( identi.ca/twitter/IRC ) \v' gpg: AD0E0C41 > > > > > > _______________________________________________ > > Infra mailing list > > Infra(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > > > > > -- > > /Alexander Rydekull > > > > _______________________________________________ > > Infra mailing list > > Infra(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > -- /Alexander Rydekull

From: "Karsten 'quaid' Wade" <kwade(a)redhat.com&gt; To: "Alexander Rydekull" <rydekull(a)gmail.com&gt; Cc: "infra" <infra(a)ovirt.org&gt; Sent: Saturday, May 11, 2013 8:04:43 AM Subject: Re: Full disk on resources.ovirt.org On 05/08/2013 01:44 AM, Alexander Rydekull wrote: > Okay, so I'm quite sick of these "full disk on linode01" issues. > > So, I propose the following solution: > > - A maintainer of jenkins publish-rpms-nightly disable the copy of several > versions of the each RPM (preferably only the latest, do we really need > more, its a nightly build afterall) > - A maintainer of jenkins publish-rpms-nightly TRY to make a link to the > files instead of copying them. > > - Until this is done, we can clear some disk (3 GB+) by doing something > like the following: ... and I just did that after it filled up again. But I suspect it's going to just reupload all those RPMs or something. I'm confused because there was suddenly 8+ GB of stuff in /var that wasn't there the other day. The /home/jenkins clean-up worked, in that it was 8 GB less in that folder tonight. I'm confused. But the below script got us back 11 GB. - Karsten > # cd /var/www/html/releases/nightly > # for yfile in $(for xfile in $(for file in $(find . -type f -name "*.rpm") > ; do echo $file | cut -d"." -f 1-2 ; done | sort -u) ; do ls ${xfile}* | > sort -n | head -n-2 ; done) ; do echo rm $yfile ; done > > This will effectively remove every RPM except the last two revisions of > each package. (Well, obviously you need to remove that very last "echo" in > the line, just putting it there not to ruins someones afternoon). > > - What I propose other then this is just putting the backups somewhere > else. > We should have sufficient space for this on another host. > If not, give ma thumbs up and I'll host the backups somewhere else for us > until we've gotten it in order. > > Any thoughts? > > > > On Wed, May 8, 2013 at 7:18 AM, Karsten 'quaid' Wade > <kwade(a)redhat.com&gt;wrote: > >> On 05/01/2013 11:51 PM, Eyal Edri wrote: >> >>>> >>>> What does /jenkins do? I find lots of RPMs in sub-directories that are >>>> apparently dailies for the last month. Can we sweep them out to get some >>>> breathing room? >>> >>> this is a temp place for publishing nightlies, but i think it may not be >> deleted probperly. >>> i added a remove function to the move cron script 'move_jenkins_nightly' >> to delete all content after it's published. >>> (deleting current data released 13GB). >> >> Looks like we have the same problem again - that directory structure is >> filled with many versions of similar files, and the disk is full again. >> I freed up enough space to keep Mailman running for now. >> >> -- >> Karsten 'quaid' Wade >> http://TheOpenSourceWay.org .^\ http://community.redhat.com >> @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 >> >> >> _______________________________________________ >> Infra mailing list >> Infra(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/infra >> >> > > -- Karsten 'quaid' Wade http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 _______________________________________________ Infra mailing list Infra(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/infra