Full disk on resources.ovirt.org
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2SXCDJNBENSTFFESVJRNU Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Just a week after I added 12 more GB to resources.ovirt.org, it's full again. Tonight looking for what is taking up that space, I'm not entirely sure. It looks as if there is more in /home than in /var, and that I think is more backups than anything. What does /jenkins do? I find lots of RPMs in sub-directories that are apparently dailies for the last month. Can we sweep them out to get some breathing room? Is there any chance we can move backup of Gerrit and Jenkins off this host? Like, immediately? I swept up some Gerrit database dumps from last week to get 53M back so I can send this email out. I notice that space is starting to fill up again. If I can't figure out what to kill that is filling the disk, I'll have to reboot. - Karsten --=20 Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 ------enig2SXCDJNBENSTFFESVJRNU Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iD8DBQFRgexm2ZIOBq0ODEERAqA7AJ0ZQXjQ27pSZNPWwJzUe3Gxv86aFQCdGt0J H8SRXhgjum2MKky7nktzyEc= =lRdO -----END PGP SIGNATURE----- ------enig2SXCDJNBENSTFFESVJRNU--
----- Original Message -----
From: "Karsten 'quaid' Wade" <kwade@redhat.com> To: "infra" <infra@ovirt.org> Sent: Thursday, May 2, 2013 7:32:38 AM Subject: Full disk on resources.ovirt.org
Just a week after I added 12 more GB to resources.ovirt.org, it's full again.
Tonight looking for what is taking up that space, I'm not entirely sure. It looks as if there is more in /home than in /var, and that I think is more backups than anything.
What does /jenkins do? I find lots of RPMs in sub-directories that are apparently dailies for the last month. Can we sweep them out to get some breathing room?
this is a temp place for publishing nightlies, but i think it may not be deleted probperly. i added a remove function to the move cron script 'move_jenkins_nightly' to delete all content after it's published. (deleting current data released 13GB).
Is there any chance we can move backup of Gerrit and Jenkins off this host? Like, immediately?
i think we can move it to either alterway1 or alterway2 baremetal servers. we need to move the backup scripts to direct there the backups.
I swept up some Gerrit database dumps from last week to get 53M back so I can send this email out. I notice that space is starting to fill up again. If I can't figure out what to kill that is filling the disk, I'll have to reboot.
- Karsten -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2DJMSCICJGDPOKOFATTSD Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 05/01/2013 11:51 PM, Eyal Edri wrote:
=20 =20 ----- Original Message -----
From: "Karsten 'quaid' Wade" <kwade@redhat.com> To: "infra" <infra@ovirt.org> Sent: Thursday, May 2, 2013 7:32:38 AM Subject: Full disk on resources.ovirt.org
Just a week after I added 12 more GB to resources.ovirt.org, it's full=
again.
Tonight looking for what is taking up that space, I'm not entirely sur= e. It looks as if there is more in /home than in /var, and that I think i= s more backups than anything.
What does /jenkins do? I find lots of RPMs in sub-directories that are=
apparently dailies for the last month. Can we sweep them out to get so= me breathing room? =20 this is a temp place for publishing nightlies, but i think it may not b= e deleted probperly. i added a remove function to the move cron script 'move_jenkins_nightly= ' to delete all content after it's published. (deleting current data released 13GB).=20
OK, that's good, but I still wonder what filled the disk yesterday? That Jenkins data has been building up over the month, I didn't see an indication that it all came in the last week. --=20 Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 ------enig2DJMSCICJGDPOKOFATTSD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iD8DBQFRgoUj2ZIOBq0ODEERArOpAKCNRSOSNMAiJcC3F/KFfAlZZPETTwCgr7wo PIUIzemMjOauxGFwV9XRAvA= =NLDn -----END PGP SIGNATURE----- ------enig2DJMSCICJGDPOKOFATTSD--
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2TDTOWNQFTGLMEGVHHSES Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 05/01/2013 11:51 PM, Eyal Edri wrote:
What does /jenkins do? I find lots of RPMs in sub-directories that are=
apparently dailies for the last month. Can we sweep them out to get so= me breathing room? =20 this is a temp place for publishing nightlies, but i think it may not b= e deleted probperly. i added a remove function to the move cron script 'move_jenkins_nightly= ' to delete all content after it's published. (deleting current data released 13GB).=20
Looks like we have the same problem again - that directory structure is filled with many versions of similar files, and the disk is full again. I freed up enough space to keep Mailman running for now. --=20 Karsten 'quaid' Wade http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 ------enig2TDTOWNQFTGLMEGVHHSES Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iD8DBQFRieA22ZIOBq0ODEERAqNXAJ42fFkEuWQf7se+c/NGqC8VI06iXwCgqilz sgQnUCt3cQGxsyQagbS2CdI= =HDtl -----END PGP SIGNATURE----- ------enig2TDTOWNQFTGLMEGVHHSES--
Okay, so I'm quite sick of these "full disk on linode01" issues. So, I propose the following solution: - A maintainer of jenkins publish-rpms-nightly disable the copy of several versions of the each RPM (preferably only the latest, do we really need more, its a nightly build afterall) - A maintainer of jenkins publish-rpms-nightly TRY to make a link to the files instead of copying them. - Until this is done, we can clear some disk (3 GB+) by doing something like the following: # cd /var/www/html/releases/nightly # for yfile in $(for xfile in $(for file in $(find . -type f -name "*.rpm") ; do echo $file | cut -d"." -f 1-2 ; done | sort -u) ; do ls ${xfile}* | sort -n | head -n-2 ; done) ; do echo rm $yfile ; done This will effectively remove every RPM except the last two revisions of each package. (Well, obviously you need to remove that very last "echo" in the line, just putting it there not to ruins someones afternoon). - What I propose other then this is just putting the backups somewhere else. We should have sufficient space for this on another host. If not, give ma thumbs up and I'll host the backups somewhere else for us until we've gotten it in order. Any thoughts? On Wed, May 8, 2013 at 7:18 AM, Karsten 'quaid' Wade <kwade@redhat.com>wrote:
On 05/01/2013 11:51 PM, Eyal Edri wrote:
What does /jenkins do? I find lots of RPMs in sub-directories that are apparently dailies for the last month. Can we sweep them out to get some breathing room?
this is a temp place for publishing nightlies, but i think it may not be deleted probperly. i added a remove function to the move cron script 'move_jenkins_nightly' to delete all content after it's published. (deleting current data released 13GB).
Looks like we have the same problem again - that directory structure is filled with many versions of similar files, and the disk is full again. I freed up enough space to keep Mailman running for now.
-- Karsten 'quaid' Wade http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- /Alexander Rydekull
----- Original Message -----
From: "Alexander Rydekull" <rydekull@gmail.com> To: "Karsten 'quaid' Wade" <kwade@redhat.com> Cc: "infra" <infra@ovirt.org> Sent: Wednesday, May 8, 2013 11:44:31 AM Subject: Re: Full disk on resources.ovirt.org
Okay, so I'm quite sick of these "full disk on linode01" issues.
So, I propose the following solution:
- A maintainer of jenkins publish-rpms-nightly disable the copy of several versions of the each RPM (preferably only the latest, do we really need more, its a nightly build afterall) - A maintainer of jenkins publish-rpms-nightly TRY to make a link to the files instead of copying them.
- Until this is done, we can clear some disk (3 GB+) by doing something like the following: # cd /var/www/html/releases/nightly # for yfile in $(for xfile in $(for file in $(find . -type f -name "*.rpm") ; do echo $file | cut -d"." -f 1-2 ; done | sort -u) ; do ls ${xfile}* | sort -n | head -n-2 ; done) ; do echo rm $yfile ; done
This will effectively remove every RPM except the last two revisions of each package. (Well, obviously you need to remove that very last "echo" in the line, just putting it there not to ruins someones afternoon).
- What I propose other then this is just putting the backups somewhere else. We should have sufficient space for this on another host. If not, give ma thumbs up and I'll host the backups somewhere else for us until we've gotten it in order.
Any thoughts?
1. Anyone with access to resources.ovirt.org can login and review the current scripts that handle the history (found at /etc/cron.d), no need to wait for a specific maintainer to be free to handle it. 2. since this is becoming a recurrent issue i put here the cleaner script (which i didn't wrote, so not sure if it's working properly) [1] http://pastebin.test.redhat.com/140876 you're welcome to review it/fix it/change it. 3. as for the cleanup [2] of temp dir @ /home/jenkins/ovirt-nightly/* which didn't delete the temp dir, it's fixed now. [2] http://pastebin.test.redhat.com/140879 i believe there was a bug in the 'move script' ,the output - it has an error on .tar.gz - http://pastebin.test.redhat.com/140881 I propose the following: 1. upload these scripts to review on gerrit upstream - gerrit.ovirt.org/jenkins (or another git infra repo ?) 2. i would move these scripts to run from jenkins rather than cron to increase visibility and prompt errors (since we won't allow root login from jenkins to resources, we'll need to run it with jenkins user and sudo). 3. move all backups from that vm to another phsical host we have (currently we only have alterway01/02).
On Wed, May 8, 2013 at 7:18 AM, Karsten 'quaid' Wade < kwade@redhat.com > wrote:
On 05/01/2013 11:51 PM, Eyal Edri wrote:
What does /jenkins do? I find lots of RPMs in sub-directories that are apparently dailies for the last month. Can we sweep them out to get some breathing room?
this is a temp place for publishing nightlies, but i think it may not be deleted probperly. i added a remove function to the move cron script 'move_jenkins_nightly' to delete all content after it's published. (deleting current data released 13GB).
Looks like we have the same problem again - that directory structure is filled with many versions of similar files, and the disk is full again. I freed up enough space to keep Mailman running for now.
-- Karsten 'quaid' Wade http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid ( identi.ca/twitter/IRC ) \v' gpg: AD0E0C41
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- /Alexander Rydekull
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Hrm, all your "pastebin.test.redhat.com" is inaccessible by me. On Wed, May 8, 2013 at 12:04 PM, Eyal Edri <eedri@redhat.com> wrote:
From: "Alexander Rydekull" <rydekull@gmail.com> To: "Karsten 'quaid' Wade" <kwade@redhat.com> Cc: "infra" <infra@ovirt.org> Sent: Wednesday, May 8, 2013 11:44:31 AM Subject: Re: Full disk on resources.ovirt.org
Okay, so I'm quite sick of these "full disk on linode01" issues.
So, I propose the following solution:
- A maintainer of jenkins publish-rpms-nightly disable the copy of several versions of the each RPM (preferably only the latest, do we really need more, its a nightly build afterall) - A maintainer of jenkins publish-rpms-nightly TRY to make a link to the files instead of copying them.
- Until this is done, we can clear some disk (3 GB+) by doing something
----- Original Message ----- like
the following: # cd /var/www/html/releases/nightly # for yfile in $(for xfile in $(for file in $(find . -type f -name "*.rpm") ; do echo $file | cut -d"." -f 1-2 ; done | sort -u) ; do ls ${xfile}* | sort -n | head -n-2 ; done) ; do echo rm $yfile ; done
This will effectively remove every RPM except the last two revisions of each package. (Well, obviously you need to remove that very last "echo" in the line, just putting it there not to ruins someones afternoon).
- What I propose other then this is just putting the backups somewhere else. We should have sufficient space for this on another host. If not, give ma thumbs up and I'll host the backups somewhere else for us until we've gotten it in order.
Any thoughts?
1. Anyone with access to resources.ovirt.org can login and review the current scripts that handle the history (found at /etc/cron.d), no need to wait for a specific maintainer to be free to handle it.
2. since this is becoming a recurrent issue i put here the cleaner script (which i didn't wrote, so not sure if it's working properly)
[1] http://pastebin.test.redhat.com/140876
you're welcome to review it/fix it/change it.
3. as for the cleanup [2] of temp dir @ /home/jenkins/ovirt-nightly/* which didn't delete the temp dir, it's fixed now.
[2] http://pastebin.test.redhat.com/140879
i believe there was a bug in the 'move script' ,the output - it has an error on .tar.gz - http://pastebin.test.redhat.com/140881
I propose the following:
1. upload these scripts to review on gerrit upstream - gerrit.ovirt.org/jenkins (or another git infra repo ?) 2. i would move these scripts to run from jenkins rather than cron to increase visibility and prompt errors (since we won't allow root login from jenkins to resources, we'll need to run it with jenkins user and sudo). 3. move all backups from that vm to another phsical host we have (currently we only have alterway01/02).
On Wed, May 8, 2013 at 7:18 AM, Karsten 'quaid' Wade < kwade@redhat.com> wrote:
On 05/01/2013 11:51 PM, Eyal Edri wrote:
What does /jenkins do? I find lots of RPMs in sub-directories that are apparently dailies for the last month. Can we sweep them out to get
some
breathing room?
this is a temp place for publishing nightlies, but i think it may not be deleted probperly. i added a remove function to the move cron script 'move_jenkins_nightly' to delete all content after it's published. (deleting current data released 13GB).
Looks like we have the same problem again - that directory structure is filled with many versions of similar files, and the disk is full again. I freed up enough space to keep Mailman running for now.
-- Karsten 'quaid' Wade http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid ( identi.ca/twitter/IRC ) \v' gpg: AD0E0C41
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- /Alexander Rydekull
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- /Alexander Rydekull
----- Original Message -----
From: "Alexander Rydekull" <rydekull@gmail.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Karsten 'quaid' Wade" <kwade@redhat.com>, "infra" <infra@ovirt.org> Sent: Wednesday, May 8, 2013 2:50:10 PM Subject: Re: Full disk on resources.ovirt.org
Hrm, all your "pastebin.test.redhat.com" is inaccessible by me.
sorry, i meant to use pastebin.com (sort of a habbit to use the internal $company one). http://pastebin.com/k8Fa0Qye http://pastebin.com/LECCHZ8e
On Wed, May 8, 2013 at 12:04 PM, Eyal Edri <eedri@redhat.com> wrote:
From: "Alexander Rydekull" <rydekull@gmail.com> To: "Karsten 'quaid' Wade" <kwade@redhat.com> Cc: "infra" <infra@ovirt.org> Sent: Wednesday, May 8, 2013 11:44:31 AM Subject: Re: Full disk on resources.ovirt.org
Okay, so I'm quite sick of these "full disk on linode01" issues.
So, I propose the following solution:
- A maintainer of jenkins publish-rpms-nightly disable the copy of several versions of the each RPM (preferably only the latest, do we really need more, its a nightly build afterall) - A maintainer of jenkins publish-rpms-nightly TRY to make a link to the files instead of copying them.
- Until this is done, we can clear some disk (3 GB+) by doing something
----- Original Message ----- like
the following: # cd /var/www/html/releases/nightly # for yfile in $(for xfile in $(for file in $(find . -type f -name "*.rpm") ; do echo $file | cut -d"." -f 1-2 ; done | sort -u) ; do ls ${xfile}* | sort -n | head -n-2 ; done) ; do echo rm $yfile ; done
This will effectively remove every RPM except the last two revisions of each package. (Well, obviously you need to remove that very last "echo" in the line, just putting it there not to ruins someones afternoon).
- What I propose other then this is just putting the backups somewhere else. We should have sufficient space for this on another host. If not, give ma thumbs up and I'll host the backups somewhere else for us until we've gotten it in order.
Any thoughts?
1. Anyone with access to resources.ovirt.org can login and review the current scripts that handle the history (found at /etc/cron.d), no need to wait for a specific maintainer to be free to handle it.
2. since this is becoming a recurrent issue i put here the cleaner script (which i didn't wrote, so not sure if it's working properly)
[1] http://pastebin.test.redhat.com/140876
you're welcome to review it/fix it/change it.
3. as for the cleanup [2] of temp dir @ /home/jenkins/ovirt-nightly/* which didn't delete the temp dir, it's fixed now.
[2] http://pastebin.test.redhat.com/140879
i believe there was a bug in the 'move script' ,the output - it has an error on .tar.gz - http://pastebin.test.redhat.com/140881
I propose the following:
1. upload these scripts to review on gerrit upstream - gerrit.ovirt.org/jenkins (or another git infra repo ?) 2. i would move these scripts to run from jenkins rather than cron to increase visibility and prompt errors (since we won't allow root login from jenkins to resources, we'll need to run it with jenkins user and sudo). 3. move all backups from that vm to another phsical host we have (currently we only have alterway01/02).
On Wed, May 8, 2013 at 7:18 AM, Karsten 'quaid' Wade < kwade@redhat.com> wrote:
On 05/01/2013 11:51 PM, Eyal Edri wrote:
What does /jenkins do? I find lots of RPMs in sub-directories that are apparently dailies for the last month. Can we sweep them out to get
some
breathing room?
this is a temp place for publishing nightlies, but i think it may not be deleted probperly. i added a remove function to the move cron script 'move_jenkins_nightly' to delete all content after it's published. (deleting current data released 13GB).
Looks like we have the same problem again - that directory structure is filled with many versions of similar files, and the disk is full again. I freed up enough space to keep Mailman running for now.
-- Karsten 'quaid' Wade http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid ( identi.ca/twitter/IRC ) \v' gpg: AD0E0C41
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- /Alexander Rydekull
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- /Alexander Rydekull
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2MUULEBIDLRFLPPKNAADK Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 05/08/2013 01:44 AM, Alexander Rydekull wrote:
Okay, so I'm quite sick of these "full disk on linode01" issues. =20 So, I propose the following solution: =20 - A maintainer of jenkins publish-rpms-nightly disable the copy of seve= ral versions of the each RPM (preferably only the latest, do we really need=
more, its a nightly build afterall) - A maintainer of jenkins publish-rpms-nightly TRY to make a link to th= e files instead of copying them. =20 - Until this is done, we can clear some disk (3 GB+) by doing something=
like the following:
# cd /var/www/html/releases/nightly # for yfile in $(for xfile in $(for file in $(find . -type f -name "*.r=
=2E.. and I just did that after it filled up again. But I suspect it's going to just reupload all those RPMs or something. I'm confused because there was suddenly 8+ GB of stuff in /var that wasn't there the other day. The /home/jenkins clean-up worked, in that it was 8 GB less in that folder tonight. I'm confused. But the below script got us back 11 GB. - Karsten pm")
; do echo $file | cut -d"." -f 1-2 ; done | sort -u) ; do ls ${xfile}* = | sort -n | head -n-2 ; done) ; do echo rm $yfile ; done =20 This will effectively remove every RPM except the last two revisions of=
each package. (Well, obviously you need to remove that very last "echo"= in the line, just putting it there not to ruins someones afternoon). =20 - What I propose other then this is just putting the backups somewhere else. We should have sufficient space for this on another host. If not, give ma thumbs up and I'll host the backups somewhere else for = us until we've gotten it in order. =20 Any thoughts? =20 =20 =20 On Wed, May 8, 2013 at 7:18 AM, Karsten 'quaid' Wade <kwade@redhat.com>= wrote: =20
On 05/01/2013 11:51 PM, Eyal Edri wrote:
What does /jenkins do? I find lots of RPMs in sub-directories that a=
re
apparently dailies for the last month. Can we sweep them out to get = some breathing room?
this is a temp place for publishing nightlies, but i think it may not= be deleted probperly. i added a remove function to the move cron script 'move_jenkins_night= ly' to delete all content after it's published. (deleting current data released 13GB).
Looks like we have the same problem again - that directory structure i= s filled with many versions of similar files, and the disk is full again= =2E I freed up enough space to keep Mailman running for now.
-- Karsten 'quaid' Wade http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
=20 =20
--=20 Karsten 'quaid' Wade http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 ------enig2MUULEBIDLRFLPPKNAADK Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iD8DBQFRjdFs2ZIOBq0ODEERAgHrAKCVm8mUqsNALMmJ1MR9Wwb2Qpnt9ACfRQmX j2J0ecFM/Ixc+Gvuzm2y97M= =6R+J -----END PGP SIGNATURE----- ------enig2MUULEBIDLRFLPPKNAADK--
i just loaded the relevant existing scripts to review on gerrit: http://gerrit.ovirt.org/#/c/14665/ i'm sure they can be changed/improved to better handle this. i propose we'll fix them and reploy them on linode so we can put this behind us. Eyal. ----- Original Message -----
From: "Karsten 'quaid' Wade" <kwade@redhat.com> To: "Alexander Rydekull" <rydekull@gmail.com> Cc: "infra" <infra@ovirt.org> Sent: Saturday, May 11, 2013 8:04:43 AM Subject: Re: Full disk on resources.ovirt.org
On 05/08/2013 01:44 AM, Alexander Rydekull wrote:
Okay, so I'm quite sick of these "full disk on linode01" issues.
So, I propose the following solution:
- A maintainer of jenkins publish-rpms-nightly disable the copy of several versions of the each RPM (preferably only the latest, do we really need more, its a nightly build afterall) - A maintainer of jenkins publish-rpms-nightly TRY to make a link to the files instead of copying them.
- Until this is done, we can clear some disk (3 GB+) by doing something like the following:
... and I just did that after it filled up again. But I suspect it's going to just reupload all those RPMs or something. I'm confused because there was suddenly 8+ GB of stuff in /var that wasn't there the other day. The /home/jenkins clean-up worked, in that it was 8 GB less in that folder tonight. I'm confused. But the below script got us back 11 GB.
- Karsten
# cd /var/www/html/releases/nightly # for yfile in $(for xfile in $(for file in $(find . -type f -name "*.rpm") ; do echo $file | cut -d"." -f 1-2 ; done | sort -u) ; do ls ${xfile}* | sort -n | head -n-2 ; done) ; do echo rm $yfile ; done
This will effectively remove every RPM except the last two revisions of each package. (Well, obviously you need to remove that very last "echo" in the line, just putting it there not to ruins someones afternoon).
- What I propose other then this is just putting the backups somewhere else. We should have sufficient space for this on another host. If not, give ma thumbs up and I'll host the backups somewhere else for us until we've gotten it in order.
Any thoughts?
On Wed, May 8, 2013 at 7:18 AM, Karsten 'quaid' Wade <kwade@redhat.com>wrote:
On 05/01/2013 11:51 PM, Eyal Edri wrote:
What does /jenkins do? I find lots of RPMs in sub-directories that are apparently dailies for the last month. Can we sweep them out to get some breathing room?
this is a temp place for publishing nightlies, but i think it may not be deleted probperly. i added a remove function to the move cron script 'move_jenkins_nightly' to delete all content after it's published. (deleting current data released 13GB).
Looks like we have the same problem again - that directory structure is filled with many versions of similar files, and the disk is full again. I freed up enough space to keep Mailman running for now.
-- Karsten 'quaid' Wade http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Karsten 'quaid' Wade http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
participants (3)
-
Alexander Rydekull -
Eyal Edri -
Karsten 'quaid' Wade