how to get files from jenkins.ovirt.org to www.ovirt.org
Hi, after looking into the process and testing it via jenkins.ovirt.info, here is my proposal: 1. Nightly job [1] will collect rpms from all ovirt projects (other jenkins jobs that build rpms per commit). 1.1 all files will be kept in a flat dir under $WORKSPCE/rpms 1.2 using 'publish over ssh' [2] plugin the job will then copy files to their destination according to their names. [3] for e.g 'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch 'ovirt-engine-3.1.0-3.fc17.src.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src 'ovirt-engine-3.1.0-3.bz2' ==> ovirt.org/var/www/html/releases/nightly/src' 1.3 publish ssh plugin can execute a command after its done copying the file, so all is left is to run 'createrepo' on the relevant dirs. [4] 2. a separate cron cleaning script needs to run on ovirt.org to delete old rpms and run createrepo. +1 for pushing this forward or comments? Eyal. [1] http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/, this job currently works only for fedora17 rpms (that's what we have now). we'll need to improve it to be more generic (maybe as a matrix job) or duplicate it for each opertating system. [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH+Plugin, using pub/private key with user jenkins to connect [3] jenkins user will need r/q permissions to the nightly folder [4] will need sudo access for it.
On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote:
Hi,
after looking into the process and testing it via jenkins.ovirt.info, here is my proposal:
1. Nightly job [1] will collect rpms from all ovirt projects (other jenkins jobs that build rpms per commit). 1.1 all files will be kept in a flat dir under $WORKSPCE/rpms 1.2 using 'publish over ssh' [2] plugin the job will then copy files to their destination according to their names. [3] for e.g 'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch 'ovirt-engine-3.1.0-3.fc17.src.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src 'ovirt-engine-3.1.0-3.bz2' ==> ovirt.org/var/www/html/releases/nightly/src' 1.3 publish ssh plugin can execute a command after its done copying the file, so all is left is to run 'createrepo' on the relevant dirs. [4]
2. a separate cron cleaning script needs to run on ovirt.org to delete old rpms and run createrepo.
Maybe upload to a non-published directory first, like /home/jenkins/nightly, and have the cron job handle cleanup of old rpms, adding of new rpms, and running createrepo? If not, how is the cron job going to cleanup old rpms? Uploading to a static location that isn't public (and is under /home/jenkins) would make [3] and [4] unnecessary. I think I'd rather avoid having a non-user with sudo and r/w on a publicly served folder. The publish ssh plugin can touch a file /home/jenkins/nightly/FINISHED when it's done, the cron job locally on ovirt.org can watch for that file and then process the directory structure. Mike
+1 for pushing this forward or comments?
Eyal.
[1] http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/, this job currently works only for fedora17 rpms (that's what we have now). we'll need to improve it to be more generic (maybe as a matrix job) or duplicate it for each opertating system. [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH +Plugin, using pub/private key with user jenkins to connect [3] jenkins user will need r/q permissions to the nightly folder [4] will need sudo access for it. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
----- Original Message -----
From: "Mike Burns" <mburns@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 31, 2012 3:33:01 PM Subject: Re: how to get files from jenkins.ovirt.org to www.ovirt.org
On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote:
Hi,
after looking into the process and testing it via jenkins.ovirt.info, here is my proposal:
1. Nightly job [1] will collect rpms from all ovirt projects (other jenkins jobs that build rpms per commit). 1.1 all files will be kept in a flat dir under $WORKSPCE/rpms 1.2 using 'publish over ssh' [2] plugin the job will then copy files to their destination according to their names. [3] for e.g 'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch 'ovirt-engine-3.1.0-3.fc17.src.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src 'ovirt-engine-3.1.0-3.bz2' ==> ovirt.org/var/www/html/releases/nightly/src' 1.3 publish ssh plugin can execute a command after its done copying the file, so all is left is to run 'createrepo' on the relevant dirs. [4]
2. a separate cron cleaning script needs to run on ovirt.org to delete old rpms and run createrepo.
Maybe upload to a non-published directory first, like /home/jenkins/nightly, and have the cron job handle cleanup of old rpms, adding of new rpms, and running createrepo?
sure, i've got not problem with that.. there is one thing i don't understand, and sorry if not an expert in yum: we say we want to keep history for X builds, will all versions be kept under one repository? so 'createrepo' will run on all versions in that directory? so running yum install file.verX will work even if the repo has a x+1 version?
If not, how is the cron job going to cleanup old rpms?
Uploading to a static location that isn't public (and is under /home/jenkins) would make [3] and [4] unnecessary. I think I'd rather avoid having a non-user with sudo and r/w on a publicly served folder.
sounds right.
The publish ssh plugin can touch a file /home/jenkins/nightly/FINISHED when it's done, the cron job locally on ovirt.org can watch for that file and then process the directory structure.
good idea, we can have the cronjob monitor the dir for the FINISHED file, and delete it when it's done.
Mike
+1 for pushing this forward or comments?
Eyal.
[1] http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/, this job currently works only for fedora17 rpms (that's what we have now). we'll need to improve it to be more generic (maybe as a matrix job) or duplicate it for each opertating system. [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH +Plugin, using pub/private key with user jenkins to connect [3] jenkins user will need r/q permissions to the nightly folder [4] will need sudo access for it. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
On 07/31/2012 08:51 AM, Eyal Edri wrote:
----- Original Message -----
From: "Mike Burns" <mburns@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 31, 2012 3:33:01 PM Subject: Re: how to get files from jenkins.ovirt.org to www.ovirt.org
On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote:
Hi,
after looking into the process and testing it via jenkins.ovirt.info, here is my proposal:
1. Nightly job [1] will collect rpms from all ovirt projects (other jenkins jobs that build rpms per commit). 1.1 all files will be kept in a flat dir under $WORKSPCE/rpms 1.2 using 'publish over ssh' [2] plugin the job will then copy files to their destination according to their names. [3] for e.g 'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch 'ovirt-engine-3.1.0-3.fc17.src.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src 'ovirt-engine-3.1.0-3.bz2' ==> ovirt.org/var/www/html/releases/nightly/src' 1.3 publish ssh plugin can execute a command after its done copying the file, so all is left is to run 'createrepo' on the relevant dirs. [4]
2. a separate cron cleaning script needs to run on ovirt.org to delete old rpms and run createrepo.
Maybe upload to a non-published directory first, like /home/jenkins/nightly, and have the cron job handle cleanup of old rpms, adding of new rpms, and running createrepo? sure, i've got not problem with that..
there is one thing i don't understand, and sorry if not an expert in yum:
we say we want to keep history for X builds, will all versions be kept under one repository? so 'createrepo' will run on all versions in that directory?
so running yum install file.verX will work even if the repo has a x+1 version?
If you look at existing repo's many of them will include old versions of the package yum / create repo will simple use the latest version in the repo that meets all the requirements. Example when we first moved from vdsm 4.9 to 4.10 the repo's got really screwy until all the packages were rebuilt. In that case the older 4.9 builds would get used until all the packages were in place then the newer versions would get used.
If not, how is the cron job going to cleanup old rpms?
Uploading to a static location that isn't public (and is under /home/jenkins) would make [3] and [4] unnecessary. I think I'd rather avoid having a non-user with sudo and r/w on a publicly served folder.
sounds right.
The publish ssh plugin can touch a file /home/jenkins/nightly/FINISHED when it's done, the cron job locally on ovirt.org can watch for that file and then process the directory structure.
+1 This is the approach I was thinking about.
good idea, we can have the cronjob monitor the dir for the FINISHED file, and delete it when it's done.
Mike
+1 for pushing this forward or comments?
Eyal.
[1] http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/, this job currently works only for fedora17 rpms (that's what we have now). we'll need to improve it to be more generic (maybe as a matrix job) or duplicate it for each opertating system. [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH +Plugin, using pub/private key with user jenkins to connect [3] jenkins user will need r/q permissions to the nightly folder [4] will need sudo access for it. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Thanks Robert Middleswarth @rmiddle (twitter/IRC)
On Tue, 2012-07-31 at 09:13 -0400, Robert Middleswarth wrote:
On 07/31/2012 08:51 AM, Eyal Edri wrote:
----- Original Message -----
From: "Mike Burns" <mburns@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 31, 2012 3:33:01 PM Subject: Re: how to get files from jenkins.ovirt.org to www.ovirt.org
On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote:
Hi,
after looking into the process and testing it via jenkins.ovirt.info, here is my proposal:
1. Nightly job [1] will collect rpms from all ovirt projects (other jenkins jobs that build rpms per commit). 1.1 all files will be kept in a flat dir under $WORKSPCE/rpms 1.2 using 'publish over ssh' [2] plugin the job will then copy files to their destination according to their names. [3] for e.g 'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch 'ovirt-engine-3.1.0-3.fc17.src.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src 'ovirt-engine-3.1.0-3.bz2' ==> ovirt.org/var/www/html/releases/nightly/src' 1.3 publish ssh plugin can execute a command after its done copying the file, so all is left is to run 'createrepo' on the relevant dirs. [4]
2. a separate cron cleaning script needs to run on ovirt.org to delete old rpms and run createrepo.
Maybe upload to a non-published directory first, like /home/jenkins/nightly, and have the cron job handle cleanup of old rpms, adding of new rpms, and running createrepo? sure, i've got not problem with that..
there is one thing i don't understand, and sorry if not an expert in yum:
we say we want to keep history for X builds, will all versions be kept under one repository? so 'createrepo' will run on all versions in that directory?
so running yum install file.verX will work even if the repo has a x+1 version?
If you look at existing repo's many of them will include old versions of the package yum / create repo will simple use the latest version in the repo that meets all the requirements. Example when we first moved from vdsm 4.9 to 4.10 the repo's got really screwy until all the packages were rebuilt. In that case the older 4.9 builds would get used until all the packages were in place then the newer versions would get used.
If we still want to keep multiple night's versions, then we can have the cron job run and create a new directory called: cd /var/www/html/releases mkdir nightly-$(date +%Y%m%d) # move all content from /home/jenkins to new nightly-<date> directory ln -snf nightly-$(date +%Y%m%d) nightly rm -rf nightly-$(date -d "3 days ago" +%Y%m%d) Mike
If not, how is the cron job going to cleanup old rpms?
Uploading to a static location that isn't public (and is under /home/jenkins) would make [3] and [4] unnecessary. I think I'd rather avoid having a non-user with sudo and r/w on a publicly served folder.
sounds right.
The publish ssh plugin can touch a file /home/jenkins/nightly/FINISHED when it's done, the cron job locally on ovirt.org can watch for that file and then process the directory structure.
+1 This is the approach I was thinking about.
good idea, we can have the cronjob monitor the dir for the FINISHED file, and delete it when it's done.
Mike
+1 for pushing this forward or comments?
Eyal.
[1] http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/, this job currently works only for fedora17 rpms (that's what we have now). we'll need to improve it to be more generic (maybe as a matrix job) or duplicate it for each opertating system. [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH +Plugin, using pub/private key with user jenkins to connect [3] jenkins user will need r/q permissions to the nightly folder [4] will need sudo access for it. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
On 07/31/2012 11:03 AM, Mike Burns wrote:
On Tue, 2012-07-31 at 09:13 -0400, Robert Middleswarth wrote:
----- Original Message -----
From: "Mike Burns" <mburns@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 31, 2012 3:33:01 PM Subject: Re: how to get files from jenkins.ovirt.org to www.ovirt.org
On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote:
Hi,
after looking into the process and testing it via jenkins.ovirt.info, here is my proposal:
1. Nightly job [1] will collect rpms from all ovirt projects (other jenkins jobs that build rpms per commit). 1.1 all files will be kept in a flat dir under $WORKSPCE/rpms 1.2 using 'publish over ssh' [2] plugin the job will then copy files to their destination according to their names. [3] for e.g 'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch 'ovirt-engine-3.1.0-3.fc17.src.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src 'ovirt-engine-3.1.0-3.bz2' ==> ovirt.org/var/www/html/releases/nightly/src' 1.3 publish ssh plugin can execute a command after its done copying the file, so all is left is to run 'createrepo' on the relevant dirs. [4]
2. a separate cron cleaning script needs to run on ovirt.org to delete old rpms and run createrepo.
Maybe upload to a non-published directory first, like /home/jenkins/nightly, and have the cron job handle cleanup of old rpms, adding of new rpms, and running createrepo? sure, i've got not problem with that..
there is one thing i don't understand, and sorry if not an expert in yum:
we say we want to keep history for X builds, will all versions be kept under one repository? so 'createrepo' will run on all versions in that directory?
so running yum install file.verX will work even if the repo has a x+1 version? If you look at existing repo's many of them will include old versions of
On 07/31/2012 08:51 AM, Eyal Edri wrote: the package yum / create repo will simple use the latest version in the repo that meets all the requirements. Example when we first moved from vdsm 4.9 to 4.10 the repo's got really screwy until all the packages were rebuilt. In that case the older 4.9 builds would get used until all the packages were in place then the newer versions would get used.
If we still want to keep multiple night's versions, then we can have the cron job run and create a new directory called:
cd /var/www/html/releases mkdir nightly-$(date +%Y%m%d) # move all content from /home/jenkins to new nightly-<date> directory ln -snf nightly-$(date +%Y%m%d) nightly rm -rf nightly-$(date -d "3 days ago" +%Y%m%d)
Mike Why recreate the wheel. www.hyperdrifter.com/software/tidy_rpm_cache.html
Example usages: tidy-rpm-cache.py --dir=/tmp/packages --num-obsolete=3 Thanks Robert
If not, how is the cron job going to cleanup old rpms?
Uploading to a static location that isn't public (and is under /home/jenkins) would make [3] and [4] unnecessary. I think I'd rather avoid having a non-user with sudo and r/w on a publicly served folder.
sounds right.
The publish ssh plugin can touch a file /home/jenkins/nightly/FINISHED when it's done, the cron job locally on ovirt.org can watch for that file and then process the directory structure. +1 This is the approach I was thinking about. good idea, we can have the cronjob monitor the dir for the FINISHED file, and delete it when it's done.
Mike
+1 for pushing this forward or comments?
Eyal.
[1] http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/, this job currently works only for fedora17 rpms (that's what we have now). we'll need to improve it to be more generic (maybe as a matrix job) or duplicate it for each opertating system. [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH +Plugin, using pub/private key with user jenkins to connect [3] jenkins user will need r/q permissions to the nightly folder [4] will need sudo access for it. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Thanks Robert Middleswarth @rmiddle (twitter/IRC)
On Tue, 2012-07-31 at 14:02 -0400, Robert Middleswarth wrote:
On 07/31/2012 11:03 AM, Mike Burns wrote:
On Tue, 2012-07-31 at 09:13 -0400, Robert Middleswarth wrote:
----- Original Message -----
From: "Mike Burns" <mburns@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 31, 2012 3:33:01 PM Subject: Re: how to get files from jenkins.ovirt.org to www.ovirt.org
On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote:
Hi,
after looking into the process and testing it via jenkins.ovirt.info, here is my proposal:
1. Nightly job [1] will collect rpms from all ovirt projects (other jenkins jobs that build rpms per commit). 1.1 all files will be kept in a flat dir under $WORKSPCE/rpms 1.2 using 'publish over ssh' [2] plugin the job will then copy files to their destination according to their names. [3] for e.g 'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch 'ovirt-engine-3.1.0-3.fc17.src.rpm' ==> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src 'ovirt-engine-3.1.0-3.bz2' ==> ovirt.org/var/www/html/releases/nightly/src' 1.3 publish ssh plugin can execute a command after its done copying the file, so all is left is to run 'createrepo' on the relevant dirs. [4]
2. a separate cron cleaning script needs to run on ovirt.org to delete old rpms and run createrepo.
Maybe upload to a non-published directory first, like /home/jenkins/nightly, and have the cron job handle cleanup of old rpms, adding of new rpms, and running createrepo? sure, i've got not problem with that..
there is one thing i don't understand, and sorry if not an expert in yum:
we say we want to keep history for X builds, will all versions be kept under one repository? so 'createrepo' will run on all versions in that directory?
so running yum install file.verX will work even if the repo has a x+1 version? If you look at existing repo's many of them will include old versions of
On 07/31/2012 08:51 AM, Eyal Edri wrote: the package yum / create repo will simple use the latest version in the repo that meets all the requirements. Example when we first moved from vdsm 4.9 to 4.10 the repo's got really screwy until all the packages were rebuilt. In that case the older 4.9 builds would get used until all the packages were in place then the newer versions would get used.
If we still want to keep multiple night's versions, then we can have the cron job run and create a new directory called:
cd /var/www/html/releases mkdir nightly-$(date +%Y%m%d) # move all content from /home/jenkins to new nightly-<date> directory ln -snf nightly-$(date +%Y%m%d) nightly rm -rf nightly-$(date -d "3 days ago" +%Y%m%d)
Mike Why recreate the wheel. www.hyperdrifter.com/software/tidy_rpm_cache.html
Example usages: tidy-rpm-cache.py --dir=/tmp/packages --num-obsolete=3
Works for me... Mike
Thanks Robert
If not, how is the cron job going to cleanup old rpms?
Uploading to a static location that isn't public (and is under /home/jenkins) would make [3] and [4] unnecessary. I think I'd rather avoid having a non-user with sudo and r/w on a publicly served folder.
sounds right.
The publish ssh plugin can touch a file /home/jenkins/nightly/FINISHED when it's done, the cron job locally on ovirt.org can watch for that file and then process the directory structure. +1 This is the approach I was thinking about. good idea, we can have the cronjob monitor the dir for the FINISHED file, and delete it when it's done.
Mike
+1 for pushing this forward or comments?
Eyal.
[1] http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/, this job currently works only for fedora17 rpms (that's what we have now). we'll need to improve it to be more generic (maybe as a matrix job) or duplicate it for each opertating system. [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH +Plugin, using pub/private key with user jenkins to connect [3] jenkins user will need r/q permissions to the nightly folder [4] will need sudo access for it. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
participants (3)
-
Eyal Edri -
Mike Burns -
Robert Middleswarth