From: "Mike Burns" <mburns(a)redhat.com&gt; To: "Eyal Edri" <eedri(a)redhat.com&gt; Cc: "infra" <infra(a)ovirt.org&gt; Sent: Tuesday, July 31, 2012 3:33:01 PM Subject: Re: how to get files from jenkins.ovirt.org to www.ovirt.org On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote: > Hi, > > after looking into the process and testing it via > jenkins.ovirt.info, > here is my proposal: > > 1. Nightly job [1] will collect rpms from all ovirt projects (other > jenkins jobs that build rpms per commit). > 1.1 all files will be kept in a flat dir under $WORKSPCE/rpms > 1.2 using 'publish over ssh' [2] plugin the job will then copy > files to their destination according to their names. [3] > for e.g 'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==> > ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch > 'ovirt-engine-3.1.0-3.fc17.src.rpm' ==> > ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src > 'ovirt-engine-3.1.0-3.bz2' ==> > ovirt.org/var/www/html/releases/nightly/src' > 1.3 publish ssh plugin can execute a command after its done > copying > the file, so all is left is to run 'createrepo' on the relevant > dirs. > [4] > > > 2. a separate cron cleaning script needs to run on ovirt.org to > delete > old rpms and run createrepo. > Maybe upload to a non-published directory first, like /home/jenkins/nightly, and have the cron job handle cleanup of old rpms, adding of new rpms, and running createrepo?

If not, how is the cron job going to cleanup old rpms? Uploading to a static location that isn't public (and is under /home/jenkins) would make [3] and [4] unnecessary. I think I'd rather avoid having a non-user with sudo and r/w on a publicly served folder.

The publish ssh plugin can touch a file /home/jenkins/nightly/FINISHED when it's done, the cron job locally on ovirt.org can watch for that file and then process the directory structure.

Mike > > +1 for pushing this forward or comments? > > Eyal. > > > [1] > http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/, > this job currently works only for fedora17 rpms (that's what we > have > now). > we'll need to improve it to be more generic (maybe as a matrix > job) or duplicate it for each opertating system. > [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH > +Plugin, using pub/private key with user jenkins to connect > [3] jenkins user will need r/q permissions to the nightly folder > [4] will need sudo access for it. > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra

On 07/31/2012 08:51 AM, Eyal Edri wrote: > > ----- Original Message ----- >> From: "Mike Burns" <mburns(a)redhat.com&gt; >> To: "Eyal Edri" <eedri(a)redhat.com&gt; >> Cc: "infra" <infra(a)ovirt.org&gt; >> Sent: Tuesday, July 31, 2012 3:33:01 PM >> Subject: Re: how to get files from jenkins.ovirt.org to www.ovirt.org >> >> On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote: >>> Hi, >>> >>> after looking into the process and testing it via >>> jenkins.ovirt.info, >>> here is my proposal: >>> >>> 1. Nightly job [1] will collect rpms from all ovirt projects (other >>> jenkins jobs that build rpms per commit). >>> 1.1 all files will be kept in a flat dir under $WORKSPCE/rpms >>> 1.2 using 'publish over ssh' [2] plugin the job will then copy >>> files to their destination according to their names. [3] >>> for e.g 'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==> >>> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch >>> 'ovirt-engine-3.1.0-3.fc17.src.rpm' ==> >>> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src >>> 'ovirt-engine-3.1.0-3.bz2' ==> >>> ovirt.org/var/www/html/releases/nightly/src' >>> 1.3 publish ssh plugin can execute a command after its done >>> copying >>> the file, so all is left is to run 'createrepo' on the relevant >>> dirs. >>> [4] >>> >>> >>> 2. a separate cron cleaning script needs to run on ovirt.org to >>> delete >>> old rpms and run createrepo. >>> >> Maybe upload to a non-published directory first, >> like /home/jenkins/nightly, and have the cron job handle cleanup of >> old >> rpms, adding of new rpms, and running createrepo? > sure, i've got not problem with that.. > > there is one thing i don't understand, and sorry if not an expert in yum: > > we say we want to keep history for X builds, will all versions be kept under one repository? > so 'createrepo' will run on all versions in that directory? > > so running yum install file.verX will work even if the repo has a x+1 version? If you look at existing repo's many of them will include old versions of the package yum / create repo will simple use the latest version in the repo that meets all the requirements. Example when we first moved from vdsm 4.9 to 4.10 the repo's got really screwy until all the packages were rebuilt. In that case the older 4.9 builds would get used until all the packages were in place then the newer versions would get used.

> >> If not, how is the cron job going to cleanup old rpms? >> >> Uploading to a static location that isn't public (and is >> under /home/jenkins) would make [3] and [4] unnecessary. I think I'd >> rather avoid having a non-user with sudo and r/w on a publicly served >> folder. >> > sounds right. > >> The publish ssh plugin can touch a file >> /home/jenkins/nightly/FINISHED >> when it's done, the cron job locally on ovirt.org can watch for that >> file and then process the directory structure. +1 This is the approach I was thinking about. > good idea, we can have the cronjob monitor the dir for the FINISHED file, > and delete it when it's done. > >> Mike >> >>> +1 for pushing this forward or comments? >>> >>> Eyal. >>> >>> >>> [1] >>> http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/, >>> this job currently works only for fedora17 rpms (that's what we >>> have >>> now). >>> we'll need to improve it to be more generic (maybe as a matrix >>> job) or duplicate it for each opertating system. >>> [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH >>> +Plugin, using pub/private key with user jenkins to connect >>> [3] jenkins user will need r/q permissions to the nightly folder >>> [4] will need sudo access for it. >>> _______________________________________________ >>> Infra mailing list >>> Infra(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/infra >> >> > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra

On 07/31/2012 11:03 AM, Mike Burns wrote: > On Tue, 2012-07-31 at 09:13 -0400, Robert Middleswarth wrote: >> On 07/31/2012 08:51 AM, Eyal Edri wrote: >>> ----- Original Message ----- >>>> From: "Mike Burns" <mburns(a)redhat.com&gt; >>>> To: "Eyal Edri" <eedri(a)redhat.com&gt; >>>> Cc: "infra" <infra(a)ovirt.org&gt; >>>> Sent: Tuesday, July 31, 2012 3:33:01 PM >>>> Subject: Re: how to get files from jenkins.ovirt.org to www.ovirt.org >>>> >>>> On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote: >>>>> Hi, >>>>> >>>>> after looking into the process and testing it via >>>>> jenkins.ovirt.info, >>>>> here is my proposal: >>>>> >>>>> 1. Nightly job [1] will collect rpms from all ovirt projects (other >>>>> jenkins jobs that build rpms per commit). >>>>> 1.1 all files will be kept in a flat dir under $WORKSPCE/rpms >>>>> 1.2 using 'publish over ssh' [2] plugin the job will then copy >>>>> files to their destination according to their names. [3] >>>>> for e.g 'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==> >>>>> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch >>>>> 'ovirt-engine-3.1.0-3.fc17.src.rpm' ==> >>>>> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src >>>>> 'ovirt-engine-3.1.0-3.bz2' ==> >>>>> ovirt.org/var/www/html/releases/nightly/src' >>>>> 1.3 publish ssh plugin can execute a command after its done >>>>> copying >>>>> the file, so all is left is to run 'createrepo' on the relevant >>>>> dirs. >>>>> [4] >>>>> >>>>> >>>>> 2. a separate cron cleaning script needs to run on ovirt.org to >>>>> delete >>>>> old rpms and run createrepo. >>>>> >>>> Maybe upload to a non-published directory first, >>>> like /home/jenkins/nightly, and have the cron job handle cleanup of >>>> old >>>> rpms, adding of new rpms, and running createrepo? >>> sure, i've got not problem with that.. >>> >>> there is one thing i don't understand, and sorry if not an expert in yum: >>> >>> we say we want to keep history for X builds, will all versions be kept under one repository? >>> so 'createrepo' will run on all versions in that directory? >>> >>> so running yum install file.verX will work even if the repo has a x+1 version? >> If you look at existing repo's many of them will include old versions of >> the package yum / create repo will simple use the latest version in the >> repo that meets all the requirements. Example when we first moved from >> vdsm 4.9 to 4.10 the repo's got really screwy until all the packages >> were rebuilt. In that case the older 4.9 builds would get used until >> all the packages were in place then the newer versions would get used. > > If we still want to keep multiple night's versions, then we can have the > cron job run and create a new directory called: > > cd /var/www/html/releases > mkdir nightly-$(date +%Y%m%d) > # move all content from /home/jenkins to new nightly-<date> directory > ln -snf nightly-$(date +%Y%m%d) nightly > rm -rf nightly-$(date -d "3 days ago" +%Y%m%d) > > Mike Why recreate the wheel. www.hyperdrifter.com/software/tidy_rpm_cache.html Example usages: tidy-rpm-cache.py --dir=/tmp/packages --num-obsolete=3

Thanks Robert > >>>> If not, how is the cron job going to cleanup old rpms? >>>> >>>> Uploading to a static location that isn't public (and is >>>> under /home/jenkins) would make [3] and [4] unnecessary. I think I'd >>>> rather avoid having a non-user with sudo and r/w on a publicly served >>>> folder. >>>> >>> sounds right. >>> >>>> The publish ssh plugin can touch a file >>>> /home/jenkins/nightly/FINISHED >>>> when it's done, the cron job locally on ovirt.org can watch for that >>>> file and then process the directory structure. >> +1 This is the approach I was thinking about. >>> good idea, we can have the cronjob monitor the dir for the FINISHED file, >>> and delete it when it's done. >>> >>>> Mike >>>> >>>>> +1 for pushing this forward or comments? >>>>> >>>>> Eyal. >>>>> >>>>> >>>>> [1] >>>>> http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/, >>>>> this job currently works only for fedora17 rpms (that's what we >>>>> have >>>>> now). >>>>> we'll need to improve it to be more generic (maybe as a matrix >>>>> job) or duplicate it for each opertating system. >>>>> [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH >>>>> +Plugin, using pub/private key with user jenkins to connect >>>>> [3] jenkins user will need r/q permissions to the nightly folder >>>>> [4] will need sudo access for it. >>>>> _______________________________________________ >>>>> Infra mailing list >>>>> Infra(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/infra >>>> >>> _______________________________________________ >>> Infra mailing list >>> Infra(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/infra >> >