how to get files from jenkins.ovirt.org to www.ovirt.org

[Mike Burns]

On Tue, 2012-07-31 at 14:02 -0400, Robert Middleswarth wrote:
> On 07/31/2012 11:03 AM, Mike Burns wrote:
> > On Tue, 2012-07-31 at 09:13 -0400, Robert Middleswarth wrote:
> >> On 07/31/2012 08:51 AM, Eyal Edri wrote:
> >>> ----- Original Message -----
> >>>> From: "Mike Burns" <mburns at redhat.com>
> >>>> To: "Eyal Edri" <eedri at redhat.com>
> >>>> Cc: "infra" <infra at ovirt.org>
> >>>> Sent: Tuesday, July 31, 2012 3:33:01 PM
> >>>> Subject: Re: how to get files from jenkins.ovirt.org to www.ovirt.org
> >>>>
> >>>> On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote:
> >>>>> Hi,
> >>>>>
> >>>>> after looking into the process and testing it via
> >>>>> jenkins.ovirt.info,
> >>>>> here is my proposal:
> >>>>>
> >>>>> 1. Nightly job [1] will collect rpms from all ovirt projects (other
> >>>>> jenkins jobs that build rpms per commit).
> >>>>>      1.1 all files will be kept in a flat dir under $WORKSPCE/rpms
> >>>>>      1.2 using 'publish over ssh' [2] plugin the job will then copy
> >>>>> files to their destination according to their names. [3]
> >>>>>          for e.g 	'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==>
> >>>>> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch
> >>>>>                   'ovirt-engine-3.1.0-3.fc17.src.rpm' ==>
> >>>>> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src
> >>>>>                   'ovirt-engine-3.1.0-3.bz2' ==>
> >>>>> ovirt.org/var/www/html/releases/nightly/src'
> >>>>>      1.3 publish ssh plugin can execute a command after its done
> >>>>>      copying
> >>>>> the file, so all is left is to run 'createrepo' on the relevant
> >>>>> dirs.
> >>>>> [4]
> >>>>>    
> >>>>>
> >>>>> 2. a separate cron cleaning script needs to run on ovirt.org to
> >>>>> delete
> >>>>> old rpms and run createrepo.
> >>>>>
> >>>> Maybe upload to a non-published directory first,
> >>>> like /home/jenkins/nightly, and have the cron job handle cleanup of
> >>>> old
> >>>> rpms, adding of new rpms, and running createrepo?
> >>> sure, i've got not problem with that..
> >>>
> >>> there is one thing i don't understand, and sorry if not an expert in yum:
> >>>
> >>> we say we want to keep history for X builds, will all versions be kept under one repository?
> >>> so 'createrepo' will run on all versions in that directory?
> >>>
> >>> so running yum install file.verX will work even if the repo has a x+1 version?
> >> If you look at existing repo's many of them will include old versions of
> >> the package yum / create repo will simple use the latest version in the
> >> repo that meets all the requirements.  Example when we first moved from
> >> vdsm 4.9 to 4.10 the repo's got really screwy until all the packages
> >> were rebuilt.  In that case the older 4.9 builds would get used until
> >> all the packages were in place then the newer versions would get used.
> >
> > If we still want to keep multiple night's versions, then we can have the
> > cron job run and create a new directory called:
> >
> > cd /var/www/html/releases
> > mkdir nightly-$(date +%Y%m%d)
> > # move all content from /home/jenkins to new nightly-<date> directory
> > ln -snf nightly-$(date +%Y%m%d) nightly
> > rm -rf nightly-$(date -d "3 days ago" +%Y%m%d)
> >
> > Mike
> Why recreate the wheel. www.hyperdrifter.com/software/tidy_rpm_cache.html
> 
> Example usages:  tidy-rpm-cache.py --dir=/tmp/packages --num-obsolete=3

Works for me...

Mike
> 
> Thanks
> Robert
> >
> >>>> If not, how is the cron job going to cleanup old rpms?
> >>>>
> >>>> Uploading to a static location that isn't public (and is
> >>>> under /home/jenkins) would make [3] and [4] unnecessary.  I think I'd
> >>>> rather avoid having a non-user with sudo and r/w on a publicly served
> >>>> folder.
> >>>>
> >>> sounds right.
> >>>
> >>>> The publish ssh plugin can touch a file
> >>>> /home/jenkins/nightly/FINISHED
> >>>> when it's done, the cron job locally on ovirt.org can watch for that
> >>>> file and then process the directory structure.
> >> +1 This is the approach I was thinking about.
> >>> good idea, we can have the cronjob monitor the dir for the FINISHED file,
> >>> and delete it when it's done.
> >>>
> >>>> Mike
> >>>>
> >>>>> +1 for pushing this forward or comments?
> >>>>>
> >>>>> Eyal.
> >>>>>
> >>>>>
> >>>>> [1]
> >>>>> http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/,
> >>>>> this job currently works only for fedora17 rpms (that's what we
> >>>>> have
> >>>>> now).
> >>>>>       we'll need to improve it to be more generic (maybe as a matrix
> >>>>> job) or duplicate it for each opertating system.
> >>>>> [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH
> >>>>> +Plugin, using pub/private key with user jenkins to connect
> >>>>> [3] jenkins user will need r/q permissions to the nightly folder
> >>>>> [4] will need sudo access for it.
> >>>>> _______________________________________________
> >>>>> Infra mailing list
> >>>>> Infra at ovirt.org
> >>>>> http://lists.ovirt.org/mailman/listinfo/infra
> >>>>
> >>> _______________________________________________
> >>> Infra mailing list
> >>> Infra at ovirt.org
> >>> http://lists.ovirt.org/mailman/listinfo/infra
> >>
> >
> 
>