how to get files from jenkins.ovirt.org to www.ovirt.org

[Robert Middleswarth]

On 07/31/2012 11:03 AM, Mike Burns wrote:
> On Tue, 2012-07-31 at 09:13 -0400, Robert Middleswarth wrote:
>> On 07/31/2012 08:51 AM, Eyal Edri wrote:
>>> ----- Original Message -----
>>>> From: "Mike Burns" <mburns at redhat.com>
>>>> To: "Eyal Edri" <eedri at redhat.com>
>>>> Cc: "infra" <infra at ovirt.org>
>>>> Sent: Tuesday, July 31, 2012 3:33:01 PM
>>>> Subject: Re: how to get files from jenkins.ovirt.org to www.ovirt.org
>>>>
>>>> On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote:
>>>>> Hi,
>>>>>
>>>>> after looking into the process and testing it via
>>>>> jenkins.ovirt.info,
>>>>> here is my proposal:
>>>>>
>>>>> 1. Nightly job [1] will collect rpms from all ovirt projects (other
>>>>> jenkins jobs that build rpms per commit).
>>>>>      1.1 all files will be kept in a flat dir under $WORKSPCE/rpms
>>>>>      1.2 using 'publish over ssh' [2] plugin the job will then copy
>>>>> files to their destination according to their names. [3]
>>>>>          for e.g 	'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==>
>>>>> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch
>>>>>                   'ovirt-engine-3.1.0-3.fc17.src.rpm' ==>
>>>>> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src
>>>>>                   'ovirt-engine-3.1.0-3.bz2' ==>
>>>>> ovirt.org/var/www/html/releases/nightly/src'
>>>>>      1.3 publish ssh plugin can execute a command after its done
>>>>>      copying
>>>>> the file, so all is left is to run 'createrepo' on the relevant
>>>>> dirs.
>>>>> [4]
>>>>>    
>>>>>
>>>>> 2. a separate cron cleaning script needs to run on ovirt.org to
>>>>> delete
>>>>> old rpms and run createrepo.
>>>>>
>>>> Maybe upload to a non-published directory first,
>>>> like /home/jenkins/nightly, and have the cron job handle cleanup of
>>>> old
>>>> rpms, adding of new rpms, and running createrepo?
>>> sure, i've got not problem with that..
>>>
>>> there is one thing i don't understand, and sorry if not an expert in yum:
>>>
>>> we say we want to keep history for X builds, will all versions be kept under one repository?
>>> so 'createrepo' will run on all versions in that directory?
>>>
>>> so running yum install file.verX will work even if the repo has a x+1 version?
>> If you look at existing repo's many of them will include old versions of
>> the package yum / create repo will simple use the latest version in the
>> repo that meets all the requirements.  Example when we first moved from
>> vdsm 4.9 to 4.10 the repo's got really screwy until all the packages
>> were rebuilt.  In that case the older 4.9 builds would get used until
>> all the packages were in place then the newer versions would get used.
>
> If we still want to keep multiple night's versions, then we can have the
> cron job run and create a new directory called:
>
> cd /var/www/html/releases
> mkdir nightly-$(date +%Y%m%d)
> # move all content from /home/jenkins to new nightly-<date> directory
> ln -snf nightly-$(date +%Y%m%d) nightly
> rm -rf nightly-$(date -d "3 days ago" +%Y%m%d)
>
> Mike
Why recreate the wheel. www.hyperdrifter.com/software/tidy_rpm_cache.html

Example usages:  tidy-rpm-cache.py --dir=/tmp/packages --num-obsolete=3

Thanks
Robert
>
>>>> If not, how is the cron job going to cleanup old rpms?
>>>>
>>>> Uploading to a static location that isn't public (and is
>>>> under /home/jenkins) would make [3] and [4] unnecessary.  I think I'd
>>>> rather avoid having a non-user with sudo and r/w on a publicly served
>>>> folder.
>>>>
>>> sounds right.
>>>
>>>> The publish ssh plugin can touch a file
>>>> /home/jenkins/nightly/FINISHED
>>>> when it's done, the cron job locally on ovirt.org can watch for that
>>>> file and then process the directory structure.
>> +1 This is the approach I was thinking about.
>>> good idea, we can have the cronjob monitor the dir for the FINISHED file,
>>> and delete it when it's done.
>>>
>>>> Mike
>>>>
>>>>> +1 for pushing this forward or comments?
>>>>>
>>>>> Eyal.
>>>>>
>>>>>
>>>>> [1]
>>>>> http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/,
>>>>> this job currently works only for fedora17 rpms (that's what we
>>>>> have
>>>>> now).
>>>>>       we'll need to improve it to be more generic (maybe as a matrix
>>>>> job) or duplicate it for each opertating system.
>>>>> [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH
>>>>> +Plugin, using pub/private key with user jenkins to connect
>>>>> [3] jenkins user will need r/q permissions to the nightly folder
>>>>> [4] will need sudo access for it.
>>>>> _______________________________________________
>>>>> Infra mailing list
>>>>> Infra at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>>
>>> _______________________________________________
>>> Infra mailing list
>>> Infra at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>


-- 
Thanks
Robert Middleswarth
@rmiddle (twitter/IRC)