how to get files from jenkins.ovirt.org to www.ovirt.org

[Robert Middleswarth]

On 07/31/2012 08:51 AM, Eyal Edri wrote:
>
> ----- Original Message -----
>> From: "Mike Burns" <mburns at redhat.com>
>> To: "Eyal Edri" <eedri at redhat.com>
>> Cc: "infra" <infra at ovirt.org>
>> Sent: Tuesday, July 31, 2012 3:33:01 PM
>> Subject: Re: how to get files from jenkins.ovirt.org to www.ovirt.org
>>
>> On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote:
>>> Hi,
>>>
>>> after looking into the process and testing it via
>>> jenkins.ovirt.info,
>>> here is my proposal:
>>>
>>> 1. Nightly job [1] will collect rpms from all ovirt projects (other
>>> jenkins jobs that build rpms per commit).
>>>     1.1 all files will be kept in a flat dir under $WORKSPCE/rpms
>>>     1.2 using 'publish over ssh' [2] plugin the job will then copy
>>> files to their destination according to their names. [3]
>>>         for e.g 	'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==>
>>> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch
>>>                  'ovirt-engine-3.1.0-3.fc17.src.rpm' ==>
>>> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src
>>>                  'ovirt-engine-3.1.0-3.bz2' ==>
>>> ovirt.org/var/www/html/releases/nightly/src'
>>>     1.3 publish ssh plugin can execute a command after its done
>>>     copying
>>> the file, so all is left is to run 'createrepo' on the relevant
>>> dirs.
>>> [4]
>>>   
>>>
>>> 2. a separate cron cleaning script needs to run on ovirt.org to
>>> delete
>>> old rpms and run createrepo.
>>>
>> Maybe upload to a non-published directory first,
>> like /home/jenkins/nightly, and have the cron job handle cleanup of
>> old
>> rpms, adding of new rpms, and running createrepo?
> sure, i've got not problem with that..
>
> there is one thing i don't understand, and sorry if not an expert in yum:
>
> we say we want to keep history for X builds, will all versions be kept under one repository?
> so 'createrepo' will run on all versions in that directory?
>
> so running yum install file.verX will work even if the repo has a x+1 version?
If you look at existing repo's many of them will include old versions of 
the package yum / create repo will simple use the latest version in the 
repo that meets all the requirements.  Example when we first moved from 
vdsm 4.9 to 4.10 the repo's got really screwy until all the packages 
were rebuilt.  In that case the older 4.9 builds would get used until 
all the packages were in place then the newer versions would get used.
>
>> If not, how is the cron job going to cleanup old rpms?
>>
>> Uploading to a static location that isn't public (and is
>> under /home/jenkins) would make [3] and [4] unnecessary.  I think I'd
>> rather avoid having a non-user with sudo and r/w on a publicly served
>> folder.
>>
> sounds right.
>
>> The publish ssh plugin can touch a file
>> /home/jenkins/nightly/FINISHED
>> when it's done, the cron job locally on ovirt.org can watch for that
>> file and then process the directory structure.
+1 This is the approach I was thinking about.
> good idea, we can have the cronjob monitor the dir for the FINISHED file,
> and delete it when it's done.
>
>> Mike
>>
>>> +1 for pushing this forward or comments?
>>>
>>> Eyal.
>>>
>>>
>>> [1]
>>> http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/,
>>> this job currently works only for fedora17 rpms (that's what we
>>> have
>>> now).
>>>      we'll need to improve it to be more generic (maybe as a matrix
>>> job) or duplicate it for each opertating system.
>>> [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH
>>> +Plugin, using pub/private key with user jenkins to connect
>>> [3] jenkins user will need r/q permissions to the nightly folder
>>> [4] will need sudo access for it.
>>> _______________________________________________
>>> Infra mailing list
>>> Infra at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>>
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra


-- 
Thanks
Robert Middleswarth
@rmiddle (twitter/IRC)