On Mon, Dec 6, 2021 at 1:44 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:


Il giorno lun 6 dic 2021 alle ore 12:18 Martin Perina <mperina@redhat.com> ha scritto:
Hi,

I've tried to move ovirt-engine-extensions-api to github and setup GH action to build to have some sort of a basic template for all our Java based projects:


During the work I discovered some parts which should be further discussed:

1. Most of projects requires packages from ovirt-master-release RPM repositories to build itself
    - In past we have been using https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm to fetch the latest version of this RPM
    - Do we plan to keep it and have this URL as the main source for oVirt master repositories?

No.

Fixed, thanks!

https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L32

 
    - ovirt-engine-extensions-api projects requires only Virt SIG repo so I've applied below hack to provide it:
    - I don't like the solution, it would be much nicer to have a constant URL for the latest ovirt-release-master RPM

For CentOS Virt SIG specifically I haven't yet built the 4.5 release rpms for CentOS . Once they'll be ready you should be able to just dnf install the centos release ovirt-4.5 rpm to enable them. I'll try to handle it before Christmas.

No worries, we are still far from the stable oVirt 4.5 release :-( And build jobs should depend on ovirt-release-master anyway ...

 

2. Providing RPM repositories for different OS inside single build artifact
    - I wanted to build the project for both CentOS Stream 8 and 9, but I figured out that upload-artifact action merges results for all different OS builds into a single directory, so created repositories are mixed together.

This may have a solution, try using the wildcard pattern https://github.com/actions/upload-artifact#upload-using-a-wildcard-pattern

For now the solution works fine, but we just need to agree on standards. For example:

Archive:  artifacts.zip
    centos-stream-8/*.rpm
    centos-stream-8/repodata/*
    centos-stream-9/*.rpm
    centos-stream-9/repodata/
 
 
    - I've use GH strategy option and used centos-stream-8 for CS8 and centos-stream-9 for CS9 builds:
    - Do we already have that documented so an OST plugin to use those artifacts per OS can be created? If not, where to put it?

Didn't see one yet.

 

3. Using maven cache for Java project builds
    - When build maven projects in jenkins.ovirt.org we were using artifactory.ovirt.org to cache maven artifacts (and also save download bandwidth to PHX datacenter)
    - It doesn't make to use artifactory.ovirt.org inside GH actions, so I've tried to define a GH action specific cache:

make a lot of sense.

 

4. Caching required dependencies
    - Above maven dependencies helps for maven invocations before RPM build but it doesn't help to fetch all required RPM dependencies for pro RPM maven based build
    - Right now it's almost 500 GB of packages to download and install on top of the default container and it for ovirt-engine-extensions-api it takes 50 % of the whole build
      time just to download and install those packages (I know it's just 1 minute, but anyway :-)
    - I've noticed that VDSM is using its own customized container
    - Wouldn't it be worthwhile to also create customized containers for Java based projects?

I got suggestion from @Yaniv Kaul to do something as in Gluster:  https://github.com/gluster/Gluster-Builds - they are building Gluster RPMs by running a container (the 'build container'), which has the required dependencies and they updates once a month, or when there's a new dependency. 

If someone volunteer to maintain those containers, it may save build time and bandwidth.

It's not a must for the switch, but it can be really useful. If there is no volunteer till, I will try to take a look, but not earlier than January

 

5. Differences between CS8 and CS9
    - There are again some disturbing changes between CS8 and CS9:
        - PowerTools repo from CS8 is now called CRB in CS9
        - pki-deps javapackages-tools modules from CS8 don't exist in CS9
        - Core DNF plugins are not installed by default in CS9 container
    - To bypass those differences I've create 2 different shell scripts and plug them into a single workflow job:

6. git commands cannot be easily used inside GH action
    - GH doesn't checkout the project completely during checkout action, so .git directory is not present
    - In the past I've been using 'git archive' to create .tar.gz source archive, but on GH it needed to be switched to a classic tar
    - In the past I've been using 'git rev-list' to fetch short git hash for snapshot builds, but on GH it needed to be switched to below hack

You can dnf install git before running checkout action and then configure checkout action to download full git content.
```
    - uses: actions/checkout@v2
      with:
        fetch-depth: 0
```

Ah, I found the issue, git command needs to be installed before performing a checkout. If the git is not installed, then GH provide checkout without .git directory
If git is installed, then 'git rev-parse' and 'git archive' work fine even with the default fetch-depth (which means only latest commit):

https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L72
 
 


Regards,
Martin

--
Martin Perina
Manager, Software Engineering
Red Hat Czech s.r.o.
_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PBUC7OKVHJJPDLI6C2YAMZNE7LGCF73A/


--

Sandro Bonazzola

MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV

Red Hat EMEA

sbonazzo@redhat.com   

Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.




--
Martin Perina
Manager, Software Engineering
Red Hat Czech s.r.o.