[ovirt-devel] Re: Switching to GH actions build: some not-yet-clear parts and some tips

Il giorno lun 6 dic 2021 alle ore 12:18 Martin Perina <mperina(a)redhat.com&gt; ha scritto: > Hi, > > I've tried to move ovirt-engine-extensions-api > <https://github.com/ovirt/ovirt-engine-extensions-api> to github and > setup GH action to build to have some sort of a basic template for all our > Java based projects: > > > https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/... > > During the work I discovered some parts which should be further discussed: > > 1. Most of projects requires packages from ovirt-master-release RPM > repositories to build itself > - In past we have been using > https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm to > fetch the latest version of this RPM > - Do we plan to keep it and have this URL as the main source for > oVirt master repositories? > No. Please follow https://ovirt.org/develop/dev-process/install-nightly-snapshot.html

> - ovirt-engine-extensions-api projects requires only Virt SIG repo so > I've applied below hack to provide it: > > https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automat... > - I don't like the solution, it would be much nicer to have a constant > URL for the latest ovirt-release-master RPM > Instrucitons in https://ovirt.org/develop/dev-process/install-nightly-snapshot.html whould be stable. For CentOS Virt SIG specifically I haven't yet built the 4.5 release rpms for CentOS . Once they'll be ready you should be able to just dnf install the centos release ovirt-4.5 rpm to enable them. I'll try to handle it before Christmas.

> > 2. Providing RPM repositories for different OS inside single build > artifact > - I wanted to build the project for both CentOS Stream 8 and 9, but I > figured out that upload-artifact action merges results for all different OS > builds into a single directory, so created repositories are mixed together. > This may have a solution, try using the wildcard pattern https://github.com/actions/upload-artifact#upload-using-a-wildcard-pattern

> - I've use GH strategy option and used centos-stream-8 for CS8 and > centos-stream-9 for CS9 builds: > > https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/... > - Do we already have that documented so an OST plugin to use those > artifacts per OS can be created? If not, where to put it? > Didn't see one yet. > > 3. Using maven cache for Java project builds > - When build maven projects in jenkins.ovirt.org we were using > artifactory.ovirt.org to cache maven artifacts (and also save download > bandwidth to PHX datacenter) > - It doesn't make to use artifactory.ovirt.org inside GH actions, so > I've tried to define a GH action specific cache: > > https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/... > make a lot of sense. > > 4. Caching required dependencies > - Above maven dependencies helps for maven invocations before RPM > build but it doesn't help to fetch all required RPM dependencies for pro > RPM maven based build > - Right now it's almost 500 GB of packages to download and install on > top of the default container and it for ovirt-engine-extensions-api it > takes 50 % of the whole build > time just to download and install those packages (I know it's just > 1 minute, but anyway :-) > - I've noticed that VDSM is using its own customized container > <https://github.com/oVirt/vdsm/blob/master/.github/workflows/ci.yml#L8> > - Wouldn't it be worthwhile to also create customized containers for > Java based projects? > I got suggestion from @Yaniv Kaul <ykaul(a)redhat.com&gt; to do something as in Gluster: https://github.com/gluster/Gluster-Builds - they are building Gluster RPMs by running a container (the 'build container'), which has the required dependencies and they updates once a month, or when there's a new dependency. If someone volunteer to maintain those containers, it may save build time and bandwidth.

> > 5. Differences between CS8 and CS9 > - There are again some disturbing changes between CS8 and CS9: > - PowerTools repo from CS8 is now called CRB in CS9 > - pki-deps javapackages-tools modules from CS8 don't exist in CS9 > - Core DNF plugins are not installed by default in CS9 container > - To bypass those differences I've create 2 different shell scripts > and plug them into a single workflow job: > > https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/... > > https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automat... > > https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automat... > > 6. git commands cannot be easily used inside GH action > - GH doesn't checkout the project completely during checkout action, > so .git directory is not present > - In the past I've been using 'git archive' > <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;... > to create .tar.gz source archive, but on GH it needed to be switched to a classic > tar > <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automat... > - In the past I've been using 'git rev-list' > <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;... > to fetch short git hash for snapshot builds, but on GH it needed to be > switched to below hack > <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/... > You can dnf install git before running checkout action and then configure checkout action to download full git content. ``` - uses: actions/checkout@v2 with: fetch-depth: 0 ```

> > > Regards, > Martin > > -- > Martin Perina > Manager, Software Engineering > Red Hat Czech s.r.o. > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PBUC7OKVHJJ... > -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*