Switching to GH actions build: some not-yet-clear parts and some tips
Hi, I've tried to move ovirt-engine-extensions-api <https://github.com/ovirt/ovirt-engine-extensions-api> to github and setup GH action to build to have some sort of a basic template for all our Java based projects: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor... During the work I discovered some parts which should be further discussed: 1. Most of projects requires packages from ovirt-master-release RPM repositories to build itself - In past we have been using https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm to fetch the latest version of this RPM - Do we plan to keep it and have this URL as the main source for oVirt master repositories? - ovirt-engine-extensions-api projects requires only Virt SIG repo so I've applied below hack to provide it: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation... - I don't like the solution, it would be much nicer to have a constant URL for the latest ovirt-release-master RPM 2. Providing RPM repositories for different OS inside single build artifact - I wanted to build the project for both CentOS Stream 8 and 9, but I figured out that upload-artifact action merges results for all different OS builds into a single directory, so created repositories are mixed together. - I've use GH strategy option and used centos-stream-8 for CS8 and centos-stream-9 for CS9 builds: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor... - Do we already have that documented so an OST plugin to use those artifacts per OS can be created? If not, where to put it? 3. Using maven cache for Java project builds - When build maven projects in jenkins.ovirt.org we were using artifactory.ovirt.org to cache maven artifacts (and also save download bandwidth to PHX datacenter) - It doesn't make to use artifactory.ovirt.org inside GH actions, so I've tried to define a GH action specific cache: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor... 4. Caching required dependencies - Above maven dependencies helps for maven invocations before RPM build but it doesn't help to fetch all required RPM dependencies for pro RPM maven based build - Right now it's almost 500 GB of packages to download and install on top of the default container and it for ovirt-engine-extensions-api it takes 50 % of the whole build time just to download and install those packages (I know it's just 1 minute, but anyway :-) - I've noticed that VDSM is using its own customized container <https://github.com/oVirt/vdsm/blob/master/.github/workflows/ci.yml#L8> - Wouldn't it be worthwhile to also create customized containers for Java based projects? 5. Differences between CS8 and CS9 - There are again some disturbing changes between CS8 and CS9: - PowerTools repo from CS8 is now called CRB in CS9 - pki-deps javapackages-tools modules from CS8 don't exist in CS9 - Core DNF plugins are not installed by default in CS9 container - To bypass those differences I've create 2 different shell scripts and plug them into a single workflow job: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor... https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation... https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation... 6. git commands cannot be easily used inside GH action - GH doesn't checkout the project completely during checkout action, so .git directory is not present - In the past I've been using 'git archive' <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;f=automation/build-artifacts.sh;h=965012c1195eaf288e149bde71e64b2e9647b2c4;hb=refs/heads/master#l45> to create .tar.gz source archive, but on GH it needed to be switched to a classic tar <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation/build-artifacts.sh#L17> - In the past I've been using 'git rev-list' <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;f=automation/build-artifacts.sh;h=965012c1195eaf288e149bde71e64b2e9647b2c4;hb=refs/heads/master#l39> to fetch short git hash for snapshot builds, but on GH it needed to be switched to below hack <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L47> Regards, Martin -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.
On 6. 12. 2021, at 12:16, Martin Perina <mperina@redhat.com> wrote:
Hi,
I've tried to move ovirt-engine-extensions-api <https://github.com/ovirt/ovirt-engine-extensions-api> to github and setup GH action to build to have some sort of a basic template for all our Java based projects:
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor... <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml>
During the work I discovered some parts which should be further discussed:
1. Most of projects requires packages from ovirt-master-release RPM repositories to build itself - In past we have been using https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm <https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm> to fetch the latest version of this RPM - Do we plan to keep it and have this URL as the main source for oVirt master repositories? - ovirt-engine-extensions-api projects requires only Virt SIG repo so I've applied below hack to provide it: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation... <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation/prepare-env-cs8.sh#L10> - I don't like the solution, it would be much nicer to have a constant URL for the latest ovirt-release-master RPM
I'd say the canonical way to use it should be dnf copr enable ovirt/ovirt-master-snapshot dnf install ovirt-release-master
2. Providing RPM repositories for different OS inside single build artifact - I wanted to build the project for both CentOS Stream 8 and 9, but I figured out that upload-artifact action merges results for all different OS builds into a single directory, so created repositories are mixed together. - I've use GH strategy option and used centos-stream-8 for CS8 and centos-stream-9 for CS9 builds: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor... <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L14> - Do we already have that documented so an OST plugin to use those artifacts per OS can be created? If not, where to put it?
not yet. but we wanted to "develop" a unified artifact structiure with repodata so taht it can be then consumed as a repo elsewhere
3. Using maven cache for Java project builds - When build maven projects in jenkins.ovirt.org <http://jenkins.ovirt.org/> we were using artifactory.ovirt.org <http://artifactory.ovirt.org/> to cache maven artifacts (and also save download bandwidth to PHX datacenter) - It doesn't make to use artifactory.ovirt.org <http://artifactory.ovirt.org/> inside GH actions, so I've tried to define a GH action specific cache: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor... <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L37>
4. Caching required dependencies - Above maven dependencies helps for maven invocations before RPM build but it doesn't help to fetch all required RPM dependencies for pro RPM maven based build - Right now it's almost 500 GB of packages to download and install on top of the default container and it for ovirt-engine-extensions-api it takes 50 % of the whole build time just to download and install those packages (I know it's just 1 minute, but anyway :-) - I've noticed that VDSM is using its own customized container <https://github.com/oVirt/vdsm/blob/master/.github/workflows/ci.yml#L8> - Wouldn't it be worthwhile to also create customized containers for Java based projects?
I think our own container with these dependencies would be great
5. Differences between CS8 and CS9 - There are again some disturbing changes between CS8 and CS9: - PowerTools repo from CS8 is now called CRB in CS9 - pki-deps javapackages-tools modules from CS8 don't exist in CS9 - Core DNF plugins are not installed by default in CS9 container - To bypass those differences I've create 2 different shell scripts and plug them into a single workflow job: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor... <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L33> https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation... <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation/prepare-env-cs8.sh> https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation... <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation/prepare-env-cs9.sh>
6. git commands cannot be easily used inside GH action - GH doesn't checkout the project completely during checkout action, so .git directory is not present - In the past I've been using 'git archive' <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;f=automation/build-artifacts.sh;h=965012c1195eaf288e149bde71e64b2e9647b2c4;hb=refs/heads/master#l45> to create .tar.gz source archive, but on GH it needed to be switched to a classic tar <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation/build-artifacts.sh#L17> - In the past I've been using 'git rev-list' <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;f=automation/build-artifacts.sh;h=965012c1195eaf288e149bde71e64b2e9647b2c4;hb=refs/heads/master#l39> to fetch short git hash for snapshot builds, but on GH it needed to be switched to below hack <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L47>
did you try options from https://github.com/actions/checkout#usage? Thanks, michal
Regards, Martin
-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o. _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PBUC7OKVHJJPDL...
Il giorno lun 6 dic 2021 alle ore 12:18 Martin Perina <mperina@redhat.com> ha scritto:
Hi,
I've tried to move ovirt-engine-extensions-api <https://github.com/ovirt/ovirt-engine-extensions-api> to github and setup GH action to build to have some sort of a basic template for all our Java based projects:
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor...
During the work I discovered some parts which should be further discussed:
1. Most of projects requires packages from ovirt-master-release RPM repositories to build itself - In past we have been using https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm to fetch the latest version of this RPM - Do we plan to keep it and have this URL as the main source for oVirt master repositories?
No. Please follow https://ovirt.org/develop/dev-process/install-nightly-snapshot.html
- ovirt-engine-extensions-api projects requires only Virt SIG repo so I've applied below hack to provide it:
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation...
- I don't like the solution, it would be much nicer to have a constant
URL for the latest ovirt-release-master RPM
Instrucitons in https://ovirt.org/develop/dev-process/install-nightly-snapshot.html whould be stable. For CentOS Virt SIG specifically I haven't yet built the 4.5 release rpms for CentOS . Once they'll be ready you should be able to just dnf install the centos release ovirt-4.5 rpm to enable them. I'll try to handle it before Christmas.
2. Providing RPM repositories for different OS inside single build artifact - I wanted to build the project for both CentOS Stream 8 and 9, but I figured out that upload-artifact action merges results for all different OS builds into a single directory, so created repositories are mixed together.
This may have a solution, try using the wildcard pattern https://github.com/actions/upload-artifact#upload-using-a-wildcard-pattern
- I've use GH strategy option and used centos-stream-8 for CS8 and centos-stream-9 for CS9 builds:
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor... - Do we already have that documented so an OST plugin to use those artifacts per OS can be created? If not, where to put it?
Didn't see one yet.
3. Using maven cache for Java project builds - When build maven projects in jenkins.ovirt.org we were using artifactory.ovirt.org to cache maven artifacts (and also save download bandwidth to PHX datacenter) - It doesn't make to use artifactory.ovirt.org inside GH actions, so I've tried to define a GH action specific cache:
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor...
make a lot of sense.
4. Caching required dependencies - Above maven dependencies helps for maven invocations before RPM build but it doesn't help to fetch all required RPM dependencies for pro RPM maven based build - Right now it's almost 500 GB of packages to download and install on top of the default container and it for ovirt-engine-extensions-api it takes 50 % of the whole build time just to download and install those packages (I know it's just 1 minute, but anyway :-) - I've noticed that VDSM is using its own customized container <https://github.com/oVirt/vdsm/blob/master/.github/workflows/ci.yml#L8> - Wouldn't it be worthwhile to also create customized containers for Java based projects?
I got suggestion from @Yaniv Kaul <ykaul@redhat.com> to do something as in Gluster: https://github.com/gluster/Gluster-Builds - they are building Gluster RPMs by running a container (the 'build container'), which has the required dependencies and they updates once a month, or when there's a new dependency. If someone volunteer to maintain those containers, it may save build time and bandwidth.
5. Differences between CS8 and CS9 - There are again some disturbing changes between CS8 and CS9: - PowerTools repo from CS8 is now called CRB in CS9 - pki-deps javapackages-tools modules from CS8 don't exist in CS9 - Core DNF plugins are not installed by default in CS9 container - To bypass those differences I've create 2 different shell scripts and plug them into a single workflow job:
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor...
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation...
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation...
6. git commands cannot be easily used inside GH action - GH doesn't checkout the project completely during checkout action, so .git directory is not present - In the past I've been using 'git archive' <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;f=automation/build-artifacts.sh;h=965012c1195eaf288e149bde71e64b2e9647b2c4;hb=refs/heads/master#l45> to create .tar.gz source archive, but on GH it needed to be switched to a classic tar <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation/build-artifacts.sh#L17> - In the past I've been using 'git rev-list' <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;f=automation/build-artifacts.sh;h=965012c1195eaf288e149bde71e64b2e9647b2c4;hb=refs/heads/master#l39> to fetch short git hash for snapshot builds, but on GH it needed to be switched to below hack <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L47>
You can dnf install git before running checkout action and then configure checkout action to download full git content. ``` - uses: actions/checkout@v2 with: fetch-depth: 0 ```
Regards, Martin
-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o. _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PBUC7OKVHJJPDL...
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*
On Mon, Dec 6, 2021 at 1:44 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno lun 6 dic 2021 alle ore 12:18 Martin Perina <mperina@redhat.com> ha scritto:
Hi,
I've tried to move ovirt-engine-extensions-api <https://github.com/ovirt/ovirt-engine-extensions-api> to github and setup GH action to build to have some sort of a basic template for all our Java based projects:
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor...
During the work I discovered some parts which should be further discussed:
1. Most of projects requires packages from ovirt-master-release RPM repositories to build itself - In past we have been using https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm to fetch the latest version of this RPM - Do we plan to keep it and have this URL as the main source for oVirt master repositories?
No. Please follow https://ovirt.org/develop/dev-process/install-nightly-snapshot.html
Fixed, thanks! https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor...
- ovirt-engine-extensions-api projects requires only Virt SIG repo so I've applied below hack to provide it:
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation...
- I don't like the solution, it would be much nicer to have a constant
URL for the latest ovirt-release-master RPM
Instrucitons in https://ovirt.org/develop/dev-process/install-nightly-snapshot.html whould be stable. For CentOS Virt SIG specifically I haven't yet built the 4.5 release rpms for CentOS . Once they'll be ready you should be able to just dnf install the centos release ovirt-4.5 rpm to enable them. I'll try to handle it before Christmas.
No worries, we are still far from the stable oVirt 4.5 release :-( And build jobs should depend on ovirt-release-master anyway ...
2. Providing RPM repositories for different OS inside single build artifact - I wanted to build the project for both CentOS Stream 8 and 9, but I figured out that upload-artifact action merges results for all different OS builds into a single directory, so created repositories are mixed together.
This may have a solution, try using the wildcard pattern https://github.com/actions/upload-artifact#upload-using-a-wildcard-pattern
For now the solution works fine, but we just need to agree on standards. For example: Archive: artifacts.zip centos-stream-8/*.rpm centos-stream-8/repodata/* centos-stream-9/*.rpm centos-stream-9/repodata/
- I've use GH strategy option and used centos-stream-8 for CS8 and centos-stream-9 for CS9 builds:
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor... - Do we already have that documented so an OST plugin to use those artifacts per OS can be created? If not, where to put it?
Didn't see one yet.
3. Using maven cache for Java project builds - When build maven projects in jenkins.ovirt.org we were using artifactory.ovirt.org to cache maven artifacts (and also save download bandwidth to PHX datacenter) - It doesn't make to use artifactory.ovirt.org inside GH actions, so I've tried to define a GH action specific cache:
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor...
make a lot of sense.
4. Caching required dependencies - Above maven dependencies helps for maven invocations before RPM build but it doesn't help to fetch all required RPM dependencies for pro RPM maven based build - Right now it's almost 500 GB of packages to download and install on top of the default container and it for ovirt-engine-extensions-api it takes 50 % of the whole build time just to download and install those packages (I know it's just 1 minute, but anyway :-) - I've noticed that VDSM is using its own customized container <https://github.com/oVirt/vdsm/blob/master/.github/workflows/ci.yml#L8> - Wouldn't it be worthwhile to also create customized containers for Java based projects?
I got suggestion from @Yaniv Kaul <ykaul@redhat.com> to do something as in Gluster: https://github.com/gluster/Gluster-Builds - they are building Gluster RPMs by running a container (the 'build container'), which has the required dependencies and they updates once a month, or when there's a new dependency.
If someone volunteer to maintain those containers, it may save build time and bandwidth.
It's not a must for the switch, but it can be really useful. If there is no volunteer till, I will try to take a look, but not earlier than January
5. Differences between CS8 and CS9 - There are again some disturbing changes between CS8 and CS9: - PowerTools repo from CS8 is now called CRB in CS9 - pki-deps javapackages-tools modules from CS8 don't exist in CS9 - Core DNF plugins are not installed by default in CS9 container - To bypass those differences I've create 2 different shell scripts and plug them into a single workflow job:
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor...
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation...
https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation...
6. git commands cannot be easily used inside GH action - GH doesn't checkout the project completely during checkout action, so .git directory is not present - In the past I've been using 'git archive' <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;f=automation/build-artifacts.sh;h=965012c1195eaf288e149bde71e64b2e9647b2c4;hb=refs/heads/master#l45> to create .tar.gz source archive, but on GH it needed to be switched to a classic tar <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation/build-artifacts.sh#L17> - In the past I've been using 'git rev-list' <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;f=automation/build-artifacts.sh;h=965012c1195eaf288e149bde71e64b2e9647b2c4;hb=refs/heads/master#l39> to fetch short git hash for snapshot builds, but on GH it needed to be switched to below hack <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L47>
You can dnf install git before running checkout action and then configure checkout action to download full git content. ``` - uses: actions/checkout@v2 with: fetch-depth: 0 ```
Ah, I found the issue, git command needs to be installed before performing a checkout. If the git is not installed, then GH provide checkout without .git directory If git is installed, then 'git rev-parse' and 'git archive' work fine even with the default fetch-depth (which means only latest commit): https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wor...
Regards, Martin
-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o. _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PBUC7OKVHJJPDL...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <https://www.redhat.com/>
*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*
-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.
participants (3)
-
Martin Perina -
Michal Skrivanek -
Sandro Bonazzola