Container-based CI backend is now available for use
A little less then a month ago I sent an email to this list telling you all about ongoing work to create a new container-based backend for the oVirt CI system. I'm pleased to announce that we have managed to finally merged an initial set of patches implementing that backend yesterday, and it is now available for general use. *What? Were? How do I use it?* Documentation about how to use the new backend is now available in read the docs <https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index.html> . *Wait! I needed it to do X which it doesn't!* For the time being the new backend lacks some features that some may consider to be essential, such as automated cloning of patch source code and build artifact collection. We already have implemented patches providing a substantial amount of additional functionality, and hopefully we will be able to merge them soon. Following is a list of those patches and the features they implement: 1. Automated source cloning support: - 104213 <https://gerrit.ovirt.org/104213>: Implement STDCI DSL support for initContainers - 104590 <https://gerrit.ovirt.org/104590>: STDCI DSL: Add the `decorate` option - 104668 <https://gerrit.ovirt.org/104668>: Document source cloning extension for containers 2. Artifact collection support - 104690 <https://gerrit.ovirt.org/104690>: Added NFS server container image - 104273 <https://gerrit.ovirt.org/104273>: STDCI PODS: Unique UID for each job build's POD - 104756 <https://gerrit.ovirt.org/104756>: pipeline-loader: refactor: separate podspec func - 104757 <https://gerrit.ovirt.org/104757>: pipeline-loader: refactor: Use podspec struct def - 104766 <https://gerrit.ovirt.org/104766>: STDCI PODs: Add artifact collection logic - 105522 <https://gerrit.ovirt.org/105522>: Documented artifact collection in containers 3. Extended log collection - 104842 <https://gerrit.ovirt.org/104842>: STDCI PODs: Add POD log collection - 105523 <https://gerrit.ovirt.org/105523>: Documented log collection in containers 4. Privileged container support - 104786 <https://gerrit.ovirt.org/104786>: STDCI DSL: Enable privileged containers 5. Support for using containers in gating jobs: - 104804 <https://gerrit.ovirt.org/104804>: standard-stage: refactor: move DSL to a library - 104811 <https://gerrit.ovirt.org/104811>: gate: Support getting suits from STDCI DSL 6. Providing the `extra_sources` file to OST suit containers: - 104843 <https://gerrit.ovirt.org/104843>: stdci_runner: Create extra_sources for PODs 7. Support for mirror injection and upstream source cloning - 104917 <https://gerrit.ovirt.org/104917>: Added a container with STDCI tools - 104918 <https://gerrit.ovirt.org/104918>: decorate.py: Add script - 104989 <https://gerrit.ovirt.org/104989>: STDCI DSL: Use `tools` container for `decorate` - 104994 <https://gerrit.ovirt.org/104994>: stdci_runner: Inject mirrors in PODs As you can see, we have quite a big pile of reviews to do, as always, help is very welcome... Regards, Barak. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
*Update: *We have now merged the automated cloning support feature, the currently merged code should already be applicable for a wide range of uses including running 'check-patch' workloads. On Thu, 12 Dec 2019 at 09:00, Barak Korren <bkorren@redhat.com> wrote:
A little less then a month ago I sent an email to this list telling you all about ongoing work to create a new container-based backend for the oVirt CI system.
I'm pleased to announce that we have managed to finally merged an initial set of patches implementing that backend yesterday, and it is now available for general use.
*What? Were? How do I use it?*
Documentation about how to use the new backend is now available in read the docs <https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index.html> .
*Wait! I needed it to do X which it doesn't!*
For the time being the new backend lacks some features that some may consider to be essential, such as automated cloning of patch source code and build artifact collection. We already have implemented patches providing a substantial amount of additional functionality, and hopefully we will be able to merge them soon. Following is a list of those patches and the features they implement:
1. Automated source cloning support: - 104213 <https://gerrit.ovirt.org/104213>: Implement STDCI DSL support for initContainers - 104590 <https://gerrit.ovirt.org/104590>: STDCI DSL: Add the `decorate` option - 104668 <https://gerrit.ovirt.org/104668>: Document source cloning extension for containers 2. Artifact collection support - 104690 <https://gerrit.ovirt.org/104690>: Added NFS server container image - 104273 <https://gerrit.ovirt.org/104273>: STDCI PODS: Unique UID for each job build's POD - 104756 <https://gerrit.ovirt.org/104756>: pipeline-loader: refactor: separate podspec func - 104757 <https://gerrit.ovirt.org/104757>: pipeline-loader: refactor: Use podspec struct def - 104766 <https://gerrit.ovirt.org/104766>: STDCI PODs: Add artifact collection logic - 105522 <https://gerrit.ovirt.org/105522>: Documented artifact collection in containers 3. Extended log collection - 104842 <https://gerrit.ovirt.org/104842>: STDCI PODs: Add POD log collection - 105523 <https://gerrit.ovirt.org/105523>: Documented log collection in containers 4. Privileged container support - 104786 <https://gerrit.ovirt.org/104786>: STDCI DSL: Enable privileged containers 5. Support for using containers in gating jobs: - 104804 <https://gerrit.ovirt.org/104804>: standard-stage: refactor: move DSL to a library - 104811 <https://gerrit.ovirt.org/104811>: gate: Support getting suits from STDCI DSL 6. Providing the `extra_sources` file to OST suit containers: - 104843 <https://gerrit.ovirt.org/104843>: stdci_runner: Create extra_sources for PODs 7. Support for mirror injection and upstream source cloning - 104917 <https://gerrit.ovirt.org/104917>: Added a container with STDCI tools - 104918 <https://gerrit.ovirt.org/104918>: decorate.py: Add script - 104989 <https://gerrit.ovirt.org/104989>: STDCI DSL: Use `tools` container for `decorate` - 104994 <https://gerrit.ovirt.org/104994>: stdci_runner: Inject mirrors in PODs
As you can see, we have quite a big pile of reviews to do, as always, help is very welcome...
Regards, Barak.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
*Update #2: *We have now merged all the patches that deal with artifact and log collection. And have updated the documentation <https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index.html> accordingly. The container-based backend should now be usable for the vast majority of the CI use cases. We do have some more features coming down the line geared towards more sophisticated use cases such as running OST suits and integrating with gating and change-queue flows. those Include: 1. Supporting the use of privileged containers 2. Invoking the container-based backed from the gating jobs 3. Generating and providing the `extra_sources` file 4. Runtime injection of YUM mirror URLs 5. Support for storing and using secret data such as password and auth tokens. I invite everyone to start moving workloads to the new system and enjoy the enhanced speed and reliability. On Sun, 15 Dec 2019 at 14:23, Barak Korren <bkorren@redhat.com> wrote:
*Update: *We have now merged the automated cloning support feature, the currently merged code should already be applicable for a wide range of uses including running 'check-patch' workloads.
On Thu, 12 Dec 2019 at 09:00, Barak Korren <bkorren@redhat.com> wrote:
A little less then a month ago I sent an email to this list telling you all about ongoing work to create a new container-based backend for the oVirt CI system.
I'm pleased to announce that we have managed to finally merged an initial set of patches implementing that backend yesterday, and it is now available for general use.
*What? Were? How do I use it?*
Documentation about how to use the new backend is now available in read the docs <https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index.html> .
*Wait! I needed it to do X which it doesn't!*
For the time being the new backend lacks some features that some may consider to be essential, such as automated cloning of patch source code and build artifact collection. We already have implemented patches providing a substantial amount of additional functionality, and hopefully we will be able to merge them soon. Following is a list of those patches and the features they implement:
1. Automated source cloning support: - 104213 <https://gerrit.ovirt.org/104213>: Implement STDCI DSL support for initContainers - 104590 <https://gerrit.ovirt.org/104590>: STDCI DSL: Add the `decorate` option - 104668 <https://gerrit.ovirt.org/104668>: Document source cloning extension for containers 2. Artifact collection support - 104690 <https://gerrit.ovirt.org/104690>: Added NFS server container image - 104273 <https://gerrit.ovirt.org/104273>: STDCI PODS: Unique UID for each job build's POD - 104756 <https://gerrit.ovirt.org/104756>: pipeline-loader: refactor: separate podspec func - 104757 <https://gerrit.ovirt.org/104757>: pipeline-loader: refactor: Use podspec struct def - 104766 <https://gerrit.ovirt.org/104766>: STDCI PODs: Add artifact collection logic - 105522 <https://gerrit.ovirt.org/105522>: Documented artifact collection in containers 3. Extended log collection - 104842 <https://gerrit.ovirt.org/104842>: STDCI PODs: Add POD log collection - 105523 <https://gerrit.ovirt.org/105523>: Documented log collection in containers 4. Privileged container support - 104786 <https://gerrit.ovirt.org/104786>: STDCI DSL: Enable privileged containers 5. Support for using containers in gating jobs: - 104804 <https://gerrit.ovirt.org/104804>: standard-stage: refactor: move DSL to a library - 104811 <https://gerrit.ovirt.org/104811>: gate: Support getting suits from STDCI DSL 6. Providing the `extra_sources` file to OST suit containers: - 104843 <https://gerrit.ovirt.org/104843>: stdci_runner: Create extra_sources for PODs 7. Support for mirror injection and upstream source cloning - 104917 <https://gerrit.ovirt.org/104917>: Added a container with STDCI tools - 104918 <https://gerrit.ovirt.org/104918>: decorate.py: Add script - 104989 <https://gerrit.ovirt.org/104989>: STDCI DSL: Use `tools` container for `decorate` - 104994 <https://gerrit.ovirt.org/104994>: stdci_runner: Inject mirrors in PODs
As you can see, we have quite a big pile of reviews to do, as always, help is very welcome...
Regards, Barak.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
On Tue, Dec 31, 2019 at 9:01 AM Barak Korren <bkorren@redhat.com> wrote:
*Update #2: *We have now merged all the patches that deal with artifact and log collection. And have updated the documentation <https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index.html> accordingly. The container-based backend should now be usable for the vast majority of the CI use cases.
We do have some more features coming down the line geared towards more sophisticated use cases such as running OST suits and integrating with gating and change-queue flows. those Include:
1. Supporting the use of privileged containers
What can I do to actually try this? I am failing because our container is not on the whitelist. Can you please point me in the right direction? Btw. The patch that I was trying this on is https://gerrit.ovirt.org/c/107831/
1. Invoking the container-based backed from the gating jobs 2. Generating and providing the `extra_sources` file 3. Runtime injection of YUM mirror URLs 4. Support for storing and using secret data such as password and auth tokens.
I invite everyone to start moving workloads to the new system and enjoy the enhanced speed and reliability.
On Sun, 15 Dec 2019 at 14:23, Barak Korren <bkorren@redhat.com> wrote:
*Update: *We have now merged the automated cloning support feature, the currently merged code should already be applicable for a wide range of uses including running 'check-patch' workloads.
On Thu, 12 Dec 2019 at 09:00, Barak Korren <bkorren@redhat.com> wrote:
A little less then a month ago I sent an email to this list telling you all about ongoing work to create a new container-based backend for the oVirt CI system.
I'm pleased to announce that we have managed to finally merged an initial set of patches implementing that backend yesterday, and it is now available for general use.
*What? Were? How do I use it?*
Documentation about how to use the new backend is now available in read the docs <https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index.html> .
*Wait! I needed it to do X which it doesn't!*
For the time being the new backend lacks some features that some may consider to be essential, such as automated cloning of patch source code and build artifact collection. We already have implemented patches providing a substantial amount of additional functionality, and hopefully we will be able to merge them soon. Following is a list of those patches and the features they implement:
1. Automated source cloning support: - 104213 <https://gerrit.ovirt.org/104213>: Implement STDCI DSL support for initContainers - 104590 <https://gerrit.ovirt.org/104590>: STDCI DSL: Add the `decorate` option - 104668 <https://gerrit.ovirt.org/104668>: Document source cloning extension for containers 2. Artifact collection support - 104690 <https://gerrit.ovirt.org/104690>: Added NFS server container image - 104273 <https://gerrit.ovirt.org/104273>: STDCI PODS: Unique UID for each job build's POD - 104756 <https://gerrit.ovirt.org/104756>: pipeline-loader: refactor: separate podspec func - 104757 <https://gerrit.ovirt.org/104757>: pipeline-loader: refactor: Use podspec struct def - 104766 <https://gerrit.ovirt.org/104766>: STDCI PODs: Add artifact collection logic - 105522 <https://gerrit.ovirt.org/105522>: Documented artifact collection in containers 3. Extended log collection - 104842 <https://gerrit.ovirt.org/104842>: STDCI PODs: Add POD log collection - 105523 <https://gerrit.ovirt.org/105523>: Documented log collection in containers 4. Privileged container support - 104786 <https://gerrit.ovirt.org/104786>: STDCI DSL: Enable privileged containers 5. Support for using containers in gating jobs: - 104804 <https://gerrit.ovirt.org/104804>: standard-stage: refactor: move DSL to a library - 104811 <https://gerrit.ovirt.org/104811>: gate: Support getting suits from STDCI DSL 6. Providing the `extra_sources` file to OST suit containers: - 104843 <https://gerrit.ovirt.org/104843>: stdci_runner: Create extra_sources for PODs 7. Support for mirror injection and upstream source cloning - 104917 <https://gerrit.ovirt.org/104917>: Added a container with STDCI tools - 104918 <https://gerrit.ovirt.org/104918>: decorate.py: Add script - 104989 <https://gerrit.ovirt.org/104989>: STDCI DSL: Use `tools` container for `decorate` - 104994 <https://gerrit.ovirt.org/104994>: stdci_runner: Inject mirrors in PODs
As you can see, we have quite a big pile of reviews to do, as always, help is very welcome...
Regards, Barak.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HYZVFWHYDNUY73...
-- Ales Musil Software Engineer - RHV Network Red Hat EMEA <https://www.redhat.com> amusil@redhat.com IM: amusil <https://red.ht/sig>
On Tue, Mar 24, 2020 at 1:50 PM Ales Musil <amusil@redhat.com> wrote:
On Tue, Dec 31, 2019 at 9:01 AM Barak Korren <bkorren@redhat.com> wrote:
*Update #2: *We have now merged all the patches that deal with artifact and log collection. And have updated the documentation <https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index.html> accordingly. The container-based backend should now be usable for the vast majority of the CI use cases.
We do have some more features coming down the line geared towards more sophisticated use cases such as running OST suits and integrating with gating and change-queue flows. those Include:
1. Supporting the use of privileged containers
What can I do to actually try this? I am failing because our container is not on the whitelist. Can you please point me in the right direction? Btw. The patch that I was trying this on is https://gerrit.ovirt.org/c/107831/
Any update on this?
1. Invoking the container-based backed from the gating jobs 2. Generating and providing the `extra_sources` file 3. Runtime injection of YUM mirror URLs 4. Support for storing and using secret data such as password and auth tokens.
I invite everyone to start moving workloads to the new system and enjoy the enhanced speed and reliability.
On Sun, 15 Dec 2019 at 14:23, Barak Korren <bkorren@redhat.com> wrote:
*Update: *We have now merged the automated cloning support feature, the currently merged code should already be applicable for a wide range of uses including running 'check-patch' workloads.
On Thu, 12 Dec 2019 at 09:00, Barak Korren <bkorren@redhat.com> wrote:
A little less then a month ago I sent an email to this list telling you all about ongoing work to create a new container-based backend for the oVirt CI system.
I'm pleased to announce that we have managed to finally merged an initial set of patches implementing that backend yesterday, and it is now available for general use.
*What? Were? How do I use it?*
Documentation about how to use the new backend is now available in read the docs <https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index.html> .
*Wait! I needed it to do X which it doesn't!*
For the time being the new backend lacks some features that some may consider to be essential, such as automated cloning of patch source code and build artifact collection. We already have implemented patches providing a substantial amount of additional functionality, and hopefully we will be able to merge them soon. Following is a list of those patches and the features they implement:
1. Automated source cloning support: - 104213 <https://gerrit.ovirt.org/104213>: Implement STDCI DSL support for initContainers - 104590 <https://gerrit.ovirt.org/104590>: STDCI DSL: Add the `decorate` option - 104668 <https://gerrit.ovirt.org/104668>: Document source cloning extension for containers 2. Artifact collection support - 104690 <https://gerrit.ovirt.org/104690>: Added NFS server container image - 104273 <https://gerrit.ovirt.org/104273>: STDCI PODS: Unique UID for each job build's POD - 104756 <https://gerrit.ovirt.org/104756>: pipeline-loader: refactor: separate podspec func - 104757 <https://gerrit.ovirt.org/104757>: pipeline-loader: refactor: Use podspec struct def - 104766 <https://gerrit.ovirt.org/104766>: STDCI PODs: Add artifact collection logic - 105522 <https://gerrit.ovirt.org/105522>: Documented artifact collection in containers 3. Extended log collection - 104842 <https://gerrit.ovirt.org/104842>: STDCI PODs: Add POD log collection - 105523 <https://gerrit.ovirt.org/105523>: Documented log collection in containers 4. Privileged container support - 104786 <https://gerrit.ovirt.org/104786>: STDCI DSL: Enable privileged containers 5. Support for using containers in gating jobs: - 104804 <https://gerrit.ovirt.org/104804>: standard-stage: refactor: move DSL to a library - 104811 <https://gerrit.ovirt.org/104811>: gate: Support getting suits from STDCI DSL 6. Providing the `extra_sources` file to OST suit containers: - 104843 <https://gerrit.ovirt.org/104843>: stdci_runner: Create extra_sources for PODs 7. Support for mirror injection and upstream source cloning - 104917 <https://gerrit.ovirt.org/104917>: Added a container with STDCI tools - 104918 <https://gerrit.ovirt.org/104918>: decorate.py: Add script - 104989 <https://gerrit.ovirt.org/104989>: STDCI DSL: Use `tools` container for `decorate` - 104994 <https://gerrit.ovirt.org/104994>: stdci_runner: Inject mirrors in PODs
As you can see, we have quite a big pile of reviews to do, as always, help is very welcome...
Regards, Barak.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HYZVFWHYDNUY73...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil <https://red.ht/sig>
-- Ales Musil Software Engineer - RHV Network Red Hat EMEA <https://www.redhat.com> amusil@redhat.com IM: amusil <https://red.ht/sig>
participants (2)
-
Ales Musil -
Barak Korren