- Infra - oVirt List Archives

[JIRA] (OVIRT-1867) Allow embedded secrets inside the source repo for CI
by Barak Korren (oVirt JIRA) 30 Jan '18

30 Jan '18

This is a multi-part message in MIME format... ------------=_1517318718-26640-235 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit [ https://ovirt-jira.atlassian.net/browse/OVIRT-1867?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1867: -------------------------------- Epic Link: OVIRT-400 Issue Type: New Feature (was: By-EMAIL) > Allow embedded secrets inside the source repo for CI > ---------------------------------------------------- > > Key: OVIRT-1867 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1867 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Components: Standard CI (Pipelines), STDCI DSL > Reporter: Roman Mohr > Assignee: infra > > In order to improve the self-service capabilities of standard-ci it is > important for projects, that they can add their own secrets to projects (to > reach external services, e.g. docker hub, ...). > Travis has a very nice system which helps engineers there: > https://docs.travis-ci.com/user/encryption-keys/ > Basically the CI system needs to generate a public/private key pair for > every enabled git repo. The engineer simply fetches the public key via a > well know URL and encrypts the secrets. Then the encrypted secret can be > made part of the source repo. Before the tests are run the CI system > decrypts the secrets. Than can play together pretty well with Jenkinsfiles > too. > Benefit: > * Less manual intervention from CI team to add secrets to jobs > * Strengthen the config-in-code thinking -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100077) ------------=_1517318718-26640-235 Content-Type: text/html; charset="UTF-8" Content-Disposition: inline Content-Transfer-Encoding: 7bit <html><body> <pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1867?page=com.atlassian.jira.… ]</pre> <h3>Barak Korren updated OVIRT-1867:</h3> <pre> Epic Link: OVIRT-400 Issue Type: New Feature (was: By-EMAIL)</pre> <blockquote><h3>Allow embedded secrets inside the source repo for CI</h3> <pre> Key: OVIRT-1867 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1867 Project: oVirt - virtualization made easy Issue Type: New Feature Components: Standard CI (Pipelines), STDCI DSL Reporter: Roman Mohr Assignee: infra</pre> <p>In order to improve the self-service capabilities of standard-ci it is important for projects, that they can add their own secrets to projects (to reach external services, e.g. docker hub, …). Travis has a very nice system which helps engineers there: <a href="https://docs.travis-ci.com/user/encryption-keys/">https://docs.travis-ci.com/user/encryption-keys/</a> Basically the CI system needs to generate a public/private key pair for every enabled git repo. The engineer simply fetches the public key via a well know URL and encrypts the secrets. Then the encrypted secret can be made part of the source repo. Before the tests are run the CI system decrypts the secrets. Than can play together pretty well with Jenkinsfiles too. Benefit:</p> <pre>* Less manual intervention from CI team to add secrets to jobs * Strengthen the config-in-code thinking</pre></blockquote> <p>— This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100077)</p> <img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BJ33BSM3…" alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/> </body></html> ------------=_1517318718-26640-235--

1 0

[JIRA] (OVIRT-1867) Allow embedded secrets inside the source repo for CI
by Barak Korren (oVirt JIRA) 30 Jan '18

30 Jan '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1867?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1867: -------------------------------- Epic Link: OVIRT-400 > Allow embedded secrets inside the source repo for CI > ---------------------------------------------------- > > Key: OVIRT-1867 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1867 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Components: Standard CI (Pipelines), STDCI DSL > Reporter: Roman Mohr > Assignee: infra > > In order to improve the self-service capabilities of standard-ci it is > important for projects, that they can add their own secrets to projects (to > reach external services, e.g. docker hub, ...). > Travis has a very nice system which helps engineers there: > https://docs.travis-ci.com/user/encryption-keys/ > Basically the CI system needs to generate a public/private key pair for > every enabled git repo. The engineer simply fetches the public key via a > well know URL and encrypts the secrets. Then the encrypted secret can be > made part of the source repo. Before the tests are run the CI system > decrypts the secrets. Than can play together pretty well with Jenkinsfiles > too. > Benefit: > * Less manual intervention from CI team to add secrets to jobs > * Strengthen the config-in-code thinking -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100077)

1 0

[JIRA] (OVIRT-1868) Allow engineers to write Jenkinsfiles
by Barak Korren (oVirt JIRA) 30 Jan '18

30 Jan '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1868?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1868: -------------------------------- Component/s: Standard CI (Pipelines) > Allow engineers to write Jenkinsfiles > ------------------------------------- > > Key: OVIRT-1868 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1868 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Components: Standard CI (Pipelines) > Reporter: Roman Mohr > Assignee: infra > > Looks like standard-ci switched internally to use Jenkinsfiles. However it > would be very valuable for engineers, if they could just write their > Jenkinsfile, instead of all the usual standard-ci yamls/scripts. > With the Jenkinsfile the chroot based approach seems to be pretty obsolete, > if you allow people to use the docker agent for the Jenkinsfile. For > KubeVirt it would make standard-ci finally really valuable. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100077)

1 0

[JIRA] (OVIRT-1868) Allow engineers to write Jenkinsfiles
by Barak Korren (oVirt JIRA) 30 Jan '18

30 Jan '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1868?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1868: -------------------------------- Epic Link: OVIRT-400 > Allow engineers to write Jenkinsfiles > ------------------------------------- > > Key: OVIRT-1868 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1868 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Reporter: Roman Mohr > Assignee: infra > > Looks like standard-ci switched internally to use Jenkinsfiles. However it > would be very valuable for engineers, if they could just write their > Jenkinsfile, instead of all the usual standard-ci yamls/scripts. > With the Jenkinsfile the chroot based approach seems to be pretty obsolete, > if you allow people to use the docker agent for the Jenkinsfile. For > KubeVirt it would make standard-ci finally really valuable. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100077)

1 0

[JIRA] (OVIRT-1868) Allow engineers to write Jenkinsfiles
by Barak Korren (oVirt JIRA) 30 Jan '18

30 Jan '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1868?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1868: -------------------------------- Epic Link: OVIRT-400 > Allow engineers to write Jenkinsfiles > ------------------------------------- > > Key: OVIRT-1868 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1868 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Reporter: Roman Mohr > Assignee: infra > > Looks like standard-ci switched internally to use Jenkinsfiles. However it > would be very valuable for engineers, if they could just write their > Jenkinsfile, instead of all the usual standard-ci yamls/scripts. > With the Jenkinsfile the chroot based approach seems to be pretty obsolete, > if you allow people to use the docker agent for the Jenkinsfile. For > KubeVirt it would make standard-ci finally really valuable. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100077)

1 0

[JIRA] (OVIRT-1868) Allow engineers to write Jenkinsfiles
by Barak Korren (oVirt JIRA) 30 Jan '18

30 Jan '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1868?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1868: -------------------------------- Issue Type: New Feature (was: By-EMAIL) > Allow engineers to write Jenkinsfiles > ------------------------------------- > > Key: OVIRT-1868 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1868 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Reporter: Roman Mohr > Assignee: infra > > Looks like standard-ci switched internally to use Jenkinsfiles. However it > would be very valuable for engineers, if they could just write their > Jenkinsfile, instead of all the usual standard-ci yamls/scripts. > With the Jenkinsfile the chroot based approach seems to be pretty obsolete, > if you allow people to use the docker agent for the Jenkinsfile. For > KubeVirt it would make standard-ci finally really valuable. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100077)

1 0

[JIRA] (OVIRT-1868) Allow engineers to write Jenkinsfiles
by Barak Korren (oVirt JIRA) 30 Jan '18

30 Jan '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1868?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-1868: ------------------------------------- {quote} Looks like standard-ci switched internally to use Jenkinsfiles. However it would be very valuable for engineers, if they could just write their Jenkinsfile, instead of all the usual standard-ci yamls/scripts. {quote} We use pipelines - Jenkinfiles are pipeline-based but also impose a very specific workflow, while generic pipelines are far more flexible. {quote} With the Jenkinsfile the chroot based approach seems to be pretty obsolete, {quote} Nope,there are quite a few things the current approach makes easy to do that Jenkinsfile would make complex or impossible. The main point of the current approach is not the use of mock/chroot, which is an implementation detail, but the use of a standardized generic configuration format, where different backends could be implemented. It allows the CI team to deal effectively with how CI resources are distributed and managed while allowing developers to just specify what they need. With Jenkinsfile the developers would have to deal with a lot of the mechanics or resource allocation and management, while the CI team will be unable to control how resources are used or to ensure they are returned to usable state when jobs finish. The configuration formal also allows the CI system to gain a good knowledge about the dependencies for a given test/build process. This allows it to to control how and which versions of these dependencies are provided. (For example, when you ask for yum repositories, the CI system ensure you get the same view of these repositories from the point in time in which the jobs started, even if they get updated while the job is running). Jenkinsfile, as it stands, will also chain us to a very specific CI orchestration engine. Don't get me wrong, Jenkins is nice, but its good to know we can switch the backend and have that switch be mostly transparent to project developers. Its also very convenient to have a client-side mini-backend, it would be quite impossible to make such a backend for a Jenkinsfile. {quote} if you allow people to use the docker agent for the Jenkinsfile. For KubeVirt it would make standard-ci finally really valuable. {quote} STDCI have been quite valuable so far TYVM. Using the docker agent as you suggest will actually make it lees so because it will chain you to a very specific method of resource allocation. It will make it harder to do things like ensure a certain test script gets its own bare-metal machine to run on. Docker containers are useful - but many people couldn't care less, they just know they need to target 'el7' and need a few dependencies for that, forcing then to then deal with finding or making a suitable container image will just divert efforts from the more important work of writing the actual code to be tested. The main thing that Jenkinsfiles have to offer that STDCI does not support currently is the ability to sequentially chain multiple steps together and have control flow to decide which steps get executed. We are going to make STDCI provide that, and we're going to try to do that in a syntax that would be familiar to anyone who is familiar with Jenkinsfiles. But it is going to be a subset of the Jenkinsfie syntax. Things like node() for example, are probably not going to be supported since STDCI already provides higher-level ways of requesting node allocation. > Allow engineers to write Jenkinsfiles > ------------------------------------- > > Key: OVIRT-1868 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1868 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Roman Mohr > Assignee: infra > > Looks like standard-ci switched internally to use Jenkinsfiles. However it > would be very valuable for engineers, if they could just write their > Jenkinsfile, instead of all the usual standard-ci yamls/scripts. > With the Jenkinsfile the chroot based approach seems to be pretty obsolete, > if you allow people to use the docker agent for the Jenkinsfile. For > KubeVirt it would make standard-ci finally really valuable. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100077)

1 0

[CQ]: 86903, 1 (ovirt-engine-api-model) failed "ovirt-4.2" system tests, but isn't the failure root cause
by oVirt Jenkins 30 Jan '18

30 Jan '18

A system test invoked by the "ovirt-4.2" change queue including change 86903,1 (ovirt-engine-api-model) failed. However, this change seems not to be the root cause for this failure. Change 86692,1 (ovirt-engine-api-model) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 86692,1 (ovirt-engine-api- model) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/86903/1 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/86692/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/372/

1 0

[JIRA] (OVIRT-1867) Allow embedded secrets inside the source repo for CI
by Roman Mohr (oVirt JIRA) 30 Jan '18

30 Jan '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1867?page=com.atlassian.jira.… ] Roman Mohr commented on OVIRT-1867: ----------------------------------- In my opinion, having the secrets encrypted in the code is better, since you don't need accounts for engineers on CI to do that. A public rsa key can be made accessible for the whole world. With other words, the issue is exactly about the self-service and not about how to bind existing secrets. It is also related to OVIRT-1868. In combination with a Jenkinsfile, pretty much the whole yaml/script/chroot can be made obsolete, however that is a different story. > Allow embedded secrets inside the source repo for CI > ---------------------------------------------------- > > Key: OVIRT-1867 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1867 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Components: Standard CI (Pipelines), STDCI DSL > Reporter: Roman Mohr > Assignee: infra > > In order to improve the self-service capabilities of standard-ci it is > important for projects, that they can add their own secrets to projects (to > reach external services, e.g. docker hub, ...). > Travis has a very nice system which helps engineers there: > https://docs.travis-ci.com/user/encryption-keys/ > Basically the CI system needs to generate a public/private key pair for > every enabled git repo. The engineer simply fetches the public key via a > well know URL and encrypts the secrets. Then the encrypted secret can be > made part of the source repo. Before the tests are run the CI system > decrypts the secrets. Than can play together pretty well with Jenkinsfiles > too. > Benefit: > * Less manual intervention from CI team to add secrets to jobs > * Strengthen the config-in-code thinking -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100077)

1 0

[CQ]: 86869, 1 (otopi) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 30 Jan '18

30 Jan '18

A system test invoked by the "ovirt-master" change queue including change 86869,1 (otopi) failed. However, this change seems not to be the root cause for this failure. Change 86679,4 (otopi) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 86679,4 (otopi) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/86869/1 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/86679/4 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5160/

1 0