[JIRA] (OVIRT-1867) Allow embedded secrets inside the source repo for CI
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1867?page=com.atlassian.jir... ]
Barak Korren commented on OVIRT-1867:
-------------------------------------
[~rmohr(a)redhat.com] the main issue I have with the concept you suggest is that it chains the code to a specific instance of the CI system.
The point of STDCI is to be a standard - you can take a compliant project and build/test it the same way on different CI systems. They way you suggest to handle secrets - it essentially chains the project to a specific CI system instance - the one that knows the right private key.
This concept makes perfect sense for PAAS providers like Travis that want to lock you into the single instance their platform...
Our view of credentials is also slightly different - instead of a developer providing his own credentials for using service X, he just asks for access for service X, and it becomes the CI system's responsibility to figure out how to provide access to that service.
Having said the above, implementing what you ask for is not difficult, so we may add this soon as an additional feature for our existing credentials support. The main challenge would be to find where to store all the private keys and provide access to the public keys. Our system doesn't really have a UI that is not linked to a specific build/test run, since so far the assumption has always been that all communication with the CI system is done via commits or comments to the SCM.
> Allow embedded secrets inside the source repo for CI
> ----------------------------------------------------
>
> Key: OVIRT-1867
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1867
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Components: Standard CI (Pipelines), STDCI DSL
> Reporter: Roman Mohr
> Assignee: infra
>
> In order to improve the self-service capabilities of standard-ci it is
> important for projects, that they can add their own secrets to projects (to
> reach external services, e.g. docker hub, ...).
> Travis has a very nice system which helps engineers there:
> https://docs.travis-ci.com/user/encryption-keys/
> Basically the CI system needs to generate a public/private key pair for
> every enabled git repo. The engineer simply fetches the public key via a
> well know URL and encrypts the secrets. Then the encrypted secret can be
> made part of the source repo. Before the tests are run the CI system
> decrypts the secrets. Than can play together pretty well with Jenkinsfiles
> too.
> Benefit:
> * Less manual intervention from CI team to add secrets to jobs
> * Strengthen the config-in-code thinking
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100077)
6 years, 9 months
Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1224
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
Changes:
[Sandro Bonazzola] copy-create-job: drop EOLed Fedoras
[Dafna Ron] otopi - dropping fc24
[Dafna Ron] ovirt-hosted-engine-setup - dropping fc24
[Daniel Belenky] Mock runner to search full script path
[Sandro Bonazzola] otopi: 4.2 and latest fedora
[Sandro Bonazzola] ovirt-hosted-engine-setup: 4.2 and new Fedoras
[Sandro Bonazzola] ovirt-live: drop project
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_6...>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 453d3f78c51516ce19b686b9ac141d2ced2260d7 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 453d3f78c51516ce19b686b9ac141d2ced2260d7
Commit message: "ovirt-live: drop project"
> git rev-list ffe020fb18b7f872bc817ba9eec0c0d5a82bc489 # timeout=10
[system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins4363023534009612747.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 112, in retrieveAllMD
repo._revertOldRepoXML()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1331, in _revertOldRepoXML
os.rename(old_data['old_local'], old_data['local'])
OSError: [Errno 2] No such file or directory
Build step 'Execute shell' marked build as failure
6 years, 9 months
[JIRA] (OVIRT-1867) Allow embedded secrets inside the source repo for CI
by Barak Korren (oVirt JIRA)
This is a multi-part message in MIME format...
------------=_1517318718-26640-235
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1867?page=com.atlassian.jir... ]
Barak Korren updated OVIRT-1867:
--------------------------------
Epic Link: OVIRT-400
Issue Type: New Feature (was: By-EMAIL)
> Allow embedded secrets inside the source repo for CI
> ----------------------------------------------------
>
> Key: OVIRT-1867
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1867
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Components: Standard CI (Pipelines), STDCI DSL
> Reporter: Roman Mohr
> Assignee: infra
>
> In order to improve the self-service capabilities of standard-ci it is
> important for projects, that they can add their own secrets to projects (to
> reach external services, e.g. docker hub, ...).
> Travis has a very nice system which helps engineers there:
> https://docs.travis-ci.com/user/encryption-keys/
> Basically the CI system needs to generate a public/private key pair for
> every enabled git repo. The engineer simply fetches the public key via a
> well know URL and encrypts the secrets. Then the encrypted secret can be
> made part of the source repo. Before the tests are run the CI system
> decrypts the secrets. Than can play together pretty well with Jenkinsfiles
> too.
> Benefit:
> * Less manual intervention from CI team to add secrets to jobs
> * Strengthen the config-in-code thinking
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100077)
------------=_1517318718-26640-235
Content-Type: text/html; charset="UTF-8"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
<html><body>
<pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1867?page=com.atlassian.jir... ]</pre>
<h3>Barak Korren updated OVIRT-1867:</h3>
<pre> Epic Link: OVIRT-400
Issue Type: New Feature (was: By-EMAIL)</pre>
<blockquote><h3>Allow embedded secrets inside the source repo for CI</h3>
<pre> Key: OVIRT-1867
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1867
Project: oVirt - virtualization made easy
Issue Type: New Feature
Components: Standard CI (Pipelines), STDCI DSL
Reporter: Roman Mohr
Assignee: infra</pre>
<p>In order to improve the self-service capabilities of standard-ci it is important for projects, that they can add their own secrets to projects (to reach external services, e.g. docker hub, …). Travis has a very nice system which helps engineers there: <a href="https://docs.travis-ci.com/user/encryption-keys/">https://docs.travis-ci.com/user/encryption-keys/</a> Basically the CI system needs to generate a public/private key pair for every enabled git repo. The engineer simply fetches the public key via a well know URL and encrypts the secrets. Then the encrypted secret can be made part of the source repo. Before the tests are run the CI system decrypts the secrets. Than can play together pretty well with Jenkinsfiles too. Benefit:</p>
<pre>* Less manual intervention from CI team to add secrets to jobs
* Strengthen the config-in-code thinking</pre></blockquote>
<p>— This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100077)</p>
<img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BJ33BS..." alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/>
</body></html>
------------=_1517318718-26640-235--
6 years, 9 months
[JIRA] (OVIRT-1867) Allow embedded secrets inside the source repo for CI
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1867?page=com.atlassian.jir... ]
Barak Korren updated OVIRT-1867:
--------------------------------
Epic Link: OVIRT-400
> Allow embedded secrets inside the source repo for CI
> ----------------------------------------------------
>
> Key: OVIRT-1867
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1867
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Components: Standard CI (Pipelines), STDCI DSL
> Reporter: Roman Mohr
> Assignee: infra
>
> In order to improve the self-service capabilities of standard-ci it is
> important for projects, that they can add their own secrets to projects (to
> reach external services, e.g. docker hub, ...).
> Travis has a very nice system which helps engineers there:
> https://docs.travis-ci.com/user/encryption-keys/
> Basically the CI system needs to generate a public/private key pair for
> every enabled git repo. The engineer simply fetches the public key via a
> well know URL and encrypts the secrets. Then the encrypted secret can be
> made part of the source repo. Before the tests are run the CI system
> decrypts the secrets. Than can play together pretty well with Jenkinsfiles
> too.
> Benefit:
> * Less manual intervention from CI team to add secrets to jobs
> * Strengthen the config-in-code thinking
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100077)
6 years, 9 months
[JIRA] (OVIRT-1868) Allow engineers to write Jenkinsfiles
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1868?page=com.atlassian.jir... ]
Barak Korren updated OVIRT-1868:
--------------------------------
Component/s: Standard CI (Pipelines)
> Allow engineers to write Jenkinsfiles
> -------------------------------------
>
> Key: OVIRT-1868
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1868
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Components: Standard CI (Pipelines)
> Reporter: Roman Mohr
> Assignee: infra
>
> Looks like standard-ci switched internally to use Jenkinsfiles. However it
> would be very valuable for engineers, if they could just write their
> Jenkinsfile, instead of all the usual standard-ci yamls/scripts.
> With the Jenkinsfile the chroot based approach seems to be pretty obsolete,
> if you allow people to use the docker agent for the Jenkinsfile. For
> KubeVirt it would make standard-ci finally really valuable.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100077)
6 years, 9 months
[JIRA] (OVIRT-1868) Allow engineers to write Jenkinsfiles
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1868?page=com.atlassian.jir... ]
Barak Korren updated OVIRT-1868:
--------------------------------
Epic Link: OVIRT-400
> Allow engineers to write Jenkinsfiles
> -------------------------------------
>
> Key: OVIRT-1868
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1868
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Reporter: Roman Mohr
> Assignee: infra
>
> Looks like standard-ci switched internally to use Jenkinsfiles. However it
> would be very valuable for engineers, if they could just write their
> Jenkinsfile, instead of all the usual standard-ci yamls/scripts.
> With the Jenkinsfile the chroot based approach seems to be pretty obsolete,
> if you allow people to use the docker agent for the Jenkinsfile. For
> KubeVirt it would make standard-ci finally really valuable.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100077)
6 years, 9 months
[JIRA] (OVIRT-1868) Allow engineers to write Jenkinsfiles
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1868?page=com.atlassian.jir... ]
Barak Korren updated OVIRT-1868:
--------------------------------
Epic Link: OVIRT-400
> Allow engineers to write Jenkinsfiles
> -------------------------------------
>
> Key: OVIRT-1868
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1868
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Reporter: Roman Mohr
> Assignee: infra
>
> Looks like standard-ci switched internally to use Jenkinsfiles. However it
> would be very valuable for engineers, if they could just write their
> Jenkinsfile, instead of all the usual standard-ci yamls/scripts.
> With the Jenkinsfile the chroot based approach seems to be pretty obsolete,
> if you allow people to use the docker agent for the Jenkinsfile. For
> KubeVirt it would make standard-ci finally really valuable.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100077)
6 years, 9 months
[JIRA] (OVIRT-1868) Allow engineers to write Jenkinsfiles
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1868?page=com.atlassian.jir... ]
Barak Korren updated OVIRT-1868:
--------------------------------
Issue Type: New Feature (was: By-EMAIL)
> Allow engineers to write Jenkinsfiles
> -------------------------------------
>
> Key: OVIRT-1868
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1868
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Reporter: Roman Mohr
> Assignee: infra
>
> Looks like standard-ci switched internally to use Jenkinsfiles. However it
> would be very valuable for engineers, if they could just write their
> Jenkinsfile, instead of all the usual standard-ci yamls/scripts.
> With the Jenkinsfile the chroot based approach seems to be pretty obsolete,
> if you allow people to use the docker agent for the Jenkinsfile. For
> KubeVirt it would make standard-ci finally really valuable.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100077)
6 years, 9 months
[JIRA] (OVIRT-1868) Allow engineers to write Jenkinsfiles
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1868?page=com.atlassian.jir... ]
Barak Korren commented on OVIRT-1868:
-------------------------------------
{quote}
Looks like standard-ci switched internally to use Jenkinsfiles. However it
would be very valuable for engineers, if they could just write their
Jenkinsfile, instead of all the usual standard-ci yamls/scripts.
{quote}
We use pipelines - Jenkinfiles are pipeline-based but also impose a very specific workflow, while generic pipelines are far more flexible.
{quote}
With the Jenkinsfile the chroot based approach seems to be pretty obsolete,
{quote}
Nope,there are quite a few things the current approach makes easy to do that Jenkinsfile would make complex or impossible.
The main point of the current approach is not the use of mock/chroot, which is an implementation detail, but the use of a standardized generic configuration format, where different backends could be implemented.
It allows the CI team to deal effectively with how CI resources are distributed and managed while allowing developers to just specify what they need. With Jenkinsfile the developers would have to deal with a lot of the mechanics or resource allocation and management, while the CI team will be unable to control how resources are used or to ensure they are returned to usable state when jobs finish.
The configuration formal also allows the CI system to gain a good knowledge about the dependencies for a given test/build process. This allows it to to control how and which versions of these dependencies are provided. (For example, when you ask for yum repositories, the CI system ensure you get the same view of these repositories from the point in time in which the jobs started, even if they get updated while the job is running).
Jenkinsfile, as it stands, will also chain us to a very specific CI orchestration engine. Don't get me wrong, Jenkins is nice, but its good to know we can switch the backend and have that switch be mostly transparent to project developers. Its also very convenient to have a client-side mini-backend, it would be quite impossible to make such a backend for a Jenkinsfile.
{quote}
if you allow people to use the docker agent for the Jenkinsfile. For
KubeVirt it would make standard-ci finally really valuable.
{quote}
STDCI have been quite valuable so far TYVM.
Using the docker agent as you suggest will actually make it lees so because it will chain you to a very specific method of resource allocation. It will make it harder to do things like ensure a certain test script gets its own bare-metal machine to run on.
Docker containers are useful - but many people couldn't care less, they just know they need to target 'el7' and need a few dependencies for that, forcing then to then deal with finding or making a suitable container image will just divert efforts from the more important work of writing the actual code to be tested.
The main thing that Jenkinsfiles have to offer that STDCI does not support currently is the ability to sequentially chain multiple steps together and have control flow to decide which steps get executed. We are going to make STDCI provide that, and we're going to try to do that in a syntax that would be familiar to anyone who is familiar with Jenkinsfiles. But it is going to be a subset of the Jenkinsfie syntax. Things like node() for example, are probably not going to be supported since STDCI already provides higher-level ways of requesting node allocation.
> Allow engineers to write Jenkinsfiles
> -------------------------------------
>
> Key: OVIRT-1868
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1868
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Roman Mohr
> Assignee: infra
>
> Looks like standard-ci switched internally to use Jenkinsfiles. However it
> would be very valuable for engineers, if they could just write their
> Jenkinsfile, instead of all the usual standard-ci yamls/scripts.
> With the Jenkinsfile the chroot based approach seems to be pretty obsolete,
> if you allow people to use the docker agent for the Jenkinsfile. For
> KubeVirt it would make standard-ci finally really valuable.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100077)
6 years, 9 months
