<div dir="ltr">Hey,<div><br></div><div>I just got around to studying this.</div><div><br></div><div>- Nice clear email!</div><div>- Everything really makes sense.</div><div>- Thank you for fixing the -excludes thing in the yaml. That was rough :)</div><div>- <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">The graph view in Blue Ocean is easy to see and understand.</span></div><div>- "<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">We now support “sub stages” which provide the ability to run multiple</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> different scripts in parallel" -- what kind of races should we watch out for? :) For example in OST, I think I'll have to adapt docker stuff to be aware that another set of containers could be running at the same time -- not positive though.</span></div><div><br></div><div>It looks like the substages replace change_resolver in OST. Can you go into that in more detail? How does this impact running run mock_runner locally? When I run it locally it doesn't appear to paralleilize like it does in jenkins / Blue Ocean.</div><div><br></div><div>Best wishes,</div><div>Greg</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 16, 2018 at 10:17 AM, Barak Korren <span dir="ltr"><<a href="mailto:bkorren@redhat.com" target="_blank">bkorren@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The CI team is thrilled to announce the general availability of the second<br>
version of the oVirt CI standard. Work on this version included almost a<br>
complete rewrite of the CI backend. The major user-visible features are:<br>
<br>
- Project maintainers no longer need to maintain YAML in the ‘jenkins’<br>
repository. Details that were specified there, including targeted<br>
distributions, architectures and oVirt versions should now be specified in a<br>
YAML file under the project’s own repository (In a different syntax).<br>
<br>
- We now support “sub stages” which provide the ability to run multiple<br>
different scripts in parallel within the same STDCI stage. There is also a<br>
conditional syntax which allows controlling which scripts get executed<br>
according to which files were changed in the patch being tested.<br>
<br>
- The STDCI script file names and locations can now be customized via the above<br>
mentioned YAML file. This means that for e.g. using the same script for<br>
different stages can now be done by assigning it to the stages in the YAML<br>
file instead of by using symlinks.<br>
<br>
Inspecting job results in STDCI V2<br>
------------------------------<wbr>----<br>
As already mentioned, the work on STDCI V2 consisted of a major rewrite of the<br>
CI backend, one of the changes made is switching from using multiple “FreeStyle”<br>
type jobs per project to just two (pre-merge and post-merge) pipeline jobs. This<br>
has implications for the way job results are to be inspected.<br>
<br>
Since all the different parallel tasks now happen within the same job, looking<br>
at the job output can be rather confusing as it includes the merged output of<br>
all the tasks. Instead, the “Blue Ocean” view should be used. The “Blue Ocean”<br>
view displays a graphical layout of the job execution allowing one to quickly<br>
learn which parts of the job failed. It also allows drilling down and viewing<br>
the logs of individual parts of the job.<br>
<br>
Apart from using the “Blue Ocean” view, job logs are also stored as artifact<br>
files. The ‘exported-artifacts’ directory seen in the job results will now<br>
include different subdirectories for the different parts of the job. Assuming we<br>
have a ‘check-patch’ stage script running on ‘el7/x86_64’, we can find its<br>
output under ‘exported-artifacts’ in:<br>
<br>
check-patch.el7.x86_64/mock_<wbr>logs/script/stdout_stderr.log<br>
<br>
Any additional artifacts generated by the script would be present in the<br>
‘check-patch.el7.x86-64’ directory as well.<br>
<br>
I have a CI YAML file in my project already, is this really new?<br>
------------------------------<wbr>------------------------------<wbr>----<br>
We’ve been working on this for a while, and occasionally introduced V2 features<br>
into individual projects as needed. In particular, our GitHub support was always<br>
based on STDCI V2 code, so all GitHub projects (except Lago, which is<br>
‘special’…) are already using STDCI V2.<br>
<br>
A few Gerrit-based projects have already been converted to V2 as well as part of<br>
our efforts to test and debug the V2 code. Most notably, the “OST” and “Jenkins”<br>
projects have been switched, although they are running the STDCI V1 jobs as well<br>
for the time being.<br>
<br>
What is the process for switching my project to STDCI V2?<br>
------------------------------<wbr>---------------------------<br>
The CI team is going to proactively work with project maintainers to switch them<br>
to V2. The process for switching is as follows:<br>
<br>
- Send a one-line patch to the ‘jenkins’ repo to enable the V2 jobs for the<br>
project - at this point the V2 jobs will run side-by-side with the V1 jobs,<br>
and will execute the STDCI scripts on el7/x86_64.<br>
<br>
- Create an STDCI YAML file to define the target distributions, architectures<br>
and oVirt versions for the project. (See below for a sample file that would be<br>
equivalent to what many projects have defined in V1 currently). As soon as a<br>
patch with the new YAML file is submitted to the project, the V2 job will<br>
parse it and follow the instructions in it. This allows for an easy<br>
verification of the file functionality in CI.<br>
<br>
- Remove the STDCI V1 job configuration from the ‘jenkins’ repo. This should be<br>
the last patch project maintainers have to send to the ‘jenkins’ repo.<br>
<br>
What does the new YAML file look like?<br>
------------------------------<wbr>--------<br>
We defined multiple optional names for the file, so that each project owner can<br>
choose which name seems most adequate. The following names can be used:<br>
<br>
- stdci.yaml<br>
- automation.yaml<br>
- ovirtci.yaml<br>
<br>
A dot (.) can also optionally be added at the beginning of the file name to make<br>
the file hidden, the file extension could also be “yml”. If multiple matching<br>
files exist in the project repo, the first matching file according to the order<br>
listed above will be used.<br>
<br>
The file conforms to the YAML syntax. The key names in the file are<br>
case-agnostic, and hyphens (-) underscores (_) and spaces ( ) in key names are<br>
ignored. Additionally we support multiple forms of the same word so you don’t<br>
need to remember if the key should be ‘distro’, ‘distros’, ‘distributions’,<br>
‘operating-systems’ or ‘OperatingSystems’ as all these forms (and others) will<br>
work and mean the same thing.<br>
<br>
To create complex test/build matrices, ‘stage’, ‘distribution’, ‘architecture’<br>
and ‘sub-stage’ definitions can be nested within one another. We find this to be<br>
more intuitive than having to maintain tedious ‘exclude’ lists as was needed in<br>
V1.<br>
<br>
Here is an example of an STDCI V2 YAML file that is compatible with the current<br>
master branch V1 configuration of many oVirt projects:<br>
<br>
---<br>
Architectures:<br>
- x86_64:<br>
Distributions: [ "el7", "fc27" ]<br>
- ppc64le:<br>
Distribution: el7<br>
- s390x:<br>
Distribution: fc27<br>
Release Branches:<br>
master: ovirt-master<br>
<br>
Note: since the file is committed into the project’s own repo, having different<br>
configuration for different branches can be done by simply having different<br>
files in the different branches, so there is no need for a big convoluted file<br>
to configure all branches.<br>
<br>
Since the above file does not mention stages, any STDCI scripts that exists in<br>
the project repo and belong to a particular stage will be run on all specified<br>
distribution and architecture combinations. Since it is sometimes desired to run<br>
‘check-patch.sh’ on less platforms then build-artifacts for example, a slightly<br>
different file would be needed:<br>
<br>
---<br>
Architectures:<br>
- x86_64:<br>
Distributions: [ “el7”, “fc27” ]<br>
- ppc64le:<br>
Distribution: el7<br>
- s390x:<br>
Distribution: fc27<br>
Stages:<br>
- check-patch:<br>
Architecture: x86_64<br>
Distribution: el7<br>
- build-artifacts<br>
Release Branches:<br>
master: ovirt-master<br>
<br>
The above file makes ‘check-patch’ run only on el7/x86_64, while build-artifacts<br>
runs on all platforms specified and check-merged would not run at all because it<br>
is not listed in the file.<br>
<br>
Great efforts have been made to make the file format very flexible but intuitive<br>
to use. Additionally there are many defaults in place to allow specifying<br>
complex behaviours with very brief YAML code. For further details about the file<br>
format, please see the documentation linked below.<br>
<br>
About the relation between STDCI V2 and the change-queue<br>
------------------------------<wbr>--------------------------<br>
In STDCI V1 the change queue that would run the OST tests and release a given<br>
patch was determined by looking at the “version” part of the name of the<br>
project’s build-artifacts jobs that got invoked for the patch.<br>
<br>
This was confusing for people as most people understood “version” to mean the<br>
internal version for their own project rather then the oVirt version.<br>
<br>
In V2 we decided to be more explicit and simply include a map from branches to<br>
change queues in the YAML configuration under the “release-branches” option, as<br>
can be seen in the examples above.<br>
<br>
We also chose to no longer allow specifying the oVirt version as a shorthand for<br>
the equivalent queue name (E.g specifying ‘4.2’ instead of ‘ovirt-4.2’), this<br>
should reduce the chance for confusing between project versions and queue names,<br>
and also allows us to create and use change queues for projects that are not<br>
oVirt.<br>
<br>
A project can choose not to include a “release-branches” option, in which case<br>
its patches will not get submitted to any queues.<br>
<br>
Further information<br>
-------------------<br>
The documentation for STDCI can be found at [1].<br>
<br>
The documentation update for V2 are still in progress and expected to be merged<br>
soon. In the meatine, the GitHub-specific documentation [2] already provides a<br>
great deal of information which is relevant for V2.<br>
<br>
[1]: <a href="http://ovirt-infra-docs.readthedocs.io/en/latest/CI/Build_and_test_standards" rel="noreferrer" target="_blank">http://ovirt-infra-docs.<wbr>readthedocs.io/en/latest/CI/<wbr>Build_and_test_standards</a><br>
[2]: <a href="http://ovirt-infra-docs.readthedocs.io/en/latest/CI/Using_STDCI_with_GitHub" rel="noreferrer" target="_blank">http://ovirt-infra-docs.<wbr>readthedocs.io/en/latest/CI/<wbr>Using_STDCI_with_GitHub</a><br>
<br>
---<br>
Barak Korren<br>
RHV DevOps team , RHCE, RHCi<br>
Red Hat EMEA<br>
<a href="http://redhat.com" rel="noreferrer" target="_blank">redhat.com</a> | TRIED. TESTED. TRUSTED. | <a href="http://redhat.com/trusted" rel="noreferrer" target="_blank">redhat.com/trusted</a><br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>GREG</span> <span>SHEREMETA</span></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX</span><span style="color:rgb(170,170,170);margin:0px"></span></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat NA<span><br><br></span></a></p><p style="font-family:overpass,sans-serif;margin:0px 0px 6px;font-size:10px;color:rgb(153,153,153)"><span style="margin:0px;padding:0px"><a href="mailto:gshereme@redhat.com" style="color:rgb(0,136,206);margin:0px" target="_blank">gshereme@redhat.com</a> </span> <span>IRC: <span>gshereme</span></span></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img src="https://www.redhat.com/files/brand/email/sig-redhat.png" width="90" height="auto"></a></td></tr></tbody></table></div></div></div></div></div></div></div></div></div>
</div></div>