Finding the "last successful" OST CI run of something
Hi, I understand that OST (or the job(s) that run it in CI) has some logic to decide what tests need to be ran based on the components that were changed since the last run. Suppose that I want to see how the last good run of something looked like. Say the engine, or vdsm, or hosted-engine. Is there a simple way to find that out? Also, we might risk here not doing some test because we think it's not needed, and perhaps we should run all tests at least once every X (say, 1 day). Do we already? How can I see logs of this? Thanks, -- Didi
On Sun, Dec 10, 2017 at 3:43 PM, Yedidyah Bar David <didi@redhat.com> wrote:
Hi,
I understand that OST (or the job(s) that run it in CI) has some logic to decide what tests need to be ran based on the components that were changed since the last run.
Indeed. OST have a tool that we call Change Resolver. What is basically does is to extract the changed files from the git commit, and return a list of suites that include this file or a symlink to it in it's directory. Also it has some logic to decide what to do for some edge cases (files outside of suites directory/files that could not be resolved)
Suppose that I want to see how the last good run of something looked like. Say the engine, or vdsm, or hosted-engine. Is there a simple way to find that out?
I think that the best place to look for those runs of OST is to go to the Change Queue Testers (the one for the version you look for) and search for a job with your requested component (engine, vdsm, ...)
Also, we might risk here not doing some test because we think it's not needed, and perhaps we should run all tests at least once every X (say, 1 day). Do we already? How can I see logs of this?
We have timed jobs for all the suites that are not part of the Change Queue. http://jenkins.ovirt.org/view/oVirt%20system%20tests/ -- DANIEL BELENKY RHV DEVOPS <https://red.ht/sig>
On 10 December 2017 at 15:43, Yedidyah Bar David <didi@redhat.com> wrote:
Hi,
I understand that OST (or the job(s) that run it in CI) has some logic to decide what tests need to be ran based on the components that were changed since the last run.
First you need to be clear about what you're asking about. There is OST, which is the lago-based set of testing suits that can be run under various conditions, and there is CQ, which collects changes made to the whole of oVirt, runs them through certain OST suits and collects packages that passed successfully in the 'tested' repo. If we're talking about CQ - it always runs the same suits regardless of what has changed.
Suppose that I want to see how the last good run of something looked like. Say the engine, or vdsm, or hosted-engine. Is there a simple way to find that out?
The short answer is no. There is no simple 100% reliable way ATM. The assumption right no is that if you`ve not been contacted by one of our team members, you can assume your changes either passed or are still queued to be tested. The reason for that is that CQ collects changes in batches and runs them together. Those batches and broken down to smaller bathes only when failures are detected. It is possible to trace an individual change through the CQ to find out if it passed or not, but a good understanding of how CQ works is required for that. One heuristic way to know what passed is to just look at the 'tested' repo. It is typically trivial to go from an RPM name to the Git commit that generated it. At some point we will make CQ report back to Gerrit. This will make knowing what happened to an individual change trivial. There has also been requests to make CQ tag projects as changes to them pass - but there needs to be an opt-in mechanism for this from project owners, as people can be weary of having an automated system write to their Git repos.
Also, we might risk here not doing some test because we think it's not needed, and perhaps we should run all tests at least once every X (say, 1 day). Do we already? How can I see logs of this?
As said before - CQ runs all the suits that were deemed stable enough for CQ on all changes, no decision logic is involved. Some of the other suits are scheduled to run on a daily basis, but those typically run against the released or snapshot repos rather then the pre-tested set of packages. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
participants (3)
-
Barak Korren -
Daniel Belenky -
Yedidyah Bar David