'change-queue' is now in full production for the 'master' flow

Yesterday we've applied a patch with JJB that made the 'master' change queue push updates into 'tested' instead of the 'experimental' flow. This essentially means that the change queue is now the main production flow for 'master'. A quick refresher on how to find and handle issues with CQ jobs: 1. You don't really need to monitor the jobs, when the CQ suspects it found a real code issue it will send infra(a)ovirt.org an email with the suspected patch. 2. The *-tester job is what actually runs the tests. Don't worry if it fails occasionally, the CQ will run bisection and tell you exactly which change caused it. 3. The output of the *-tester job is very similar to the old 'experimental' job. 4. When you get an email, you need to check the output of the job linked to it to see if this is a race-condition kind of failure, a build failure or a real regression. Unless its a build failure, you probably need to report it and document in the table. 5. If any other CQ job besides the *-tester job fails, this is a real infra issue. When you see such an issue you need to investigate it and fix the infra. Real infra issues typically affect many jobs, so once you are done fixing it, you probably need to go back and re-trigger Gerrit merge events to get packages rebuilt and submitted to the queue. 6. One very simple kind of an infra issue, is trying to submit to a queue that does not exist. You should nag maintainers to remove jobs for versions of oVirt that we don't have queues for. Good luck! -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted