- Devel - oVirt List Archives

Patch Gating summary + FAQ
by Ehud Yonasi 27 Sep '19

27 Sep '19

Hey everyone, Following the presentation we did last week [1] <https://bluejeans.com/s/zAjyX/>, I wanted to summarize the new patch gating workflow that will be pushed to oVirt soon and will impact all the developers. Summary: The purpose of the new workflow is to verify patches earlier ( shift left ) before they are merged and provide much faster feedback for developers if their patch fails OST. 1. Feedback from OST will now be posted directly to Gerrit instead of requiring human intervention from the infra team to notify developers We expect developers to check why their patch is not passing the gate (OST), debug it, find the root cause and fix it before merging the patch. 1. Any concerns regarding the stability of OST should be communicated and addressed ASAP. The status today is that if OST fails post-merge gating packages are not pushed to tested and QE doesn’t get to test them. The change with pre-merge gating is that patches won’t be merged if OST fails, so if there are any fragile or flaky tests they should be examined by their maintainers and fixed, skipped or removed by the maintainers. 1. FYI, we are not removing the Merge button at this point, so maintainers will still be able to merge patches that they believe 100% is not breaking the build and failing OST tests. Please note that merging patches that break OST will cause it to start failing for all other patches, we urge you to avoid trying to bypass it if at all possible. In the following section, I will explain more on Patch Gating, how to onboard it, etc. FAQ on oVirt’s Gating System and how to onboard your project on it: Q. What is Patch Gating? A. It is triggered pre-merge on patches and running OST as the gate system tests, unlike today where we have post-merge OST that runs the patches after the projects are merged. This means developers get early feedback on their patches if it is passing OST. Q. What causes the gating process to start? A. Once a patch is verified, passed CI and has Code-Review +2 labels, the gating process will be started. You will receive a message in the patch Q. How does it report results to my patches? A. A comment will be posted in your patch with the job URL failure. Q. How will my patch get merged? A. If the patch has passed the gating (OST), Zuul (The new CI system for patch gating) will merge the patch automatically. Q. How do I onboard my project? A. 1. Open a JIRA ticket or mail to infra-support(a)ovirt.org 2. Creating a file named 'zuul.yaml' under your project root OR `zuul.d/zuul.yaml` and fill with the following content: - project: templates: - ost-gated-project Q. My projects run on STDCI V1, is that ok? A. No, the patch gating logic runs on STDCI V2 only! meaning that you will have to shift your project to V2. If you need help regarding the transition to V2 you can open a JIRA[2] <https://ovirt-jira.atlassian.net>ticket or mail to infra-support(a)ovirt.org and visit the docs [3] <https://ovirt-infra-docs.readthedocs.io/en/latest/>. Q. What if I want to merge the patch regardless of OST results? A. If you are a maintainer of the project, you can still merge it. we are not removing the merge button option. But, merging when failing OST can break your project so merging on failure is unadvertised. Q. What if my patch failing because of dependency on different project patch? A. Patch Gating (Zuul) has a mechanism for cross-project dependency! All you need to do is to add to the commit message the patch URL you are dependent on: Depends-On: https://gerrit.ovirt.org/patch_number And they will be tested together. Note: you can have multiple dependencies. Q. How do I debug OST? A. There are various ways of looking in the logs and output for the errors: 1. Blue Ocean view, <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_gat…> you can see the jobs that were run inside the gate and find the suites which failed. 2. ci_build_summary view, An internal tool to view the threads and redirect to the specific logs/artifacts. 3. Test results analyzer, available if the tests were run. you can view the failed tests and their output and OST maintainers and your team leads should be able to assist. 4. For further learning on how to debug OST please visit the OST FAQ <https://drive.google.com/open?id=1Sohq7bdgZS341gs5-lvB9lyS0GXawqaIvuUziOWwo…> . Q. Will the current infrastructure be able to support all the patches? A. The CI team has made tremendous work in utilizing the infrastructure. The OST gating will run inside OpenShift pods unlike before as bare metals and we can gain from that right now approximately 50 pods in parallel to run simultaneously and we will review adding more if the need arises. Q. When I have multiple patches, in which order will they be tested by the gating system? A. The patches will be tested as the flow they will be merged. The gating system knows how to simulate patches post-merge. Q. What do I do if I think OST failed because of an infra issue and not my patch? A. You can contact the CI team by sending mail to infra-support(a)ovirt.org and explain your concerns + sending the patch URL. Q. Will check-merged scripts be used by the gating system? A. No, they will be used in the current workflow with OST post-merge gating system called Change-Queue. Q. Can I add my own tests to the gating system? A. The gating system is running OST tests, so if it’s a test that should be included in the OST, then yes. Q. What will happen to the old change-queue system now that we have gating? A. At this time, the change-queue system will stay and gate post-merge jobs until all of oVirt projects will onboard to patch gating. We might consider using the change-queue for further coverage of tests in the future. Q. How can I re-trigger a failed patch to the gate again? A. There are 2 options to retrigger: - If the case is to fix your patch, just uploaded a new patchset and turn the Code-Review, Verified and CI labels again. - If you want to re-trigger the same patchset again just write a comment in Gerrit: ‘ci emulate-gate please’ Q. I usually write a series of related patches that should be merged together, can the Gating system test all of them in a single test? A. No, they will be tested in parallel as the number of patches in the series. This is why we’ve increased our capacity to run OST for this case. Architectural design document [4] <https://drive.google.com/open?id=1qV_iNJL6jHARlti7zpnRZRfQM_Q9Z-w8ONvcKEmaA…> would provide you the understanding of the patch gating process with the new services we will be using. [1]: https://bluejeans.com/s/zAjyX/ [2]: https://ovirt-jira.atlassian.net [3]: https://ovirt-infra-docs.readthedocs.io/en/latest/ [4] https://drive.google.com/open?id=1qV_iNJL6jHARlti7zpnRZRfQM_Q9Z-w8ONvcKEmaA…

5 6

ovirt-engine on master compiles and runs on Java-11
by Roy Golan 27 Sep '19

27 Sep '19

Hi all, ovirt-engine project reached a nice milestone, compiling and running on Java-11 📣 The move itself needed care to see nothing breaks and with the help of OST it remaind like unbroken, but the move was mostly painless and didn't require changes to real business logic code. It was mainly to deal with 2 things, working with modules and lack of support from GWT. Modular JDK (project jigsaw) landed in java-9 is more relevant to the delivery of the sofware rather to the development of it. Specifically now you can ship your software only with a subset of the jdk, or produce your jar as a module so it can expose or hide internal with real enforcment(even no reflection). The relevant part to the change here is that java doesn't provide EE code in the jdk, specifically java.xml, java.annotation, javax.activation [1] GWT and specifically the gwt compiler doesn't recognize java syntax beyond 8, and efforts of the community to support don't look promising. Luckily since it is strictly compile-time only concern the invocation of the gwt compiler, which is invoked by maven, is instructed to use java-8 javac. This means the frontend artifacts are remain uneffected by this change. Morning comes, I want to rebase and compile the project! This should not be huge change, and is document here [2]: - install java-11-openjdk-devel - make java-11 binaries available on your path *before java-8 *(use your favorite method I use alternatives) - install maven >= 3.5.0 Please report any problem here and also if the README [2] is not helping enough please help improve it! P.S - Obviously it is strongly encouraged to get familiar with what java 9, 10, and 11 brought in and start make use of it. If you think you have code you want to contribute to master, but is going to be backported to 4.3 bare in mind the difference in java version. Thanks, Roy [1] covered in the git log https://gerrit.ovirt.org/#/c/98623/ [2] https://github.com/oVirt/ovirt-engine/blob/master/README.adoc#note-on-java-…

1 0

Setting reason for VM shutdown
by scott.fitzgerald＠oracle.com 26 Sep '19

26 Sep '19

Hi all, I have a test case as follows: Enable "require reason for vm maintenance" on Cluster Shutdown vm Ensure reason is saved A similar case for hosts is achieved by the following hostsService .hostService(host.id()) .deactivate() .reason(reason) .send(); But the shutdown option for vms does not have a reason option. How can I set the reason for shutdown of a VM through the SDK? TIA

3 3

Using SDK to find VM Network Role
by Jamie Holohan 26 Sep '19

26 Sep '19

I am trying to verify whether a Networks VM Network checkbox is checked using the SDK, but I cannot see any methods or variable with a similar name. Is it possible to find out if a network has a VM Network Role using the SDK?

2 1

Re: [CQ]: 103548,1 (vdsm) failed "ovirt-4.3" system tests
by Nir Soffer 26 Sep '19

26 Sep '19

On Thu, Sep 26, 2019 at 3:55 PM oVirt Jenkins <jenkins(a)ovirt.org> wrote: > Change 103548,1 (vdsm) is probably the reason behind recent system test > failures in the "ovirt-4.3" change queue and needs to be fixed. > > This change had been removed from the testing queue. Artifacts build from > this > change will not be released until it is fixed. > This is a bug in the change queue, because For further details about the change see: > https://gerrit.ovirt.org/#/c/103548/1 this is a travis configuration change, and it cannot break system tests :-) Do we have stats of such failures in the past? For failed test results see: > http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/2188/ Maybe we can add a simple file based condition to tell if a patch need to run system tests? For example in vdsm we have this structure: lib/ static/ tests/ automation/ .travis.ymal ... Changes in tests/ and automation/ cannot break vdsm during runtime, so there is no need to run system tests for patches modifying these directories. Here are changes stats per directory since 4.2: $ git shortlog -sn --since v4.20.0 ./tests ./automation | awk '{sum+=$1} END {print sum}' 4579 $ git shortlog -sn --since v4.20.0 ./lib ./static | awk '{sum+=$1} END {print sum}' 4413 So 50% of the changes cannot break the system and do not require system tests. SLOC Directory SLOC-by-Language (Sorted) 65205 lib python=64977,sh=228 62415 tests python=62317,sh=98 3432 vdsm_hooks python=3432 803 contrib python=740,sh=63 468 automation sh=454,python=14 313 static python=210,sh=103 289 init sh=162,python=127 112 build-aux sh=72,python=40 64 top_dir sh=64 30 doc python=30 30 helpers python=30 1 vdsm_log sh=1 0 docker (none) 0 m4 (none) We can see that the tests are about 50% of the code. This is pretty good for legacy project, but if we look in a more modern project like ovirt-imageio: common: SLOC Directory SLOC-by-Language (Sorted) 3641 test python=3641 2365 ovirt_imageio_common python=2201,ansic=164 103 top_dir python=103 daemon: SLOC Directory SLOC-by-Language (Sorted) 1588 test python=1588 792 ovirt_imageio_daemon python=792 18 top_dir python=18 0 data (none) We see that test code ratio is close to 2/1. So adding a filter per project that can tell if a patch needs system tests very important, and can speed our workflow by fact of 2-3. Adding infra-support to file a bug, we need to work on this. Nir

2 1

oVirt and CentOS Stream
by Sandro Bonazzola 26 Sep '19

26 Sep '19

Progress cannot be made without change. As technologists, we recognize this every day. Most of the time, these changes are iterative: progressive additions of features to projects like oVirt. Sometimes those changes are small, and sometimes not. And that’s, of course, just talking about our project. But one of the biggest strengths of our community’s software is that we are not alone, and because of that, changes to other projects have ripple effects that can affect our own, even in positive ways. This week, our collaborators in the CentOS Project have announced a change <https://lists.centos.org/pipermail/centos-announce/2019-September/023449.ht…> in the way their software is released moving forward. Beyond this week release of CentOS Linux 8, the CentOS team has announced CentOS Stream, a rolling release distribution that will be the "midstream" between Fedora and Red Hat Enterprise Linux. To be clear, Fedora will remain as the first upstream of RHEL. But when RHEL branches off, CentOS will be upstream for ongoing work on those RHEL versions. This change gives public visibility into ongoing work on released RHEL, and a place for developers and community projects like ours to collaborate at that level. CentOS has been heading in the direction of Stream for quite some time. CentOS SIGs — the special interest groups for virtualization, storage, config management, and so on — have given our project a way to provide input into the CentOS platform. Now, though, we can actually provide direct improvements or fixes to CentOS Stream platform that will be beneficial to our project. Long-term, those project-specific changes can find their way into the next release of RHEL, providing smoother transitions for those users of our downstream projects. We expect the positive effects to extend beyond our own project’s ecosystem. Through CentOS Stream, developers will have early access to new features and content that are being built into the upcoming RHEL version. This will help to allow next-generation applications to have compatibility with future versions of RHEL. The benefits of these changes are clear for ecosystem developers working on projects such as ours, writing hardware drivers, or extending protocols for RHEL. As we push the innovations that start in Fedora through the new CentOS Stream, the community will have a clear vision of the future of Red Hat Enterprise Linux. For users of oVirt, we expect CentOS Stream to be the preferred upstream platform on which oVirt should be run, especially with the capability for our users to now contribute changes to our software and the community-built platform on which oVirt runs. -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

OST fails on 004_basic_sanity.verify_glance_import
by Yedidyah Bar David 26 Sep '19

26 Sep '19

Hi all, [1], which I ran for verifying [2], fails for me at 004_basic_sanity.verify_glance_import . I can see several different cases of failure there reported to devel@, last one 1.5 months ago. Is this a (new) known issue? [1] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… [2] https://gerrit.ovirt.org/97698 Thanks and best regards, -- Didi

4 4

vdsm check patch failed on loopback.py#test_with_device
by Eyal Shenitzky 26 Sep '19

26 Sep '19

Hi, Jenkins check-patch job is failing on: =================================== FAILURES =================================== 15:19:02 ____________________________ test_with_device[4096] ____________________________ 15:19:02 15:19:02 tmpdir = local('/var/tmp/vdsm/test_with_device_4096_0'), sector_size = 4096 15:19:02 15:19:02 @requires_root 15:19:02 @pytest.mark.parametrize("sector_size", [ 15:19:02 None, 15:19:02 pytest.param(sc.BLOCK_SIZE_512, marks=requires_loopback_sector_size), 15:19:02 pytest.param(sc.BLOCK_SIZE_4K, marks=[ 15:19:02 requires_loopback_sector_size, 15:19:02 pytest.mark.xfail( 15:19:02 testing.on_ovirt_ci() and testing.on_fedora("29"), 15:19:02 reason="fails randomly to create loop device with 4k sector " 15:19:02 "size, only in ovirt CI - needs investigation") 15:19:02 ]), 15:19:02 ]) 15:19:02 def test_with_device(tmpdir, sector_size): 15:19:02 filename = str(tmpdir.join("file")) 15:19:02 prepare_backing_file(filename) 15:19:02 > with loopback.Device(filename, sector_size=sector_size) as device: 15:19:02 15:19:02 storage/loopback_test.py:57: 15:19:02 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 15:19:02 storage/loopback.py:99: in __enter__ 15:19:02 self.attach() 15:19:02 storage/loopback.py:66: in attach 15:19:02 out = commands.run(cmd) 15:19:02 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 15:19:02 I noticed that was a recent fix for that test - https://gerrit.ovirt.org/#/c/100928/ Link to the failing job - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/12021/consoleFull Nir, can you have a look? -- Regards, Eyal Shenitzky

2 1

ovirt-engine has been tagged (ovirt-engine-4.2.8.9)
by Tal Nisan 26 Sep '19

26 Sep '19

1 0

Outage Notification: Jenkins and Resources Are Not Accessible
by Anton Marchukov 26 Sep '19

26 Sep '19

Hello All. Please note that oVirt services hosted at PHX datacenter are not accessible. Main services running there are: jenkins.ovirt.org resources.ovirt.org This looks to be network related and we are working with the relevant teams to resolve it. Anton. -- Anton Marchukov Associate Manager - RHV DevOps - Red Hat

1 1