Hello there, I indeed missed this event. Nevertheless, I would like to add one remark: The existing suites have differences in how they run the tests and how the tests are written. At the networking team, we found the original suites way of running things and how tests are written problematic. As a result, work has been done to make it more usable: control its run flow, use advance capabilities from test frameworks (pytest) and make it as simple as possible for a developer to add a test. The result is seen in the network suite, which runs using pytest, has a lib that wraps the SDK in a more "natural" manner, includes fixtures to handle resources at different scopes and even has an ansible test. Tests are NOT dependent on each other in any way, but they may require the same resource (here fixtures play a role). You are welcome to feedback on anything you see there, we are trying to make it dead simple to write tests. Thanks, Edy. On Tue, Mar 13, 2018 at 6:50 PM, Martin Sivak <msivak@redhat.com> wrote:

Hi,

just two or three notes from me:

- I had to find a suitable hardware first - we spent quite some time to actually get OST up and running (missing CPU flags issue in my case) - he-basic-ansible-suite-4.2 failed because of package dependencies (imageio, but probably related to what Sandro already mentioned) - he-basic-suite-master fails for me because of the SDK version, but I have not way to check which version is required

Just the last three issues cost me almost the whole day of playing with lago and OST instead of actually thinking about improvements.

The contribution guide would help indeed as would better compatibility and some better way of dependency management.

Btw, what would it take to allow ansible based tests? It should not be too hard to generate an inventory file for the deployment and launch all playbooks in the test directory. This would make contribution of simple tests even easier I think.

Best regards

Martin Sivak

On Tue, Mar 13, 2018 at 5:26 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:

...

4 people accepted calendar invite: - Devin A. Bougie - Francesco Romani - Jiri Belka - suporte, logicworks

4 people tentatively accepted calendar invite: - Amnon Maimon - Andreas Bleischwitz - Arnaud Lauriou - Stephen Pesini

2 mailing lists accepted calendar invite: users@ovirt.org, devel@ovirt.org (don't ask me how) so I may have missed someone in above list

4 patches got merged: Add check for host update to the 1st host. <https://gerrit.ovirt.org/88767> Merged Yaniv Kaul <https://gerrit.ovirt.org/#/q/owner:ykaul%2540redhat.com+status:merged> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master (add_upgrade_check) <https://gerrit.ovirt.org/#/q/status:merged+project:ovirt-system-tests+branch:master+topic:add_upgrade_check> 4:10 PM basic-suite-master: add vnic_profile_mappings to register vm <https://gerrit.ovirt.org/87438> Merged Eitan Raviv <https://gerrit.ovirt.org/#/q/owner:eraviv%2540redhat.com+status:merged> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master (register-template-vnic-mapping) <https://gerrit.ovirt.org/#/q/status:merged+project:ovirt-system-tests+branch:master+topic:register-template-vnic-mapping> 2:50 PM Revert "ovirt-4.2: Skipping 002_bootstrap.update_default_cluster" <https://gerrit.ovirt.org/88881> Merged Eyal Edri <https://gerrit.ovirt.org/#/q/owner:eedri%2540redhat.com+status:merged> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:merged+project:ovirt-system-tests+branch:master> 11:36 AM seperate 4.2 tests and utils from master <https://gerrit.ovirt.org/88878> Merged Eyal Edri <https://gerrit.ovirt.org/#/q/owner:eedri%2540redhat.com+status:merged> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:merged+project:ovirt-system-tests+branch:master> 11:35 AM

13 patches has been pushed / reviewed / rebased

Add gdeploy to ovirt-4.2.repo <https://gerrit.ovirt.org/88929> Daniel Belenky <https://gerrit.ovirt.org/#/q/owner:dbelenky%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 4:53 PM Cleanup of test code - next() replaced with any() <https://gerrit.ovirt.org/88928> Martin Sivák <https://gerrit.ovirt.org/#/q/owner:msivak%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 4:51 PM Add network queues custom property and use it in the vnic profile for VM0 <https://gerrit.ovirt.org/88829> Yaniv Kaul <https://gerrit.ovirt.org/#/q/owner:ykaul%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master (multi_queue_config) <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master+topic:multi_queue_config> 4:49 PM new suite: he-basic-iscsi-suite-master <https://gerrit.ovirt.org/85838> Yuval Turgeman <https://gerrit.ovirt.org/#/q/owner:yturgema%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master (he-basic-iscsi-suite-master) <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master+topic:he-basic-iscsi-suite-master> 4:47 PM Collect host-deploy bundle from the engine <https://gerrit.ovirt.org/88925> Yedidyah Bar David <https://gerrit.ovirt.org/#/q/owner:didi%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 4:41 PM network-suite-master: Make openstack_client_config fixture available to all ... <https://gerrit.ovirt.org/88029> Merge Conflict Marcin Mirecki <https://gerrit.ovirt.org/#/q/owner:mmirecki%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 3:39 PM new suite: he-basic-ng-ansible-suite-master <https://gerrit.ovirt.org/88901> Sandro Bonazzola <https://gerrit.ovirt.org/#/q/owner:sbonazzo%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master (he-basic-ng-ansible-suite-master) <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master+topic:he-basic-ng-ansible-suite-master> 3:37 PM Enable and move additional tests to 002 <https://gerrit.ovirt.org/88711> Yaniv Kaul <https://gerrit.ovirt.org/#/q/owner:ykaul%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master (move_more_to_002) <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master+topic:move_more_to_002> 3:08 PM common: ovirt-4.2.repo <https://gerrit.ovirt.org/88904> Sandro Bonazzola <https://gerrit.ovirt.org/#/q/owner:sbonazzo%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 2:34 PM networking: Introducing test_stateless_vm_duplicate_ma c_addr_vnic_does_not_be... <https://gerrit.ovirt.org/86946> Leon Goldberg <https://gerrit.ovirt.org/#/q/owner:lgoldber%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 2:08 PM hc: Updating gdeploy conf to create vdo volumes <https://gerrit.ovirt.org/88899> Sahina Bose <https://gerrit.ovirt.org/#/q/owner:sabose%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 12:55 PM master: add USB to the sanity VM <https://gerrit.ovirt.org/88731> Michal Skrivanek <https://gerrit.ovirt.org/#/q/owner:michal.skrivanek%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 12:54 PM Test hosted-engine cleanup <https://gerrit.ovirt.org/88331> Yedidyah Bar David <https://gerrit.ovirt.org/#/q/owner:didi%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 9:39 AM

Feedback from the event: - "if we want to add many more tests to OST, and I think we do, we need to do some change there to allow that. Current framework is simply not scalable enough" - not joining the hackathon because "I'd be like an elephant in a porcelain shop" - "I'm not sure I'm OK with the flood of suites that we have - the more we have, the harder it is to sync and maintain but more importantly - to run." - "We can't keep adding new suite for each parameter we want to test, it adds overhead to monitoring, resources and maintenance." - invite wasn't clear enough. I found people on #ovirt on Freenode and on Red Hat IRC servers and redirected them to OFTC IRC server (my fault, hopefully managed to workaround it by talking to people)

Lessons learned: - Calendar invites to mailing lists doesn't work well, need a different way to track mailing list members joining the events. - Invites needs to be pedantic on how to join the event, not leaving space for interpretation and misunderstanding. - We need a contribution guide to ovirt-system-test: we need to make people comfortable in trying to add a new test and we need to ensure that we won't reject a day of work because the patch doesn't match core contributors plannings on number of suites, resources and so on - The ovirt-system-tests check patch script is not good enough. It triggers too many sequential suites on every single patch pushed, and fails due to timeout taking more than 6 hours to complete. - The way ovirt-system-test collects rpms from defined repos is not smart enough: it doesn't take the latest version of a given package, just the first found in sequential order of the repos,

Thanks everyone who participated to the event! if you have time please continue improving ovirt-system-test even if today event is almost completed!

--

SANDRO BONAZZOLA

ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D

Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

On Tue, Mar 13, 2018 at 8:00 PM, Yaniv Kaul <ykaul@redhat.com> wrote:

On Mar 13, 2018 6:27 PM, "Sandro Bonazzola" <sbonazzo@redhat.com> wrote:

4 people accepted calendar invite: - Devin A. Bougie - Francesco Romani - Jiri Belka - suporte, logicworks

4 people tentatively accepted calendar invite: - Amnon Maimon - Andreas Bleischwitz - Arnaud Lauriou - Stephen Pesini

2 mailing lists accepted calendar invite: users@ovirt.org, devel@ovirt.org (don't ask me how) so I may have missed someone in above list

4 patches got merged: Add check for host update to the 1st host. <https://gerrit.ovirt.org/88767> Merged Yaniv Kaul <https://gerrit.ovirt.org/#/q/owner:ykaul%2540redhat.com+status:merged> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master (add_upgrade_check) <https://gerrit.ovirt.org/#/q/status:merged+project:ovirt-system-tests+branch:master+topic:add_upgrade_check> 4:10 PM basic-suite-master: add vnic_profile_mappings to register vm <https://gerrit.ovirt.org/87438> Merged Eitan Raviv <https://gerrit.ovirt.org/#/q/owner:eraviv%2540redhat.com+status:merged> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master (register-template-vnic-mapping) <https://gerrit.ovirt.org/#/q/status:merged+project:ovirt-system-tests+branch:master+topic:register-template-vnic-mapping> 2:50 PM Revert "ovirt-4.2: Skipping 002_bootstrap.update_default_cluster" <https://gerrit.ovirt.org/88881> Merged Eyal Edri <https://gerrit.ovirt.org/#/q/owner:eedri%2540redhat.com+status:merged> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:merged+project:ovirt-system-tests+branch:master> 11:36 AM seperate 4.2 tests and utils from master <https://gerrit.ovirt.org/88878> Merged Eyal Edri <https://gerrit.ovirt.org/#/q/owner:eedri%2540redhat.com+status:merged> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:merged+project:ovirt-system-tests+branch:master> 11:35 AM

13 patches has been pushed / reviewed / rebased

Add gdeploy to ovirt-4.2.repo <https://gerrit.ovirt.org/88929> Daniel Belenky <https://gerrit.ovirt.org/#/q/owner:dbelenky%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 4:53 PM Cleanup of test code - next() replaced with any() <https://gerrit.ovirt.org/88928> Martin Sivák <https://gerrit.ovirt.org/#/q/owner:msivak%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 4:51 PM Add network queues custom property and use it in the vnic profile for VM0 <https://gerrit.ovirt.org/88829> Yaniv Kaul <https://gerrit.ovirt.org/#/q/owner:ykaul%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master (multi_queue_config) <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master+topic:multi_queue_config> 4:49 PM new suite: he-basic-iscsi-suite-master <https://gerrit.ovirt.org/85838> Yuval Turgeman <https://gerrit.ovirt.org/#/q/owner:yturgema%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master (he-basic-iscsi-suite-master) <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master+topic:he-basic-iscsi-suite-master> 4:47 PM Collect host-deploy bundle from the engine <https://gerrit.ovirt.org/88925> Yedidyah Bar David <https://gerrit.ovirt.org/#/q/owner:didi%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 4:41 PM network-suite-master: Make openstack_client_config fixture available to all ... <https://gerrit.ovirt.org/88029> Merge Conflict Marcin Mirecki <https://gerrit.ovirt.org/#/q/owner:mmirecki%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 3:39 PM new suite: he-basic-ng-ansible-suite-master <https://gerrit.ovirt.org/88901> Sandro Bonazzola <https://gerrit.ovirt.org/#/q/owner:sbonazzo%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master (he-basic-ng-ansible-suite-master) <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master+topic:he-basic-ng-ansible-suite-master> 3:37 PM Enable and move additional tests to 002 <https://gerrit.ovirt.org/88711> Yaniv Kaul <https://gerrit.ovirt.org/#/q/owner:ykaul%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master (move_more_to_002) <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master+topic:move_more_to_002> 3:08 PM common: ovirt-4.2.repo <https://gerrit.ovirt.org/88904> Sandro Bonazzola <https://gerrit.ovirt.org/#/q/owner:sbonazzo%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 2:34 PM networking: Introducing test_stateless_vm_duplicate_ma c_addr_vnic_does_not_be... <https://gerrit.ovirt.org/86946> Leon Goldberg <https://gerrit.ovirt.org/#/q/owner:lgoldber%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 2:08 PM hc: Updating gdeploy conf to create vdo volumes <https://gerrit.ovirt.org/88899> Sahina Bose <https://gerrit.ovirt.org/#/q/owner:sabose%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 12:55 PM master: add USB to the sanity VM <https://gerrit.ovirt.org/88731> Michal Skrivanek <https://gerrit.ovirt.org/#/q/owner:michal.skrivanek%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 12:54 PM Test hosted-engine cleanup <https://gerrit.ovirt.org/88331> Yedidyah Bar David <https://gerrit.ovirt.org/#/q/owner:didi%2540redhat.com+status:open> ovirt-system-tests <https://gerrit.ovirt.org/#/projects/ovirt-system-tests,dashboards/default> master <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-system-tests+branch:master> 9:39 AM

Nice list of patches!

Feedback from the event: - "if we want to add many more tests to OST, and I think we do, we need to do some change there to allow that. Current framework is simply not scalable enough"

More specific and constructive ideas are welcome. We know we want to move to pytest, for example.

+1, there is a lot of things we need to improve in OST, the network suite is a good example of going forward with PyTest, we have other idea for improvements like splitting the suites into multiple projects using a new feature in std-ci, dropping the reposync file or making it optional ( actually ovirt-demo-tool is already doing that, using release rpms ). If there are more ideas, we can consider doing an infra hackathon where we can focus in improving infrastructure and moving to PyTest most of the tests ( at least in master ) .

- not joining the hackathon because "I'd be like an elephant in a porcelain shop" - "I'm not sure I'm OK with the flood of suites that we have - the more we have, the harder it is to sync and maintain but more importantly - to run." - "We can't keep adding new suite for each parameter we want to test, it adds overhead to monitoring, resources and maintenance."

I tend to agree, but not sure how to elegantly solve it.

We need to think on a new design for it which includes scalability and multiple maintainers, the requirements and usage has changed significantly over the years, I also don't have an easy solution for it, yet at least.

- invite wasn't clear enough. I found people on #ovirt on Freenode and on Red Hat IRC servers and redirected them to OFTC IRC server (my fault, hopefully managed to workaround it by talking to people)

Lessons learned: - Calendar invites to mailing lists doesn't work well, need a different way to track mailing list members joining the events. - Invites needs to be pedantic on how to join the event, not leaving space for interpretation and misunderstanding. - We need a contribution guide to ovirt-system-test: we need to make people comfortable in trying to add a new test and we need to ensure that we won't reject a day of work because the patch doesn't match core contributors plannings on number of suites, resources and so on

This could have been better if the single OST maintainer could be part of the planning and also participate and give feedback and assistance, unfourtunately it was decided to do the hackhaton without him present. I agree we need to improve our contribution guide, and work has started towards it, but any specific feedback or tickets on what can be improved will surely help.

Agree. Y.

- The ovirt-system-tests check patch script is not good enough. It triggers too many sequential suites on every single patch pushed, and fails due to timeout taking more than 6 hours to complete.

This is very close to be much better, we're 1-2 weeks away from implemeting a new feature in STD CI V2 which will allow us to replace the existing 'change resolver' and eventually run all suites in check-patch in parallel, dramatically reducing runtime.

- The way ovirt-system-test collects rpms from defined repos is not smart enough: it doesn't take the latest version of a given package, just the first found in sequential order of the repos,

Its not that its not smart, its by design, repoman uses "only-latest" option to get the first RPM he finds from the repos, I'm pretty sure there was a good reason behind it when it was written a few years ago, I'll try to remember and update. In any case, the relevant patch where this was needed had a different problem with dynamic replacement of repos which we need to think on a solution for.

Thanks everyone who participated to the event! if you have time please continue improving ovirt-system-test even if today event is almost completed!

--

SANDRO BONAZZOLA

ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D

Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

-- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)