- lago-devel - oVirt List Archives

Lago Demo
by Gal Ben Haim 19 May '16

19 May '16

Hi All, I wrote a document which demonstrates a basic use case of Lago. The reader should follow the instructions in the document, and thus get a sense about Lago's capabilities and ease of use. This document should guide the user in his first experience, and this is why I'm not exposing the advanced capabilities of Lago (we will have another document for this). I would like you to try out the document, and write back a review. Link to the document: https://github.com/gbenhaim/Lago-Demo Thanks ! Gal Ben Haim gbenhaim(a)redhat.com RHEV DevOps

2 1

[2016-05-18] Weekly lago release
by David Caro 18 May '16

18 May '16

Hi everyone!! Last week's release was delayed a little bit due to the holidays, so we are releasing today, the new 0.17 version of lago comes with some interesting improvements: 9253f807: Added possibility to use env vars on disk path and url c7ff5190: Removed the repo verification code - replaced by retries and workarounds 61813f2e: Fix wait_for_ssh logic - now the times and tries when waiting for ssh make more sense, and have better logging 1ec94d55: Added LAGO_INITFILE_PATH env var when running init 86a8bde1: Setting LAGO_WORKDIR_PATH too on init 7a661154: Copy inside the prefix the deploy scripts 499c9912: Expanding the disks path on vm xml generation c5f6cafb: Using LAGO_PREFIX_DIR for the disk images too So the most change is the possibility to use env variables in the init file, and the addition of the env vars: * LAGO_PREFIX_DIR * LAGO_WORKDIR_PATH * LAGO_INITFILE_PATH (only when running lago init) That allow you to better specify relative paths to the different files (disks, scripts, urls...). And that allow to keep the workdir functional even after renaming them (mv old_workdir new_workdir). For the updated docs and changelog: http://lago.readthedocs.io/en/0.17/ For the rpms: http://resources.ovirt.org/repos/lago/stable/0.0/rpm/ -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

Can't run tests with latest Lago & ovirt-system-tests
by Yaniv Kaul 16 May '16

16 May '16

I've updated to the latest ovirt-system-tests, and apparently commit ed759a62 was breaking things to me. So I've updated Lago to latest (from http://resources.ovirt.org/repos/lago/unstable/0.0/latest/rpm/fc23/noarch/ ) - which is 0.16.12-1 now, and now things break differently, possibly by Lago commit ba6dda1e92faf26bc6842a78143ced20ae0546ff @ Create prefix internal repo: ERROR (in 0:00:00) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 248, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 584, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 595, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 166, in do_ovirt_reposetup skip_sync=skip_sync, File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 423, in prepare_repo _sync_rpm_repository(rpm_repo, reposync_yum_config, repos) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 105, in _sync_rpm_repository ret, out, _ = utils.run_command(reposync_command) AttributeError: 'module' object has no attribute 'run_command Any idea?

3 3

[2016-05-09] Lago weekly meeting minutes
by David Caro 09 May '16

09 May '16

Agenda ======== * oVirt system tests status: - 3.6 working - Master seems to fail on add-hosts still (investigating) * Merged workaround for reposync issues * Lago tutorial on it’s way - Using the unstable/latest lago (with the new patches) - Latest version: http://jenkins.ovirt.org/job/lago_master_check-patch-el7-x86_64/51/artifact… * Gal released 0.16.0 last week, kudos! * Merged patch to copy the deploy scripts inside the prefix on init, that takes us a step closer to relocatable prefixes (and exportable prefixes too) Let's have a great week! -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

[05-05-2016] New lago release - 0.16.0
by Gal Ben Haim 05 May '16

05 May '16

Hi everyone! There's a new lago release with version 0.16.0 ready for you to upgrade! One important change in version 0.16.0 is the support for the following environment variables: $LAGO_PREFIX_PATH = .lago/current $LAGO_WORKDIR_PATH = .lago it's now highly recommended to use those variables when configuring the path to the deployment scripts. Here are the other changes: * Wed May 04 2016 David Caro <dcaroest(a)redhat.com> - 0.16.0 45764a95: Fixed small typo in a string * Wed May 04 2016 Lago CI bot <dcaroest+cibot(a)redhat.com> - 0.15.5 66dec63b: Merge pull request #231 from lago-project/fix_el7_dirlock ccf6dca4: Added lock class for el7 locking 9a6e370c: Use lago.utils.LockFile instead of lockfile's one 98322f01: Removed thread pool on reposync 167c8b7d: Moved some code out of a with block db836eda: Yapf fixes * Tue May 03 2016 Lago CI bot <dcaroest+cibot(a)redhat.com> - 0.15.4 0e535676: Merge pull request #230 from lago-project/add_prefix_env_vars 17e9be31: tests:Modified deploy funct tests to use vars 1440adee: Add some lago extra vars to the env 577c442a: Expand vars and user contraptions on deploy scripts * Tue May 03 2016 Lago CI bot <dcaroest+cibot(a)redhat.com> - 0.15.3 a1a57978: Merge pull request #229 from lago-project/extra_gitignore 2a5a3359: Added extra ignores to .gitignore * Fri Apr 29 2016 Lago CI bot <dcaroest+cibot(a)redhat.com> - 0.15.2 25c91601: Merge pull request #227 from lago-project/fix_unhandled_exception 8eeb43cc: Fixed wrong error log on reposetup * Fri Apr 29 2016 Lago CI bot <dcaroest+cibot(a)redhat.com> - 0.15.1 5c5bc197: Merge pull request #226 from lago-project/fix_repoverify_check c4380a9a: Fixed repoverify not behaving like reposync d8c9ac55: Added new deps for the tests * Sat Apr 23 2016 Lago CI bot <dcaroest+cibot(a)redhat.com> - 0.15.0 4b98d690: Merge pull request #223 from lago-project/try_cleaning_reposync_caches_on_first_failure f87101ee: Reposync first retry with clean caches Enjoy! Gal Ben Haim gbenhaim(a)redhat.com RHEV-CI Team

2 1

Weekly bug status for ovirt-system-tests (using Lago) [ 01/05/16 ]
by Eyal Edri 01 May '16

01 May '16

FYI, Some updates from recent activity on ovirt-system-tests running on Lago. *Open bugs:* https://goo.gl/eUaPUY *Open issues:* Improvements on the host install issue, some env are not reproducing anymore but not sure we're 100% clear yet. *Open patches:* (reviews are welcome) https://gerrit.ovirt.org/#/q/project:ovirt-system-tests+status:open * There are quite a few open, some are just temp for helping with the debug of 'stuck in installing state' *Jenkins jobs: (failing on known issues)* http://jenkins.ovirt.org/view/oVirt%20system%20tests/ *Recently merged patches:* commit 211e475c004a2635896e528cd3aed6cd7d0dc5fd Author: Sandro Bonazzola <sbonazzo(a)redhat.com> Date: Thu Apr 28 16:44:50 2016 +0200 common: yum-repos: update glusterfs repos - dropped Gluster community repo for Fedora23: the repo is empty and Gluster community is maintaining Fedora packages within Fedora repositories. For reference: http://download.gluster.org/pub/gluster/glusterfs/LATEST/Fedora/What_About_… - on EL7 moved from Gluster community repo to CentOS Storage SIG repositories: we've been advised by Gluster community that their packaging effort is focused on CentOS Storage SIG and CentOS infrastructure is more reliable. Change-Id: I4d2c752be825fbbb9035222845728d10c5dd288c Signed-off-by: Sandro Bonazzola <sbonazzo(a)redhat.com> ---------------------------------------------------------------------------------------------------------- commit 19c102110279975d274c32477cc28c7c29e75fbc Author: David Caro <dcaroest(a)redhat.com> Date: Wed Apr 20 21:10:37 2016 +0200 Fixed answerfile for master A new question was added to the setup Change-Id: I6be958a70d67126d8067ff1855018125623b10c1 ---------------------------------------------------------------------------------------------------------- commit 5829862d17e92f91cfd04e7b9d5549cd775c800c Author: Yaniv Kaul <ykaul(a)redhat.com> Date: Mon Apr 25 14:06:27 2016 +0300 Add 'device=eth0' to EL7 deploy scripts. Apparently, /etc/sysconfig/network-scripts/ifcfg-eth0 lacks it on EL7 images. This works with NetworkManager, but not without it. When something goes bad, VDSM fails to rollback properly as its missing. For now, add it. Change-Id: I1c4ca07499a3947022e6f943cf05842ee1def408 Signed-off-by: Yaniv Kaul <ykaul(a)redhat.com> ---------------------------------------------------------------------------------------------------------- commit ddb42782968e5c6d411cdc747503fd90bed249d7 Author: David Caro <dcaroest(a)redhat.com> Date: Tue Mar 29 18:53:42 2016 +0200 Fix the master suite runs This is just copying over and adapting a bit (just so it works) the 3.6 suite into the master suite. Only for the 3.6 api, the newapi tests will be added later Still affected by the storage deactivation issues Change-Id: I55781a5d5a746aaf81e403eeeee40880beacb8a0 Signed-off-by: David Caro <dcaroest(a)redhat.com> ---------------------------------------------------------------------------------------------------------- commit c7d50cf8aebd6b566ae5c6ee8d6e3b8ec24ffc2b Author: Yaniv Kaul <ykaul(a)redhat.com> Date: Tue Apr 26 13:57:43 2016 +0300 Remove snapshot and reverting as part of the ovirt-system-tests I currently did not find a use case for it. It slows down the tests and causes (real? false?) issues with SDs. Mainly their activation and deactivation. It dramatically reduced the time to run the whole suite for me. 3-4 minutes on my desktop, and >10 minutes on my laptop. Change-Id: I9f7e509220c83805df78abb94b0ddc390c4a7f54 Signed-off-by: Yaniv Kaul <ykaul(a)redhat.com> ---------------------------------------------------------------------------------------------------------- -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

1 0

[2016-04-26] Lago weekly meeting minutes
by David Caro 26 Apr '16

26 Apr '16

Agenda ======== * Ovirt system tests 3.6 status: Gal testing 3.6.6 it locally, it’s expected to pass * Ovirt system tests master status: Affected by: https://bugzilla.redhat.com/show_bug.cgi?id=1322257 https://bugzilla.redhat.com/show_bug.cgi?id=1329317 * Had some issues with jenkins.phx.ovirt.org last week, fixed already * Currently reposetup not working on el7 systems, using old version of lockfile https://github.com/lago-project/lago/issues/225 * Released 0.15 with some improvements: Rotating the dest dir on collect Generic ‘lago deploy’ command More verbose reposetup command In progress: * Lago tutorial * Lago init file spec * Lago high level desing doc * virt.py fixes Action items: * dcaro: fix the el7 lockfile issue * dcaro: review the tutorial to unblock gbenhaim Thanks!! -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

2 2

Weekly bug status for ovirt-system-tests (using Lago) [ 24/04/16 ]
by Eyal Edri 24 Apr '16

24 Apr '16

FYI, Some updates from recent activity on ovirt-system-tests running on Lago. *Open bugs:* https://goo.gl/eUaPUY *Open issues:* Still suffering from host add failures (stuck in installing, David has been debugging it constantly with Piotr last week) *Open patches:* (reviews are welcome) https://gerrit.ovirt.org/#/q/project:ovirt-system-tests+status:open *Jenkins jobs: (failing on known issues)* http://jenkins.ovirt.org/view/oVirt%20system%20tests/ *Recently merged patches:* (Thanks again to Yaniv K!) Author: Yaniv Kaul <ykaul(a)redhat.com> Date: Wed Apr 20 23:51:55 2016 +0300 More accurate 3.6 repo 1. With this repo, both engine and hosts can be installed, without any external access, I hope. 2. Now we can test latest 3.6, not the released one. Change-Id: I9066af4f5d92e275586c29c65fe7ef7500fe5c20 Signed-off-by: Yaniv Kaul <ykaul(a)redhat.com> -------------------------------------------------------------------------------------------------------------------- Author: Yaniv Kaul <ykaul(a)redhat.com> Date: Tue Apr 12 23:00:16 2016 +0300 Fixes for FC23 local repo setup: 1. The interface may not be eth0. The new code hopefully work with any if name. Depends that the interface is the first, but let's hope it is. 2. Add 'cost=1' to ensure the local repo is preferred over other repos. Change-Id: If846d7bacdf517d1342464c210359e5a0e8cad6e Signed-off-by: Yaniv Kaul <ykaul(a)redhat.com> -------------------------------------------------------------------------------------------------------------------- Author: Yaniv Kaul <ykaul(a)redhat.com> Date: Wed Mar 30 22:46:39 2016 +0300 Basic quota tests Add quota and set it to audit mode, not more than that. Change-Id: Ia35e9b2b65c01cdcca8da7f87ed7a38f8dc2bb17 Signed-off-by: Yaniv Kaul <ykaul(a)redhat.com> -------------------------------------------------------------------------------------------------------------------- Author: Yaniv Kaul <ykaul(a)redhat.com> Date: Thu Apr 14 01:04:28 2016 +0300 Fix mount / path issues with storage_nfs_el7 provisioning script Clearly was not tested well (because of the host installation, but that's an execuse). This patch fixes it, tested on 3.6 with adding all domains, finally. It also again simplifies it - but removes the option of not mounting the export mount. Note that the ISO mount is not completely done as well, I think. Will attend to in a follow up patch. Based on David's request, the whole setup is split into functions. Tested (properly) with 3.6, including the TEMPLATES domain attachment. (The test for it should still be excluded, IMHO). Change-Id: I261f3d3f3b3647cbbc506913478ffc6c36594519 Signed-off-by: Yaniv Kaul <ykaul(a)redhat.com> -------------------------------------------------------------------------------------------------------------------- Author: Yaniv Kaul <ykaul(a)redhat.com> Date: Sun Apr 10 17:10:16 2016 +0300 Simplification and enhancements to iSCSI and NFS servers deployment Removed a lot of not needed code from the setup. Mainly, just straightforward using the disks (in case of NFS), no need for LVM stuff. Disabled firewall, enabled 'noop' scheduler. Tested with both master and 3.6. Never got to test fully master, blocked by a host installation bug. 3.6 passed well. Change-Id: I7ec630ba5fa62529421b8e8f9aec993e7cdc84b6 Signed-off-by: Yaniv Kaul <ykaul(a)redhat.com> Please comment / update on any additional info you have on this thread, Eyal.

1 0

Weekly lago release 0.15
by David Caro 23 Apr '16

23 Apr '16

Hi everyone!! It's been a couple weeks without releases, but we are back and 100% committed to keep up the pace. On this week's release: * Several reposetup command improvements (better logs, friendlier retry, ...) * Collect now rotates the destination folder if it exists * Skip vm bootstrap, what allows faster functional tests * Generic deploy subcommand, now you can use the deploy command with lago, just add a 'deploy-scripts' list entry to the 'metadata' key on any vm/domain definition, with a list of scripts and they will be run on 'lago deploy' For the full changelog, as always, you can go to: http://resources.ovirt.org/repos/lago/stable/0.0/CHANGELOG_0.0.txt And for the latest rpms: http://resources.ovirt.org/repos/lago/stable/0.0/rpm/ Enjoy! -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

Can I define somewhere the collect artifacts?
by Yaniv Kaul 14 Apr '16

14 Apr '16

I'm extending the VDSM functional tests, and saving files under /tmp/vdsm-results/ - can I simply ask somehow to get all files from there, perhaps via the init.json.ini file would be the right place to add this? TIA, Y.

3 6

Lago + ovirt-system-tests run fail on collecting logs
by Eyal Edri 13 Apr '16

13 Apr '16

After running: ./run_suite.sh --cleanup basic_suite_3.6 ./run_suite.sh basic_suite_3.6 The tests failed at [1] Does it mean the cleanup stage failed? why would log collection fail on a duplicate file? I think it will be beneficial to support multiple logs and rotate it like we do in ovirt for e.g, Since a developer might run a few times without cleanning up the env. Do we have an open issue for it? , cause I didn't find anything similar. [1]. @ Collect artifacts: ERROR (in 0:00:00) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 248, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 217, in do_ovirt_collect prefix.collect_artifacts(output) File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 598, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 928, in collect_artifacts os.makedirs(output_dir) File "/usr/lib64/python2.7/os.py", line 157, in makedirs mkdir(name, mode) OSError: [Errno 17] File exists: '/home/eedri/lago/ovirt-system-tests/test_logs/basic_suite_3.6/post-001_initialize_engine.py' -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

3 7

configuring path for reposync
by Eyal Edri 12 Apr '16

12 Apr '16

Hi, When installing lago you get these default values in : [lago] log_level = debug template_store = /var/lib/lago/store template_repos = /var/lib/lago/repos First, what is the diff between the 2? store holds the templates, what is repos? Also, when running ovirt-system-tests i see under /var/lib/lago/reposync all the repos sycned, is there away to define this path to be somewhere else? is it in the ovirt-system-tests config (if so why is it under lago dir) if not, shouldn't it be in lago.conf? -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

2 2

repeating libvirt exception in vdsm check-merged job
by Yaniv Bronheim 11 Apr '16

11 Apr '16

Hi, we keep getting the following error in vdsm automation *12:16:47* + lago start vdsm_functional_tests_host-fc23*12:16:49* current session does not belong to lago group.*12:16:49* @ Start specified VMs: *12:16:49* # Start nets: *12:16:49* * Create network vdsm_functional_tests_lago: *12:16:50* * Create network vdsm_functional_tests_lago: Success (in 0:00:00)*12:16:50* # Start nets: Success (in 0:00:00)*12:16:50* # Start vms: *12:16:50* * Starting VM vdsm_functional_tests_host-fc23: *12:16:50* libvirt: QEMU Driver error : Cannot check QEMU binary /usr/libexec/qemu-kvm: No such file or directory*12:16:50* * Starting VM vdsm_functional_tests_host-fc23: ERROR (in 0:00:00)*12:16:50* # Start vms: ERROR (in 0:00:00)*12:16:50* # Destroy network vdsm_functional_tests_lago: *12:16:50* # Destroy network vdsm_functional_tests_lago: ERROR (in 0:00:00)*12:16:50* @ Start specified VMs: ERROR (in 0:00:00)*12:16:50* Error occured, aborting*12:16:50* Traceback (most recent call last):*12:16:50* File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 661, in main*12:16:50* cli_plugins[args.verb].do_run(args)*12:16:50* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run*12:16:50* self._do_run(**vars(args))*12:16:50* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper*12:16:50* return func(*args, **kwargs)*12:16:50* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper*12:16:50* return func(*args, prefix=prefix, **kwargs)*12:16:50* File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 234, in do_start*12:16:50* prefix.start(vm_names=vm_names)*12:16:50* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 778, in start*12:16:50* self.virt_env.start(vm_names=vm_names)*12:16:50* File "/usr/lib/python2.7/site-packages/lago/virt.py", line 193, in start*12:16:50* vm.start()*12:16:50* File "/usr/lib/python2.7/site-packages/lago/virt.py", line 910, in start*12:16:50* self._env.libvirt_con.createXML(self._libvirt_xml())*12:16:50* File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3611, in createXML*12:16:50* if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)*12:16:50* libvirtError: Cannot check QEMU binary /usr/libexec/qemu-kvm: No such file or directory all jobs fail today with the same exception - http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/53/console . any idea why qemu-kvm is missing? -- *Yaniv Bronhaim.*

3 5

libvirt: QEMU Driver error : Guest agent is not responding: QEMU guest agent is not available due to an error
by Eyal Edri 10 Apr '16

10 Apr '16

Anyone run lago before on RHEL 7.2? Its possible we're hitting [1] with libvirt-1.2.17 https://bugzilla.redhat.com/show_bug.cgi?id=1090551 -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

1 0

reminder on lago khanban board
by Eyal Edri 10 Apr '16

10 Apr '16

FYI, Lago started using a khanban board [1] to prioritize and visualize the current issues which are being worked on, since it very hard to use GitHub to monitor and prioritize issues. Taiga is an open source project and was chosen (rather than Trello for e.g) because it had already integration with GitHub for syncing automatically issues to the board and moving statuses. If you want to ask for a specific issue prioritization, please use this list to communicate it, We will review issues on a weekly basis and the WIP will be shown on the board. [1] https://tree.taiga.io/project/david-caro-lago/kanban -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

1 0

Weekly bug status for ovirt-system-tests (using Lago) [ 10/04/16 ]
by Eyal Edri 10 Apr '16

10 Apr '16

A weekly reminder on open bugs & patches to keep track on blockers for oVirt testing: *Open bugs:* https://goo.gl/eUaPUY *Open issues:* Still suffering from add host failures, pending a fix from network team: https://bugzilla.redhat.com/show_bug.cgi?id=1322257 *Recently merged:* (Thanks Yaniv K!) - Multiple changes to deployment script https://gerrit.ovirt.org/#/c/55695/ - Fix IP Address of engine *https://gerrit.ovirt.org/#/c/55903/ <https://gerrit.ovirt.org/#/c/55903/>* - Improvements to the storage domain add tests - https://gerrit.ovirt.org/#/c/55233/ *Open patches:* (reviews are welcome) https://gerrit.ovirt.org/#/q/project:ovirt-system-tests+status:open - *Basic quota tests: https://gerrit.ovirt.org/#/c/55514/ <https://gerrit.ovirt.org/#/c/55514/>* - *Switched between hot-add disk and hot-add NIC tests * https://gerrit.ovirt.org/#/c/55740/ - (Attempt to reduce chance of hitting the deactivate storage bug, though the fix of this bug might be in already, so not sure its needed, Yaniv?) - WIP: Fix master (4.0) suite, still in progress https://gerrit.ovirt.org/#/c/55425/ - Merging the storage VMs into one, still in progress: https://gerrit.ovirt.org/#/c/54957/ If you have any updates on the bugs/patches please reply with more info. -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

2 1

In which order repos are searched for RPMs?
by Yaniv Kaul 05 Apr '16

05 Apr '16

If I have the RPM on the 'localsync' repo as well as externally (say, from CentOS 7 updates), which repo will it be taken from? - Simple solution: sync all that you need and disable the 'updates' repo. I'm in favor. Same for the 'base' repo and any other repo. Essentially, all that the VM needs should be in the 'localsync' repo. - Simple solution #2 : rename 'localsync' to '*a*localsync' - just to make sure it is the first looked at... Before: [root@lago_basic_suite_3_6_engine ~]# egrep "Updated|Installed" /var/log/yum.log | awk -F\: '{print $NF}' |xargs yum info |grep "From repo" |egrep -c "base|updates" 87 After the change: [root@lago_basic_suite_3_6_engine ~]# egrep "Updated|Installed" /var/log/yum.log | awk -F\: '{print $NF}' |xargs yum info |grep "From repo" |egrep -c "base|updates" 1

2 1

[2016-04-04] Lago weekly meeting minutes
by David Caro 04 Apr '16

04 Apr '16

Hi everyone, short meeting this week: Discussions: ============= * Ovirt system tests 3.6 status: - 3.6 tests running on jenkins (currently failing to add hosts) * Ovirt system tests master status: - Affected by: https://bugzilla.redhat.com/show_bug.cgi?id=1322257 * Investigating about vagrant-lago - https://docs.google.com/spreadsheets/d/1PKIlmhTd-vxEfs7JZC-mLkvWG4cBE1oOnxR… - Have to schedule a meeting to push this forward Action Items ============= * Migrate to jenkins.phx.ovirt.orgt -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

Weekly bug status for ovirt-system-tests (using Lago)
by Eyal Edri 03 Apr '16

03 Apr '16

A weekly reminder on open bugs & patches to keep track on blockers for oVirt testing: *Open bugs:* https://goo.gl/eUaPUY *Blocking bugs in potential:* - Bug 1311121 - oVirt system tests for 3.6 fail to deactivate domain on the first try - suspect race between OVF update and SPM stop (engine bug) - Patch provided, pending verification/testing on the tests *Open patches:* https://gerrit.ovirt.org/#/q/project:ovirt-system-tests+status:open - WIP:Fix the master suite runs - Improvements to storage domains add tests - Merged storage servers into one - Basic quote tests If you have any updates on the bugs/patches please reply with more info. -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

2 1

[2016-04-01] Weekly lago release, 0.14.0
by David Caro 01 Apr '16

01 Apr '16

Hi everyone! There's a new lago release, with mostly bug fixes but also some features, the most relevant are: * New default init file: Now you can use a LagoInitFile in the current dir as the default init configuration file (the usual init.json or init.yaml) so when running 'lago init' it will pick it up. * New '--set-current' option for the init command, to allow setting the prefix that's being created as the current prefix. For more detailed changelog you can go to: http://resources.ovirt.org/repos/lago/stable/0.0/CHANGELOG_0.0.txt And the rpms, as usual are at: http://resources.ovirt.org/repos/lago/stable/0.0/rpm/ Enjoy! -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

'Persistent' IP addresses for the hosts
by Yaniv Kaul 31 Mar '16

31 Mar '16

Can I ensure that in every run, host0 will get the same IP from the network IP pool? I'd imagine that they would, by the order of their init.json.in - but i'm not sure this is the case. TIA, Y.

3 8

Failing to run ovirt-system-tests using lago due to missing rpms
by Eyal Edri 29 Mar '16

29 Mar '16

Hi, Anyone encountered these errors when running 3.6 basic suite on RHEL 7.2? * STDERR Error: Package: ovirt-engine-3.6.4.1-1.el6.noarch (localsync) Requires: ovirt-iso-uploader >= 3.6.0 Error: Package: ovirt-engine-3.6.4.1-1.el6.noarch (localsync) Requires: slf4j >= 1.7.0 Available: slf4j-1.5.8-8.el6.noarch (base) slf4j = 1.5.8-8.el6 Error: Package: ovirt-engine-backend-3.6.4.1-1.el6.noarch (localsync) Requires: ovirt-host-deploy-java >= 1.4.0 Error: Package: ovirt-engine-vmconsole-proxy-helper-3.6.4.1-1.el6.noarch (localsync) Requires: ovirt-vmconsole-proxy Error: Package: ovirt-engine-3.6.4.1-1.el6.noarch (localsync) Requires: ovirt-image-uploader >= 3.6.0 Error: Package: ovirt-engine-backend-3.6.4.1-1.el6.noarch (localsync) Requires: ebay-cors-filter Error: Package: ovirt-engine-setup-base-3.6.4.1-1.el6.noarch (localsync) Requires: otopi >= 1.4.0 Error: Package: ovirt-engine-setup-plugin-ovirt-engine-3.6.4.1-1.el6.noarch (localsync) Requires: ovirt-engine-extension-aaa-jdbc Error: Package: ovirt-engine-3.6.4.1-1.el6.noarch (localsync) Requires: ovirt-engine-wildfly >= 8.2.1 Error: Package: ovirt-engine-setup-plugin-ovirt-engine-common-3.6.4.1-1.el6.noarch (localsync) Requires: ovirt-setup-lib Error: Package: ovirt-engine-backend-3.6.4.1-1.el6.noarch (localsync) Requires: vdsm-jsonrpc-java >= 1.1.8 Error: Package: ovirt-engine-backend-3.6.4.1-1.el6.noarch (localsync) Requires: ovirt-engine-extension-aaa-jdbc # [Thread-3] Deploy VM lago_basic_suite_3_6_engine: ERROR (in 0:01:59) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 46, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 472, in _deploy_host host.name(), RuntimeError: /home/eedri/lago/ovirt-system-tests/basic_suite_3.6/deploy-scripts/setup_engine.sh failed with status 1 on lago_basic_suite_3_6_engine @ Deploy oVirt environment: ERROR (in 0:01:59) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 248, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 174, in do_deploy prefix.deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 598, in wrapper func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 52, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 480, in deploy self._deploy_host, self.virt_env.get_vms().values() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 88, in invoke_in_parallel vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 46, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 472, in _deploy_host host.name(), RuntimeError: /home/eedri/lago/ovirt-system-tests/basic_suite_3.6/deploy-scripts/setup_engine.sh failed with status 1 on lago_basic_suite_3_6_engine -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

1 0

Reviews needed for ovirt-system-tests
by Eyal Edri 24 Mar '16

24 Mar '16

Hi, There has been progress with fixing a few issues on ovirt-system-tests and patches were submitted [1]. We would like to merge them ASAP so the ovirt tests for 3.6 and 4.0 can get one more step closer to working and being stable both for developers and for CI. Can you please review the pending patches so we can merge them TODAY? [1] https://gerrit.ovirt.org/#/c/53534 - needs a review https://gerrit.ovirt.org/#/c/53518/ - +1, probably can be merged. https://gerrit.ovirt.org/#/c/54738/ - +1, can get merged. https://gerrit.ovirt.org/#/c/54721/ - +1, can get merged https://gerrit.ovirt.org/#/c/51566/4 - +1, can get merged https://gerrit.ovirt.org/#/c/52681/ - not sure about this one -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

1 0

[2016-21-03] Lago weekly meeting minutes
by David Caro 21 Mar '16

21 Mar '16

Discussions: --------------- * Ovirt system tests 3.6 status: - Removed template import related tests - Using el6-base and el7-base only templates - Running ok ~50% of the time, current issues: ~ Storage domain is not being deactivated https://bugzilla.redhat.com/show_bug.cgi?id=1311121 ~ Host does not get installed in time (15min) ~ Host vm gets into non-responsive (no ssh, no console, but vm is started) * Investigating about vagrant-lago https://docs.google.com/spreadsheets/d/1PKIlmhTd-vxEfs7JZC-mLkvWG4cBE1oOnxR… * Started using taiga for task prioritization: https://tree.taiga.io/project/david-caro-lago/kanba Last week action items: ------------------------ * [ovirt-system-tests] Open bug on moving 4.0 tests to 3.6 api code https://bugzilla.redhat.com/show_bug.cgi?id=1317610 * [ovirt-system-tests] Skip the template-relate * Open thread on vagrant integration - DONE -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

2 2

merge_repos is not picking the latest version of ioprocess?
by Yaniv Kaul 21 Mar '16

21 Mar '16

The latest I'm interested in is 0.15.0-4.el7: mini@ykaul-mini:~/ovirt-system-tests$ find /var/lib/lago/reposync -name "ioprocess*.rpm" /var/lib/lago/reposync/ovirt-3.6-el7/x86_64/ioprocess-0.15.0-2.el7.x86_64.rpm /var/lib/lago/reposync/epel-el7/i/ioprocess-0.15.0-4.el7.x86_64.rpm /var/lib/lago/reposync/epel-el6/ioprocess-0.14.0-1.el6.x86_64.rpm /var/lib/lago/reposync/ovirt-3.6-el6/x86_64/ioprocess-0.15.0-2.el6.x86_64.rpm But merge picks 0.15.0-2.el7 instead: 2016-03-21 13:14:11,458::merge_repos.py::merge::123::ovirtlago.merge_repos::DEBUG::Copying /var/lib/lago/reposync/ovirt-3.6-el7/x86_64/ioprocess-0.15.0-2.el7.x86_64.rpm to output directory 2016-03-21 13:14:12,224::merge_repos.py::merge::123::ovirtlago.merge_repos::DEBUG::Copying /var/lib/lago/reposync/ovirt-3.6-el7/noarch/python-ioprocess-0.15.0-2.el7.noarch.rpm to output directory Which causes the VM to take the right one from EPEL directly (and not via the localsync). Thoughts?

2 3

Strange logs directory
by Yaniv Kaul 21 Mar '16

21 Mar '16

Did not see it before... running 0.13 (might have been there earlier, but I missed it?) mini@ykaul-mini:/home/zram/3.6/current/test_logs$ ls -l total 0 drwxrwxr-x 7 mini mini 191 מרץ 21 12:35 <nose.suite.ContextSuite context=004_basic_sanity>:setup-20160321123529 I'm running the tests onto /home/zram/3.6 directory (current and default are under it) Y.

2 3

[2016-03-18] Weekly lago release 0.13.0
by David Caro 18 Mar '16

18 Mar '16

Hi everyone! This week's release added a new verb to the default command, the collect one that was only available for the ovirt plugin. You can also now define the artifacts to be collected in the init config file. Now it better handles the artifact collection too, allowing extracting artifacts from live vms without guest agent, and allowing extracting them from any stopped vm. Also added a timeout to the ssh connection to hosts, so it will not hand forever anymore. You can get the just-baked rpms from the usual repos: http://resources.ovirt.org/repos/lago/stable/0.0/rpm/ For the full changelog you can go to: http://resources.ovirt.org/repos/lago/stable/0.0/CHANGELOG_0.0.txt Enjoy!! -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

Error with latest (0.12.6-1) RPMs
by Yaniv Kaul 16 Mar '16

16 Mar '16

File "/usr/lib/python2.7/site-packages/lago/virt.py", line 642, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/virt.py", line 648, in _get_ssh_client ssh_timeout = int(config.get('ssh_timeout')) File "/usr/lib/python2.7/site-packages/lago/config.py", line 83, in get raise KeyError(key) KeyError: 'ssh_timeout' Fixed by manually adding: ssh_tries = 50 ssh_timeout = 10

2 1

strange readthedocs page for lago latest
by Eyal Edri 16 Mar '16

16 Mar '16

http://lago.readthedocs.org/en/latest/ -- Eyal Edri Associate Manager EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

2 3

[2016-03-14] Weekly meeting minutes
by David Caro 14 Mar '16

14 Mar '16

* Discussions: - Add rpdb capabilities (even if it’s just locally): * https://github.com/tamentis/rpdb#trigger-rpdb-with-signal - No more ovirt_3.5_system-tests to check lago full functionality and ovirt_3.6 ones still failing * Action items: - [ovirt-system-tests] Open bug on moving 4.0 tests to 3.6 api code * https://bugzilla.redhat.com/show_bug.cgi?id=1317610 - [ovirt-system-tests] Skip the template-related tests for the 3.6 - Open thread on vagrant integration * Last week action items: - add --set-current option to init - OPEN * https://github.com/lago-project/lago/issues/162 - send bugzilla about nfs issues on 3.6 ovirt system test * https://bugzilla.redhat.com/show_bug.cgi?id=1311121 - Can’t hit it today - Open discussion on change/integration with vagrant - Check trello for issue management - OPEN * It requires license to use github integration :( - Add default init file in the current dir - OPEN * https://github.com/lago-project/lago/issues/165 - Add multiple prefixes definitions on the init config file - OPEN * https://github.com/lago-project/lago/issues/164 - Add multishell (cssh/dssh/tmux/screen) and tailing - OPEN * https://github.com/lago-project/lago/issues/163 -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

time keeping
by Yaniv Kaul 14 Mar '16

14 Mar '16

I'm concerned with two (different) time keeping issues: 1. Timezone - we need to ensure all VMs have the same timezone (At least by default). Best option would be to edit the deploy scripts and set the timezone to the host timezone? should be fairly easy to add @TIMEZONE@ variable to the deploy script in a similar manner to what we do today in the domain XML. It'll be in a command like 'ln -s /usr/share/zoneinfo/@TIMEZONE@ /etc/localtime 2. Time drifting - I have a feeling there's a time drift when we do all the freeze-thaw and snap/revert... Perhaps we should run 'ntpdate' after such actions?

2 2

[11-03-2016] Weekly lago release v0.12.0
by David Caro 11 Mar '16

11 Mar '16

Hi everyone! This week we introduced one big change in how lago handles prefixes, we introduced the 'workdir' concept, that is, a directory that will store one or more prefixes, let me show with an example. Imagine that you have an init.yaml that creates an environment with 1 vm, you can run init like this: lago --prefix-name one_vm init init.yaml That will create a workdir at $PWD/.lago with one prefix inside named one_vm and a soft link named 'current' that points to it, so any further command will run on that one_vm prefix: lago status # shows the one_vm prefix path Until there nothing new, but, now you can create more prefixes, if for example, if you have another init config file named two_vms.yaml, you can create another prefix inside that same workdir with: lago --prefix-name two_vms init two_vms.yaml That will add a new prefix to $PWD/.lago, it will not update the current link though, so any command will still run by default on the previous prefix, but you can change the current prefix with: lago set-current two_vms That will make any command that does not specify the --prefix-name option to use that one by default. NOTE: lago init will create only workdirs from now on, but you can still use the old prefixes if you have any around, it will autodetect them as one or the other. Some other changes: Now it will show the libvirt status of the domains Remove the memory balloon device. We don't use it. Ignore zero sized RPM files when merging repos. Added ignore-warnings option To see the full changelog: http://resources.ovirt.org/repos/lago/stable/0.0/CHANGELOG_0.0.txt And as usual, you can get the rpms from: http://resources.ovirt.org/repos/lago/stable/0.0/rpm/ Cheers! -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

Fwd: [ovirt-devel] [VDSM] Moving to py.test
by Yaniv Kaul 10 Mar '16

10 Mar '16

We probably want to move to it in ovirt-system-tests as well, to align with both VDSM and Lago itself, no? ---------- Forwarded message ---------- From: Nir Soffer <nsoffer(a)redhat.com> Date: Thu, Mar 10, 2016 at 3:15 PM Subject: [ovirt-devel] [VDSM] Moving to py.test To: devel <devel(a)ovirt.org>, Yaniv Bronheim <ybronhei(a)redhat.com>, Dan Kenigsberg <danken(a)redhat.com>, Piotr Kliczewski <pkliczew(a)redhat.com>, Francesco Romani <fromani(a)redhat.com>, Adam Litke <alitke(a)redhat.com> Hi all, In the last vdsm call we discussed moving to py.test. Here is how to run the current tests with py.test: cd tests PYTHONPATH=../lib:../vdsm py.test -v *_test.py *_tests.py *Tests.py network Currently we have some failures and errors: 8 failed, 1662 passed, 101 skipped, 14 pytest-warnings, 10 error in 86.05 seconds Some py.test nice tricks: Finding the 10 slowest tests: PYTHONPATH=../lib:../vdsm py.test -v --durations=10 vmTests.py *_test.py *_tests.py *Tests.py network Checking code coverage: PYTHONPATH=../lib:../vdsm py.test -v --cov=../vdsm --cov=vdsm *_test.py *_tests.py *Tests.py network Once we finish moving vdsm code to lib/vdsm, renaming the tests to *_test.py, and add pytest.ini, running the test will be simply: py.test Once all the tests pass, we can: - Drop all the special asserts (assertXXX), py.test uses plain asserts https://pytest.org/latest/assert.html#asserting-with-the-assert-statement - Get better error reports https://pytest.org/latest/example/reportingdemo.html - Remove unneeded classes, or inherit from any class https://pytest.org/latest/getting-started.html - Replace @permutations with more powerfull @pytest.mark.parametrize https://pytest.org/latest/parametrize.html#parametrize-basics - Replace slowtest and stresstest with @pytest.mark https://pytest.org/latest/mark.html - Replace monkeypatch.py with simpler and more powerfull pytest monkeypatch funcarg https://pytest.org/latest/monkeypatch.html#monkeypatching-mocking-modules-a… - Replace NamedTemporaryDir with pytest tmpdir funcarg https://pytest.org/latest/getting-started.html#going-functional-requesting-… - Simplify complex context managers with pytest fixtures https://pytest.org/latest/example/simple.html - Drop testrunner.py, most of testlib.py, some of testValidation.py, run_tests*.sh - And more Nir _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

2 1

Bridge Networking
by Eric Helms 08 Mar '16

08 Mar '16

I couldn't find anything in the documentation so asking here, is there a way to setup bridge networking for a set of VMs to get them public IPs? Thanks, Eric

3 7

Fwd: [VDSM] check_merged failures
by Yaniv Bronheim 08 Mar '16

08 Mar '16

forwarding to lago devels.. shouldn't be any issues with the tests. nothing changed since last time it succeeded ---------- Forwarded message ---------- From: Nir Soffer <nsoffer(a)redhat.com> Date: Tue, Mar 8, 2016 at 4:34 PM Subject: [VDSM] check_merged failures To: devel <devel(a)ovirt.org>, Yaniv Bronheim <ybronhei(a)redhat.com> Hi all, The new check_merged job is failing randomly. I have see this error today: 14:20:16 @ Start specified VMs: 14:20:16 # Start nets: 14:20:16 * Create network vdsm_functional_tests_lago: 14:20:16 libvirt: error : internal error: Network is already in use by interface 7f0c-bc616ef 14:20:16 * Create network vdsm_functional_tests_lago: ERROR (in 0:00:00) 14:20:16 # Start nets: ERROR (in 0:00:00) 14:20:16 @ Start specified VMs: ERROR (in 0:00:00) 14:20:16 Error occured, aborting 14:20:16 Traceback (most recent call last): 14:20:16 File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 539, in main 14:20:16 cli_plugins[args.verb].do_run(args) 14:20:16 File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run 14:20:16 self._do_run(**vars(args)) 14:20:16 File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 132, in wrapper 14:20:16 return func(*args, prefix=lago_prefix.Prefix(prefix_path), **kwargs) 14:20:16 File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 141, in wrapper 14:20:16 return func(*args, prefix=prefix, **kwargs) 14:20:16 File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 191, in do_start 14:20:16 prefix.start(vm_names=vm_names) 14:20:16 File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 819, in start 14:20:16 self.virt_env.start(vm_names=vm_names) 14:20:16 File "/usr/lib/python2.7/site-packages/lago/virt.py", line 187, in start 14:20:16 net.start() 14:20:16 File "/usr/lib/python2.7/site-packages/lago/virt.py", line 354, in start 14:20:16 self._env.libvirt_con.networkCreateXML(self._libvirt_xml()) 14:20:16 File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4093, in networkCreateXML 14:20:16 if ret is None:raise libvirtError('virNetworkCreateXML() failed', conn=self) 14:20:16 libvirtError: internal error: Network is already in use by interface 7f0c-bc616ef Hopefully someone is owning the infrastructure and can look into this. Nir -- *Yaniv Bronhaim.*

2 3

[2016-03-07] Lago weekly meeting minutes
by David Caro 07 Mar '16

07 Mar '16

== Discussions: * Considering taiga (https://tree.taiga.io/project/david-caro-lago/kanban) as task board (has some github integration, looking) * Check trello first * Move to vagrant/oh-my-vagrant * It turns out that oh-my-vagrant has some kind of repo crawler that can make up for the lack of an open-source vagrant boxes repo service * No layered images * The size of a vagrant box is ~2 times the size of our current images * Completely replace current virt code? Totally replace lago? * Vitali (student) proposed to write the code as project * Written in ruby (some python libs) * https://pypi.python.org/pypi/python-vagrant * Change from json to yaml - DONE * Multiple ‘prefixes’ in the same workdir workflows, now that .lago is the default prefix path (see * Should we use multiple envs per prefix? or one prefix per env? * Approved * https://github.com/lago-project/lago/pull/157 * Add default init file in the current dir * https://github.com/lago-project/lago/issues/165 * Add multiple prefixes definitions on the init config file * https://github.com/lago-project/lago/issues/164 * Add multishell (cssh/dssh/tmux/screen) and tailing * https://github.com/lago-project/lago/issues/163 == Last week action items: * Start a thread to discuss the reposync drop: * http://lists.ovirt.org/pipermail/lago-devel/Week-of-Mon-20160307/000027.html * Open github issues for all the agreed features: * https://github.com/lago-project/lago/issues/149 * https://github.com/lago-project/lago/issues/148 * https://github.com/lago-project/lago/issues/147 * Change the ovirt 4.0 publisher to publish the 3.6 sdk (until 4.0 is ready, then publish both on 4.0 repo) - DONE == Action items: * add --set-current option to init * https://github.com/lago-project/lago/issues/162 * send bugzilla about nfs issues on 3.6 ovirt system tests * https://bugzilla.redhat.com/show_bug.cgi?id=1311121 * Open discussion on change/integration with vagrant * Check trello for issue management * It requires license to use the github integration :( -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

Drop reposync command
by David Caro 07 Mar '16

07 Mar '16

Hi everyone, I'd like to start a discussion about dropping (at least temporarily) the reposync command. The reposync command is used in lago for two things: * Downloads locally all the packages defined in the repo conf passed (yum config file format) * Creates an internal_repo dir in the prefix and copies over all the packages that match the distros used in the prefix The advantage is that once synced, this allows installing any packages on the vms without having connection to the internet. The downside is that right now, it downloads everything, needed or not, so it takes a lot of space and time the first run to cache all the packages. For that reason on the ovirt system tests we removed the epel/centos base repos from the reposync (it was ~20G) but as that already makes the tests require an internet connection, makes the reposync command obsolete (it is caching only some part of the packages). So the idea is to drop the reposync command until we have a better alternative (maybe a small local proxy that caches everything on the first run or something similar) -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

3 2

[04-03-2016] Weekly lago release - 0.11.0
by David Caro 04 Mar '16

04 Mar '16

Hello everyone! Mostly maintenance week this one, almost duplicated the number of tests, and removed more than 2000 lines of unused stuff. But still a couple features got in: * new --prefix-path option, that will allow you to override the automatic search for a prefix, and allow you to run a lago command for a prefix from anywhere * Added YAML support for the init configuration files, now it's a bit more easy to write the init files (you can even reuse blocks of yaml with yaml anchors! https://en.wikipedia.org/wiki/YAML#Repeated_nodes) * Added a new 'destroy' command, that will run any stop/cleanup in the prefix and finally rm -rf it For the full changelog go: http://resources.ovirt.org/repos/lago/stable/0.0/CHANGELOG_0.0.txt You can get the new rpms from: http://resources.ovirt.org/repos/lago/stable/0.0/rpm/ Enjoy! -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

Was the check for 'lago' group membership removed?
by Yaniv Kaul 02 Mar '16

02 Mar '16

I uninstalled and reinstalled lago, which made the lago group disappear and reapper - and my user was not part of it. Then running the tests failed, as I could not write the lock file to /var/lib/lago/... Added a bit of code to the exception to see why: File "/usr/lib/python2.7/site-packages/lockfile/linklockfile.py", line 20, in acquire raise LockFailed("Failed to create %s.\nCode: %d\nReason: %s\n" % (self.unique_name, e.errno, e.strerror)) LockFailed: Failed to create /var/lib/lago/store/ykaul-mini.tlv.redhat.com-fa517700.1377-7176078113883238881. Code: 13 Reason: Permission denied And indeed, it was because I was not part of the lago group. Adding, logging out and in again solved it - but shouldn't there be such warning ? (I remember there was in the past). TIA, Y.

2 1

'make check' fails due to yaml
by Yaniv Kaul 01 Mar '16

01 Mar '16

After today's (?) changes: tests/unit/test_config.py::test_nonexistent_throws PASSED ===================================================================================================== ERRORS ====================================================================================================== ___________________________________________________________________________________ ERROR collecting tests/unit/test_dirlock.py ___________________________________________________________________________________ tests/unit/test_dirlock.py:27: in <module> from lago import dirlock lago/dirlock.py:25: in <module> import utils lago/utils.py:36: in <module> import yaml E ImportError: No module named yaml _________________________________________________________________________________ ERROR collecting tests/unit/lago/test_prefix.py _________________________________________________________________________________ tests/unit/lago/test_prefix.py:6: in <module> from lago import prefix lago/prefix.py:36: in <module> import subnet_lease lago/subnet_lease.py:45: in <module> import utils lago/utils.py:36: in <module> import yaml E ImportError: No module named yaml _________________________________________________________________________________ ERROR collecting tests/unit/lago/test_utils.py __________________________________________________________________________________ tests/unit/lago/test_utils.py:2: in <module> import yaml E ImportError: No module named yaml ======================================================================================== 7 passed, 3 error in 0.08 seconds ======================================================================================== Makefile:52: recipe for target 'check-local' failed make: *** [check-local] Error 1 I'm on: commit 5178f92f435791a778d4fbd801baab532819e7b0 Merge: 5deacae 79bb6ca Author: Lago CI bot <dcaroest+cibot(a)redhat.com> Date: Tue Mar 1 16:38:23 2016 +0100 Merge pull request #151 from lago-project/yaml_init_definition Added yaml support for the init config file

3 3

[29-02-2016] Weekly meeting minutes
by David Caro 29 Feb '16

29 Feb '16

Discussions (continued): ======================== * oVirt sdk version * On oVirt 4.0 two versions of the sdk will be supported, 3 and 4, api version 3 is provided by the ovirt-sdk-3.6 rpms, while the 4 is provided by the 4.0 ones (not ready yet), so we will be using the 3.6 until 4.0 ones are ready, when we will start to adapt to it with oVirt 5.0 as deadline * Change from json to yaml * Maybe support both at first * Nicer multiline * Comments * Nicer quoting **Approved** * Userland runs (running lago without special permissions, qemu:///session) * Network is an issue * http://blog.wikichoon.com/2016/01/qemusystem-vs-qemusession.html * https://github.com/pradels/vagrant-libvirt/issues/272 * http://comments.gmane.org/gmane.comp.emulators.libvirt/117725 * ^ have to chat with the libvirt guys on how to use that * You have to create the bridge nets with superuser privs, and then use the bridges from the conf: * A dump of the domain xml shows: <interface type='bridge'> <mac address='52:54:00:91:98:c6'/> <source bridge='virbr0'/> <target dev='tap0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> * There’s a default bridge (virbr0) always created and available on fedora boxes (by ways of /etc/qemu/bridge.conf), is what gnome-boxes uses * Might be nice to have, even requesting root pass to create bridges when needed * Multiple ‘envs’ in the same prefix workflows, now that .lago is the default prefix path (see * Should we use multiple envs per prefix? or one prefix per env? * Allow selecting the environment, and the prefix dir **Approved** * layered images for vms (will probably have to develop ourselves, at least, the client side) * Make it work on el7 too (maybe just open/push bzs to bring in the needed libguestfs features * Export/import of vm images from prefix? On the long run, being able to export/import full prefixes? * Very interesting to have * Ovirt: * ovirt-standard-ci plugin to build vdsm/ovirt rpms * Being able to run/deploy live vdsm/ovirt (unpackaged) * Drop reposync for now * Run engine in a container/chroot + vdsm in a vm? * Action items: * Start a thread to discuss the reposync drop * Open github issues for all the agreed features * Change the ovirt 4.0 publisher to publish the 3.6 sdk (until 4.0 is ready, then publish both on 4.0 repo) -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

3.6 suddenly fails
by Yaniv Kaul 28 Feb '16

28 Feb '16

Any idea what could be the problem here: @ Deploy oVirt environment: # [Thread-2] Deploy VM lago_basic_suite_3_6_host1: # [Thread-3] Deploy VM lago_basic_suite_3_6_engine: # [Thread-4] Deploy VM lago_basic_suite_3_6_storage-iscsi: # [Thread-5] Deploy VM lago_basic_suite_3_6_storage-nfs: * [Thread-2] Wait for ssh connectivity: # [Thread-6] Deploy VM lago_basic_suite_3_6_host0: * [Thread-3] Wait for ssh connectivity: * [Thread-4] Wait for ssh connectivity: * [Thread-5] Wait for ssh connectivity: * [Thread-6] Wait for ssh connectivity: * [Thread-5] Wait for ssh connectivity: Success (in 0:01:15) * [Thread-5] Run script setup_storage_nfs.sh: * [Thread-4] Wait for ssh connectivity: Success (in 0:01:17) * [Thread-4] Run script setup_storage_iscsi.sh: * [Thread-5] Run script setup_storage_nfs.sh: Success (in 0:00:09) * STDERR mke2fs 1.42.8 (20-Jun-2013) mount: wrong fs type, bad option, bad superblock on /dev/vdc1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. # [Thread-5] Deploy VM lago_basic_suite_3_6_storage-nfs: ERROR (in 0:01:25) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 45, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 469, in _deploy_host host.name(), RuntimeError: /home/mini/ovirt-system-tests/basic_suite_3.6/deploy-scripts/setup_storage_nfs.sh failed with status 64 on lago_basic_suite_3_6_storage-nfs

2 2

[26-02-2016] Weekly lago release - 0.10.0
by David Caro 26 Feb '16

26 Feb '16

Hi everyone! This week we have a couple of very interesting new features: * Using .lago as default prefix for init, that means that if you run init with just the json configuration file, it will create the prefix at $PWD/.lago, but, if you did so, then any command that you run on any dir that's under the $PWD dir that has the .lago prefix, will be run on that prefix! For example: > cd /my/awesome/project > lago init myenv.json # this create the prefix at /my/awesome/project/.lago > cd src/mymodule > lago status # this will run on the /my/awesome/project/.lago prefix automatically! > lago start > lago shell myel7_vm ... * Ouptut format for info gathering commands, for now, just the status and snapshot -l commands. Now you can add the --out-format|-f parameter to lago, and when running a command to extract info from the prefix, it will format the ouptut (yaml, json and 'default' are implemented for now). For example: > lago -f json status { "Prefix": { "Base directory": ..., "Networks": { "n0": { "gateway": "192.168.201.1", "management": true, "status": "down" }, ... }, "UUID": "fe689d3c39d011e5b22b54ee755a00ca", "VMs": { "vm0": { "NICs": { ... }, "VNC port": null, "distro": "cirros", "metadata": {}, "root password": "123456", "snapshots": "", "status": "down" } } } } * Added the public template server as the default template repo, so you don't have to supply it each time * Using 'direct' as default libguestfs backend, that should get rid of the annoying errors when bootstrapping the domains on prefix initialization. The full changelog: http://resources.ovirt.org/repos/lago/stable/0.0/CHANGELOG_0.0.txt -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 1

Fwd: [19-02-2016] Weekly lago release - 0.9.0
by David Caro 26 Feb '16

26 Feb '16

Hi everyone! This week we have a couple of very interesting new features: * Using .lago as default prefix for init, that means that if you run init with just the json configuration file, it will create the prefix at $PWD/.lago, but, if you did so, then any command that you run on any dir that's under the $PWD dir that has the .lago prefix, will be run on that prefix! For example: > cd /my/awesome/project > lago init myenv.json # this create the prefix at /my/awesome/project/.lago > cd src/mymodule > lago status # this will run on the /my/awesome/project/.lago prefix automatically! > lago start > lago shell myel7_vm ... * Ouptut format for info gathering commands, for now, just the status and snapshot -l commands. Now you can add the --out-format|-f parameter to lago, and when running a command to extract info from the prefix, it will format the ouptut (yaml, json and 'default' are implemented for now). For example: > lago -f json status { "Prefix": { "Base directory": ..., "Networks": { "n0": { "gateway": "192.168.201.1", "management": true, "status": "down" }, ... }, "UUID": "fe689d3c39d011e5b22b54ee755a00ca", "VMs": { "vm0": { "NICs": { ... }, "VNC port": null, "distro": "cirros", "metadata": {}, "root password": "123456", "snapshots": "", "status": "down" } } } } * Added the public template server as the default template repo, so you don't have to supply it each time * Using 'direct' as default libguestfs backend, that should get rid of the annoying errors when bootstrapping the domains on prefix initialization. The full changelog: http://resources.ovirt.org/repos/lago/stable/0.0/CHANGELOG_0.0.txt -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 1

Testing oVirt 4.0 with Lago - which SDK should we use?
by Yaniv Kaul 22 Feb '16

22 Feb '16

Currently, the SDK that Lago uses is (at least on my Fedora): ovirt-engine-sdk-python-3.5.1.0-2.fc23.noarch I think we may need to bring in a newer SDK. 1. Which one? 2. If master - is it backward compatible to run against 3.x already? 3. Can we have them side by side perhaps? TIA, Y.

3 3

[ovirt-system-tests] What is the correct value of the engine's answer file entry OVESETUP_CONFIG/fqdn ?
by Yaniv Kaul 22 Feb '16

22 Feb '16

[ykaul@ykaul answer-files]$ pwd /home/ykaul/ovirt-system-tests/common/answer-files [ykaul@ykaul answer-files]$ grep -R "OVESETUP_CONFIG/fqdn" * |sort el6_3.5.conf:OVESETUP_CONFIG/fqdn=str:engine el6_3.6.conf:OVESETUP_CONFIG/fqdn=str:*localhost.localdomain* el6_master.conf:OVESETUP_CONFIG/fqdn=str:engine el7_3.5.conf:OVESETUP_CONFIG/fqdn=str:engine el7_master.conf:OVESETUP_CONFIG/fqdn=str:engine Is the value localhost.localdomain correct for el6-3.6 (only?). TIA, Y.

2 2

[19-02-2016] Weekly lago release - 0.9.0
by David Caro 19 Feb '16

19 Feb '16

Hi everyone! This week release is small, and has not big changes, but a big bump was done on adding some functional tests and starting to move toward py.test for unit tests, the relevant changes: dcaf87c6: Moving to py.test 840d3956: Fix start a bridged network that's already alive a20cff51: Avoiding ssh connection for hosts that have no ovirt-scripts gned bbcebcd0: Adding support for ovirt node specific VM's You can get it fresh from: http://resources.ovirt.org/repos/lago/stable/0.0/rpm/ The full changelog: http://resources.ovirt.org/repos/lago/stable/0.0/CHANGELOG_0.0.txt Enjoy! -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

[05-02-2016] Weekly lago release - 0.8.0
by David Caro 05 Feb '16

05 Feb '16

Hi everyone! This week release has a critical bug fix and a new subcommand, the relevant changes: ee68f156: Added console cli command 54e02ff8: Adding use of backing mode for qcow2 files 6b174850: Moving from sftp to scp for copy to/from vms 24f83bf8: Fixed issue with copy-from-vm You can get it fresh from: http://resources.ovirt.org/repos/lago/stable/0.0/rpm/ The full changelog: http://resources.ovirt.org/repos/lago/stable/0.0/CHANGELOG_0.0.txt Some other news: * We did a presentation at FOSDEM! The videos are not yet ready, but you can get the slides here: http://redhat.slides.com/dcaroest/raiders_of_the_lost_testing_env * There's a new image repo with bare images being tested here: http://templates.ovirt.org/repo/unstable/ To use it just pass http://templates.ovirt.org/repo/unstable/repo.metadata as the --templates-repo-path option when running lago (take into account that you'll have to change the template names in the init.json you use too) Enjoy! -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

Re: [lago-devel] Fwd: test failed, Failed to connect to the engine
by Yaniv Kaul 02 Feb '16

02 Feb '16

On Thu, Jan 28, 2016 at 3:01 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote: > > > On Thu, Jan 28, 2016 at 2:56 PM, Maor Lipchuk <mlipchuk(a)redhat.com> wrote: > >> >> >> >> >> ----- Original Message ----- >> > From: "Yaniv Kaul" <ykaul(a)redhat.com> >> > To: "Maor Lipchuk" <mlipchuk(a)redhat.com> >> > Cc: lago-devel(a)ovirt.org >> > Sent: Thursday, January 28, 2016 12:59:38 PM >> > Subject: Re: [lago-devel] Fwd: test failed, Failed to connect to the >> engine >> > >> > On Thu, Jan 28, 2016 at 12:30 PM, Maor Lipchuk <mlipchuk(a)redhat.com> >> wrote: >> > >> > > Hi all, >> > > >> > > I've been trying to use the LAGO setup, all worked well up to the >> point >> > > which I got an error on one of the tests : >> > > >> > > Run test: 002_bootstrap.py: ERROR >> > > Error occured, aborting >> > > .... >> > > ... line 82, in do_ovirt_runtest >> > > raise RuntimeError('Some tests failed') >> > > RuntimeError: Some tests failed >> > > >> > > Looking in the nosetests it looks like it failed to connect to the >> engine: >> > > >> > > File "usr/lib/python2.7/site-packaged/pvortlago/virt.py", line 106, >> in >> > > _get_api >> > > raise RuntimeError('Failed to connect to the engine') >> > > >> > >> > Can you check if the engine is up? In my case, it was not installed, as >> the >> > answer file was incorrect. >> > See https://bugzilla.redhat.com/show_bug.cgi?id=1302045 for details and >> > solution. >> > Y. >> >> I don't see that ovirt-engine is installed on my env, but I do see that >> the ovirt-release36-002-2 is installed. >> I was trying to set the boolean to false as was mentioned in the bug to >> see if that make any difference but still didn't had much luck. >> > > Need to look at the engine's setup log, then. > I prepared a patch to the installation test that: 1. Verifies the installation succeeded (based on error code from the installation) 2. Verifies the ovirt-engine service is up. Only if both succeed, the test passes. I've cloned the specific bug above to a general bug @ https://bugzilla.redhat.com/show_bug.cgi?id=1303973 Y. > Y. > > >> >> > >> > > >> > > tests="6" error="1" >> > > >> > > Is it only a test issue or is there something wrong with my engine? >> > > >> > > Regards, >> > > Maor >> > > >> > > >> > > _______________________________________________ >> > > lago-devel mailing list >> > > lago-devel(a)ovirt.org >> > > http://lists.ovirt.org/mailman/listinfo/lago-devel >> > > >> > >> > >

1 0

Lago mailing list archives
by David Caro 02 Feb '16

02 Feb '16

Hi everyone!! Last week we realized that the archives for the mailing list were not working, and that has been solved today. Painfully all the emails that were sent before were not actually archived so they have been lost in the archives, but from now on they will be properly archived and backed up. Sorry for the inconvenience :/ -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0

TEST-thisisatestemail@akhos.redhat.com, pleaseignore@akhos.redhat.com
by David Caro 02 Feb '16

02 Feb '16

Dummy test email -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

1 0