- Devel - oVirt List Archives

Using travis yaml files to specify dependencies and tests
by David Caro 03 Feb '15

03 Feb '15

--oLBj+sq0vYjzfsbl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi everyone! After talking a bit with some of you, I think that we can start planning a common build and dependency declaration for tests for the ovirt products, to improve and automate most of the ci process and maintenance. Current status: =3D=3D Dependencies Right now we have 4 types of dependencies: * test dependencies * tarball/srcrpm build dependencies * rpm build dependencies * installation dependencies The last two are managed from the spec files through rpm/yum dependency systems. But the first ones are managed manually on the jobs on jenkins or puppet manifests. What separates it from the code that actually requires them and adds an extra layer of maintenance and synchronization between the code, the jenkins jobs and the puppet repository. =3D=3D Builds We started using autotools to build most of the projects, but it's not a global methodology and even being used on some projects, you need to tune the run for each of them, specifying different variables and running some side scripts. =3D=3D Tests Some projects use make check to run some of the tests, some tests are totally outside the code and run only in jenkins jobs. Some possible improvements: =3D=3D Tests/builds Using shell scripts: We talked before in another thread to create some generic script to build the artifacts for the product and to run the tests, namely we talked about having 3 executables (bash scripts probably) at the root of each project, that should not require any parameters: ./build-artifacts This should generate any artifacts to be archives (isos, rpms, debs, tarballs, ...) and leave them at ./exported-artifacts/ directory, for the build system to collect, removing any previous artifacts if needed. ./check_patch Runs all the tests required for any new patchset in gerrit, for non-merged changes, should be fast to run to get feedback easily ./check_merge Runs all the tests for any change that is going to be merged (right now we are not using gates, so it actually after merge, but the idea is to use this as gate for any merge to the repo). This can be more resource hungry than check_path. That way it will let you use easily any framework that you want to use for your project, but still let it be easy to maintain in the global ci/build system (you can use pip, tox, maven, gradle, autotools, make, rake, ...). This will not allow at first running tests in parallel in jenkins, but we can in the future add that possibility (for example, allowing the user to define more than one check script, like check_patch.mytest1 and check_patch.mytest2 and make jenkins run them in parallel). I started a POC of this process here [1] Using travis yaml files: Using a travis compliant yaml file [2]. That will be less flexible than the above solution and will not allow you to run tests in parallel, though it will let you use travis at any point to offload our ci if needed. =3D=3D Dependencies Using plain text files: Similar to the above scripts solution, I though of adding an extra file, with the same name, to declare the dependencies to run that script. Adding a suffix in case of different requirements for different distros (matching facter fact strings), for example: ./build-artifacts.req Main requirements file, used if no more specific one found. With a newline separated list of packages to install on the environment (jenkins will take care of which package manager to use). ./build-artifacts.req.fc20 Specific requirements for fc20 environment, replaces the general one if present. And the same for the other scripts (check_patch and check_merge). Using travis yaml file: Using a travis compliant yaml file with some extensions to declare the different dependencies for each type and os/distro. That will allow you to have only one extra file in your repo, though you'd have to duplicate some of the requirements as travis only has ubuntu and forces you to run scripts to install dependencies. What do you think? Do you have any better idea? ps. About using an external repository to store the scripts/requirements for the code. The issue with this is that it forces you to bind a code change in the code repo, to a script/dependency change in the scripts repo, and that adds a lot of extra maintenance and source of issues and failures. If you know a way of doing it like that without all the fuss, I'd love to hear it. For example, imagine that you have vdsm and the dependencies are in another repo, now you send a patch to vdsm that requires you to run a specific pep8 version to pass the patch tests, so you have to change the script repo to add that dependency, but doing that you will brake all the other patches tests because they require the older pep8 version, so you have to somehow specify in the vdsm patch that you require a specific commit from the scripts repo to be tested with... Having both in the same repo, allows you to do the code change and the dependency/script change in the same patchset, and test it right away with the correct scripts/deps. It also binds together code and tests to some point, what is nice to have in a product view, because you know for each version which tests it passed and have a better idea of the possible failures for that version. [1] http://gerrit.ovirt.org/#/admin/projects/repoman [2] http://docs.travis-ci.com/ --=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com Web: www.redhat.com RHT Global #: 82-62605 --oLBj+sq0vYjzfsbl Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUvnkrAAoJEEBxx+HSYmnD2f0H/1t40fR3h4Tlae891vmiPyYv zxfTY6SlW0CJkSEis2ED2KZTkQZwQ6clfK+aRdGzSXL0mtUL9wr3Kk1/+i4tMQD/ 5gqjP9Hc7HoMzDDvIRIvBdBm3fDgnvChkqr8xbRG1l4LIDGgDH2AE9uhkEClHpqV CdhwrYGAU9DCBcB1kINZqKPn7q42nR1fkvl2ItIlJ6eRpY51bf6PNIW2INNvVB7V 3LMkO9E2Y1uen61HZ5A7MvQ58vaJJjxM4seCrcdkbAEtHxbtPxVAEG7mXlpZBcH6 EbT1KfLAyghRcpdGxa5PsAQRHcrTre+sKhsi17fI8JZBhJVmAfdbHd2iEhMlqZ8= =LXWV -----END PGP SIGNATURE----- --oLBj+sq0vYjzfsbl--

4 6

reporting and removing unmanaged networks after deprecating org.ovirt.engine.core.common.action.VdcActionType#SetupNetworks
by Martin Mucha 03 Feb '15

03 Feb '15

Hi, I'd like to discuss how to properly report unmanaged networks and how to ask for their removal after org.ovirt.engine.core.common.action.VdcActionType#SetupNetworks is removed. We thought about several possibilities and so far the best one is following one. Reporting unmanaged networks on specific nic: ——————————————————————————————————————————————— We'd like to return new collection under: GET http://localhost:8080/api/hosts/{id}/nics/{id}/unmanagednetworks returning (reporting) unmanaged networks like this: <unmanaged_networks> <unmanaged_network> <nic_name>...</nic_name> <unmanaged_network_name>...</unmanaged_network_name> <vland_id>...</vland_id> </unmanaged_network> <unmanaged_network> ... </unmanaged_network> </unmanaged_networks> Removing unmanagedNetworks: ————————————————————————————— DELETE http://localhost:8080/api/hosts/{id}/nics/{id}/unmanagednetworks/{unmanaged… ======================= any ideas, hints, complaints, recommendations, confirmations are welcomed. Martin.

3 2

Get involved in oVirt project! February edition
by Sandro Bonazzola 03 Feb '15

03 Feb '15

Hi, do you want to get involved in oVirt project? Here are some bugs you can hopefully fix in less that one day or you can just try to reproduce providing info: Bug ID Whiteboard Status Summary 1059952 integration NEW hosted-engine --deploy (additional host) will fail if the engine is not using the default self-signed CA 1065350 integration NEW hosted-engine should prompt a question at the user when the host was already a host in the engine 1073421 integration NEW [RFE] allow additional parameter for engine-backup to omit audit_log data 1080823 integration NEW [RFE] make override of iptables configurable when using hosted-engine 1083104 integration NEW engine-setup --offline does not update versionlock Do you want something easier? Bug ID Whiteboard Status Summary 1174285 i18n NEW [de-DE] "Live Snapshot Support" reads "Live Snapsnot Support" 772931 infra NEW [RFE] Reports should include the name of the oVirt engine 1143817 integration NEW [TEXT ONLY] - Hosted Engine - Instructions for FQDN are not clear enough 1156060 integration NEW [text] engine admin password prompt consistency 1115059 network NEW Incomplete error message when adding VNIC profile to running VM 734120 storage NEW [RFE] VDSM: use virt-sparsify/zerofree to reduce image size Do you love "DevOps?", you count stable builds in jenkins ci while trying to fall a sleep? Then oVirt infra team is looking for you!, join the infra team and dive in to do the newest and coolest devops tools today! Here are some of our open tasks you can help with: https://fedorahosted.org/ovirt/report/1 You don't have programming skills, not enough time for DevOps but you want still to contribute? Here are some bugs you can take care of, also without writing a line of code: Bug ID Whiteboard Status Summary 1074545 docs NEW Error in API documentation: Create API object in python sdk 1099995 docs NEW Migrate to Hosted Engine How-To does not state all pre-reqs 1099998 docs NEW Hosted Engine documentation has several errors 1159784 docs NEW [RFE] Document when and where new features are available when upgrading cluster / datacenters 1074301 infra NEW [RFE] ovirt-shell has no man page 1120585 integration NEW update image uploader documentation 1120586 integration NEW update iso uploader documentation 1120588 integration NEW update log collector documentation Do you prefer to test things? We have some test cases[5] you can try using nightly snapshots[6] Do you want to contribute test cases? Most of the features[7] included in oVirt are missing a test case, you're welcome to contribute one! Is this the first time you try to contribute to oVirt project? You can start from here [1][2]! Don't know gerrit very well? You can find some more docs here [3]. Any other question about development? Feel free to ask on devel(a)ovirt.org or on irc channel[4]. [1] http://www.ovirt.org/Develop [2] http://www.ovirt.org/Working_with_oVirt_Gerrit [3] https://gerrit-review.googlesource.com/Documentation [4] http://www.ovirt.org/Community [5] http://www.ovirt.org/Category:TestCase [6] http://www.ovirt.org/Install_nightly_snapshot [7] http://www.ovirt.org/Category:Feature -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0

CORS enabled for oVirt REST API
by Jenny Kang 02 Feb '15

02 Feb '15

Hello, As part of my OPW project, I'm trying to build a mobile web UI for oVirt but I'm having some troubles. I cannot access the oVirt REST API because it doesn't allow cross origin resource sharing (CORS). The only way to access the API is to host the UI on the same IP as the engine. If it is enabled then people would be able to run the mobile UI directly from the desktop without hosting it anywhere. Do you have any suggestions on how to access oVirt REST API from another host inside the browser? Any plans on enabling CORS on the REST API? Thank you! Cheers Jenny

5 7

[ANN] Async release for ovirt-engine is now available.
by Sandro Bonazzola 02 Feb '15

02 Feb '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The oVirt Team has just released a new version of ovirt-engine package for 3.5.1.1 that fixes a severe regression introduced in 3.5.1[1]. We recommend to all users who have already upgraded to 3.5.1 to update to this package. Thanks, [1] https://bugzilla.redhat.com/1185615 - NPE in AddDiskCommand during ovf volume creation. - -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJUz1RxAAoJEHw3u57E4QAOZrEP/25VMKJFKB2mis6PXLC4LAE6 Z4c6gXNBFN+jMIDB7N4NIVH8YXN+wMm/iy2V16aG7MObFueIyHc2GuuFRKPdOh7k PL91aQsXolrfpU0nr7dTk6UvWIXb37NKr3IlIX2mc/03YqzA4fqOJGt53a2OSCWI bAI9TRlg1aZWdAEUQ0iBsJqvXw8GuFo+1HuuvIZhtK8zCEOd1djfmyYw89Upd7/k VpfCqatQL+Zl2dltO5tktlW5gUnmo6pBtZzKlAS1ajDhwb5NPGbFU/JlJ5ToWQD5 W6bZ84N4bGnG3LfLkYFWJhRWY2+5Vsj3Kry5mKoJZdUwjI+6LODi6uyzGCT1DegJ mUrPljMv0c0BBJzVcmiDPn2BXFLuXbkmvMgpbWqlBNSixhoXCKE8jdfSElEZ0AWZ Vav98YhNSSv1R4bf/DBj1SQVLNaW+BdU22kmqDsZK4futmOh3V38jTvTQUrpYFnu 8kYFj2GvUyC1R3W7PgswHnr1weWtEbZnvsLRBlXw/QLC2zgMUqrWlbGDKHD+e/I7 RJ0HYyUM1BIrpDjcawdA48rjm2O3Y58dUcferFJdMYzX595R1QvUo+05J9mYLJGE pLRtExjWvMGJfK30zV9LB4y6VvIbj+gwMK3NfN7vBkK7I311Rqqv3KDS6RheZECL ic6WURcJRolasO/uOH5Z =zAuW -----END PGP SIGNATURE-----

2 2

oVirt 3.6 Feature: Cumulative Network Usage Statistics
by Lior Vernia 01 Feb '15

01 Feb '15

Hello users and developers, Just put up a feature page for the aforementioned feature; in summary, to report total RX/TX statistics for hosts and VMs in oVirt. This has been requested several times on the users mailing list, and is especially useful for accounting in VDI deployments. You're more than welcome to review the feature page: http://www.ovirt.org/Features/Cumulative_RX_TX_Statistics Note that this only deals with network usage - it'll be great if we have similar features for CPU and disk usage! Yours, Lior.

3 5

[DB] Diffrent UUIDs for inserted data per installation
by Eli Mesika 29 Jan '15

29 Jan '15

Hi I am trying to cleanup all the data insertion to the engine DB and make it more general The main drive to that is DB version squashing that was done manually and therefor was error prone ... I know that both storage_pool_id (a.k.a DC) and vds_group_id (a.k.a Cluster) needs to get a different UUID per installation. But I had found that UUIDs are generated per installation for also : table | column/s ------------------------------------ [cpu_profiles] : id [gluster_services] : id [mac_pools] : id [permissions] : id, object_id [vm_device]: device_id, vm_id [vm_static] : vm_guid [vnic_profiles] : id Please let me know if any of the above should be generated using the uuid_generate_v1() function on each installation or we can have static IDs for those. Thanks Eli Mesika

4 4

Upgrade job is failing while upgrading from 3.4.4 to 3.5.2-snapshot:
by Sandro Bonazzola 29 Jan '15

29 Jan '15

Upgrade job is failing while upgrading from 3.4.4 to 3.5.2-snapshot: 2015-01-27 15:02:46 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.upgrade.asynctasks plugin.execute:866 execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20150127150216-p4ixpj.log', '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-R', '-A', '-J', '-q') stderr: psql:/usr/share/ovirt-engine/setup/dbutils/taskcleaner_sp.sql:145: ERROR: type "command_entities" does not exist FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/setup/dbutils/taskcleaner_sp.sql 2015-01-27 15:02:46 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 422, in _validateZombies self._clearZombieTasks() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 156, in _clearZombieTasks 'Failed to clear zombie tasks. Please access support ' RuntimeError: Failed to clear zombie tasks. Please access support in attempt to resolve the problem 2015-01-27 15:02:46 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Setup validation': Failed to clear zombie tasks. Please access support in attempt to resolve the problem Please fix ASAP. Reference: http://jenkins.ovirt.org/job/ovirt-engine_3.5_upgrade-from-3.4_merged/label… -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

5 7

DAO test is failing on master
by Sandro Bonazzola 28 Jan '15

28 Jan '15

http://jenkins.ovirt.org/job/ovirt-engine_master_dao-unit-tests_merged/1002… 15:50:12 Failed tests: 15:50:12 scheduleARecurringJob(org.ovirt.engine.core.utils.timer.DBSchedulerUtilQuartzImplTest): Unexpected exception occured -Failed to obtain DB connection from data source 'EngineDS': java.sql.SQLException: Connections could not be acquired from the underlying database! 15:50:12 scheduleAJob(org.ovirt.engine.core.utils.timer.DBSchedulerUtilQuartzImplTest): Unexpected exception occured -Failed to obtain DB connection from data source 'EngineDS': java.sql.SQLException: Connections could not be acquired from the underlying database! Not sure if it's infra or devel issue. -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

6 8

[QE][ACTION REQUIRED] oVirt 3.6.0 status
by Sandro Bonazzola 28 Jan '15

28 Jan '15

Hi, I haven't many news for 3.6 this week: ACTION: Feature proposed for 3.6.0 must now be collected in the 3.6 Google doc [1] and reviewed by maintainers. Finished the review process, the remaining key milestones for this release will be scheduled. For reference, external project schedules we're tracking are: Fedora 22: 2015-05-19 Foreman 1.8.0: 2015-03-01 GlusterFS 3.7: 2015-04-29 OpenStack Kilo: 2015-04-30 QEMU 2.3.0: 2015-03-27 The tracker bug for 3.6.0 [2] currently shows no blockers. There are 530 bugs [3] targeted to 3.6.0. Excluding node and documentation bugs we have 489 bugs [4] targeted to 3.6.0. [1] http://goo.gl/9X3G49 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1155425 [3] https://bugzilla.redhat.com/buglist.cgi?quicksearch=target_release%3A3.6.0%… [4] http://goo.gl/ZbUiMc -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0