December 2016 - Devel - oVirt List Archives

Re: [ovirt-devel] [VDSM] Correct implementation of virt-sysprep job
by Oved Ourfali 07 Dec '16

07 Dec '16

On Dec 7, 2016 20:24, "Michal Skrivanek" <mskrivan(a)redhat.com> wrote: > > > > On 07 Dec 2016, at 09:17, Oved Ourfali <oourfali(a)redhat.com> wrote: > >> >> >> On Tue, Dec 6, 2016 at 11:12 PM, Adam Litke <alitke(a)redhat.com> wrote: >>> >>> On 06/12/16 22:06 +0200, Arik Hadas wrote: >>>> >>>> Adam, >>> >>> >>> :) You seem upset. Sorry if I touched on a nerve... >>> >>>> Just out of curiosity: when you write "v2v has promised" - what exactly do you >>>> mean? the tool? Richard Jones (the maintainer of virt-v2v)? Shahar and I that >>>> implemented the integration with virt-v2v? I'm not aware of such a promise by >>>> any of these options :) >>> >>> >>> Some history... >>> >>> Earlier this year Nir, Francesco (added), Shahar, and I began >>> discussing the similarities between what storage needed to do with >>> external commands and what was designed specifically for v2v. I am >>> not sure if you were involved in the project at that time. The plan >>> was to create common infrastructure that could be extended to fit the >>> unique needs of the verticals. The v2v code was going to be moved >>> over to the new infrastructure (see [1]) and the only thing that >>> stopped the initial patch was lack of a VMWare testing environment for >>> verification. >>> >>> At that time storage refocused on developing verbs that used the new >>> infrastructure and have been maintaining its suitability for general >>> use. Conversion of v2v -> Host Jobs is obviously a lower priority >>> item and much more difficult now due to the early missed opportunity. >>> >>>> Anyway, let's say that you were given such a promise by someone and thus >>>> consider that mechanism to be deprecated - it doesn't really matter. >>> >>> >>> I may be biased but I think my opinion does matter. >>> >>>> The current implementation doesn't well fit to this flow (it requires >>>> per-volume job, it creates leases that are not needed for template's disks, >>>> ...) and with the "next-gen API" with proper support for virt flows not even >>>> being discussed with us (and iiuc also not with the infra team) yet, I don't >>>> understand what do you suggest except for some strong, though irrelevant, >>>> statements. >>> >>> >>> If you are willing to engage in a good-faith technical discussion I am >>> sure I can help you to understand. These operations to storage demand >>> some form of locking protection. If volume leases aren't appropriate then >>> perhaps we should use the VM Leases / xleases that Nir is finishing >>> off for 4.1 now. >>> >>>> I suggest loud and clear to reuse (not to add dependencies, not to enhance, ..) >>>> an existing mechanism for a very similar flow of virt-v2v that works well and >>>> simple. >>> >>> >>> I clearly remember discussions involving infra (hello Oved), virt >>> (hola Michal), and storage where we decided that new APIs performing >>> async operations involving external commands should use the HostJobs >>> infrastructure instead of adding more information to Host Stats. >>> These were the "famous" entity polling meetings. >>> >>> Of course plans can change but I have never been looped into any such >>> discussions. >>> >> >> Well, I think that when someone builds a good infrastructure he first needs to talk to all consumers and make sure it fits. >> In this case it seems like most work was done to fit the storage use-case, and now you check whether it can fit others as well.... >> >> IMO it makes much more sense to use events where possible (and you've promised to use those as well, but I don't see you doing that...). v2v should use events for sure, and they have promised to do that in the past, instead of using the v2v jobs. The reason events weren't used originally with the v2v feature, was that it was too risky and the events infrastructure was added too late in the game. > > > Revisiting and refactoring code which is already in use is always a bit of luxury we can rarely prioritize. So indeed v2v is not using events. The generalization work has been done to some extent, but there is no incentive to rewrite it completely. > On the other hand we are now trying to add events to migration progress reporting and hand over since that area is being touched due to post-copy enhancements. > So, when there is a practical chance to improve functionality by utilizing events it indeed should be the first choice +1. Well understood. > >> >> >>>> >>>> Do you "promise" to implement your "next gen API" for 4.1 as an alternative? >>> >>> >>> I guess we need the design first. >>> >>> >>>> On Tue, Dec 6, 2016 at 5:04 PM, Adam Litke <alitke(a)redhat.com> wrote: >>>> >>>> On 05/12/16 11:17 +0200, Arik Hadas wrote: >>>> >>>> >>>> >>>> On Mon, Dec 5, 2016 at 10:05 AM, Nir Soffer <nsoffer(a)redhat.com> wrote: >>>> >>>> On Sun, Dec 4, 2016 at 8:50 PM, Shmuel Melamud < smelamud(a)redhat.com> >>>> wrote: >>>> > >>>> > Hi! >>>> > >>>> > I'm currently working on integration of virt-sysprep into oVirt. >>>> > >>>> > Usually, if user creates a template from a regular VM, and then >>>> creates >>>> new VMs from this template, these new VMs inherit all configuration >>>> of the >>>> original VM, including SSH keys, UDEV rules, MAC addresses, system >>>> ID, >>>> hostname etc. It is unfortunate, because you cannot have two network >>>> devices with the same MAC address in the same network, for example. >>>> > >>>> > To avoid this, user must clean all machine-specific configuration >>>> from >>>> the original VM before creating a template from it. You can do this >>>> manually, but there is virt-sysprep utility that does this >>>> automatically. >>>> > >>>> > Ideally, virt-sysprep should be seamlessly integrated into >>>> template >>>> creation process. But the first step is to create a simple button: >>>> user >>>> selects a VM, clicks the button and oVirt executes virt-sysprep on >>>> the VM. >>>> > >>>> > virt-sysprep works directly on VM's filesystem. It accepts list of >>>> all >>>> disks of the VM as parameters: >>>> > >>>> > virt-sysprep -a disk1.img -a disk2.img -a disk3.img >>>> > >>>> > The architecture is as follows: command on the Engine side runs a >>>> job on >>>> VDSM side and tracks its success/failure. The job on VDSM side runs >>>> virt-sysprep. >>>> > >>>> > The question is how to implement the job correctly? >>>> > >>>> > I thought about using storage jobs, but they are designed to work >>>> only >>>> with a single volume, correct? >>>> >>>> New storage verbs are volume based. This make it easy to manage >>>> them on the engine side, and will allow parallelizing volume >>>> operations >>>> on single or multiple hosts. >>>> >>>> A storage volume job is using sanlock lease on the modified volume >>>> and volume generation number. If a host running pending jobs becomes >>>> non-responsive and cannot be fenced, we can detect the state of >>>> the job, fence the job, and start the job on another host. >>>> >>>> In the SPM task, if a host becomes non-responsive and cannot be >>>> fenced, the whole setup is stuck, there is no way to perform any >>>> storage operation. >>>> > Is is possible to use them with operation that is performed on >>>> multiple >>>> volumes? >>>> > Or, alternatively, is it possible to use some kind of 'VM jobs' - >>>> that >>>> work on VM at whole? >>>> >>>> We can do: >>>> >>>> 1. Add jobs with multiple volumes leases - can make error handling >>>> very >>>> complex. How do tell a job state if you have multiple leases? >>>> which >>>> volume generation you use? >>>> >>>> 2. Use volume job using one of the volumes (the boot volume?). This >>>> does >>>> not protect the other volumes from modification but engine is >>>> responsible >>>> for this. >>>> >>>> 3. Use new "vm jobs", using a vm lease (should be available this >>>> week >>>> on master). >>>> This protects a vm during sysprep from starting the vm. >>>> We still need a generation to detect the job state, I think we >>>> can >>>> use the sanlock >>>> lease generation for this. >>>> >>>> I like the last option since sysprep is much like running a vm. >>>> > How v2v solves this problem? >>>> >>>> It does not. >>>> >>>> v2v predates storage volume jobs. It does not use volume leases and >>>> generation >>>> and does have any way to recover if a host running v2v becomes >>>> non-responsive >>>> and cannot be fenced. >>>> >>>> It also does not use the jobs framework and does not use a thread >>>> pool for >>>> v2v jobs, so it has no limit on the number of storage operations on >>>> a host. >>>> >>>> >>>> Right, but let's be fair and present the benefits of v2v-jobs as well: >>>> 1. it is the simplest "infrastructure" in terms of LOC >>>> >>>> >>>> It is also deprecated. V2V has promised to adopt the richer Host Jobs >>>> API in the future. >>>> >>>> >>>> 2. it is the most efficient mechanism in terms of interactions between >>>> the >>>> engine and VDSM (it doesn't require new verbs/call, the data is >>>> attached to >>>> VdsStats; probably the easiest mechanism to convert to events) >>>> >>>> >>>> Engine is already polling the host jobs API so I am not sure I agree >>>> with you here. >>>> >>>> >>>> 3. it is the most efficient implementation in terms of interaction with >>>> the >>>> database (no date is persisted into the database, no polling is done) >>>> >>>> >>>> Again, we're already using the Host Jobs API. We'll gain efficiency >>>> by migrating away from the old v2v API and having a single, unified >>>> approach (Host Jobs). >>>> >>>> >>>> Currently we have 3 mechanisms to report jobs: >>>> 1. VM jobs - that is currently used for live-merge. This requires the >>>> VM entity >>>> to exist in VDSM, thus not suitable for virt-sysprep. >>>> >>>> >>>> Correct, not appropriate for this application. >>>> >>>> >>>> 2. storage jobs - complicated infrastructure, targeted for recovering >>>> from >>>> failures to maintain storage consistency. Many of the things this >>>> infrastructure knows to handle is irrelevant for virt-sysprep flow, and >>>> the >>>> fact that virt-sysprep is invoked on VM rather than particular disk >>>> makes it >>>> less suitable. >>>> >>>> >>>> These are more appropriately called HostJobs and the have the >>>> following semantics: >>>> - They represent an external process running on a single host >>>> - They are not persisted. If the host or vdsm restarts, the job is >>>> aborted >>>> - They operate on entities. Currently storage is the first adopter >>>> of the infrastructure but virt was going to adopt these for the >>>> next-gen API. Entities can be volumes, storage domains, vms, >>>> network interfaces, etc. >>>> - Job status and progress is reported by the Host Jobs API. If a job >>>> is not present, then the underlying entitie(s) must be polled by >>>> engine to determine the actual state. >>>> >>>> >>>> 3. V2V jobs - no mechanism is provided to resume failed jobs, no >>>> leases, etc >>>> >>>> >>>> This is the old infra upon which Host Jobs are built. v2v has >>>> promised to move to Host Jobs in the future so we should not add new >>>> dependencies to this code. >>>> >>>> >>>> I have some arguments for using V2V-like jobs [1]: >>>> 1. creating template from vm is rarely done - if host goes unresponsive >>>> or any >>>> other failure is detected we can just remove the template and report >>>> the error >>>> >>>> >>>> We can chose this error handling with Host Jobs as well. >>>> >>>> >>>> 2. the phase of virt-sysprep is, unlike typical storage operation, >>>> short - >>>> reducing the risk of failures during the process >>>> >>>> >>>> Reduced risk of failures is never an excuse to have lax error >>>> handling. The storage flavored host jobs provide tons of utilities >>>> for making error handling standardized, easy to implement, and >>>> correct. >>>> >>>> >>>> 3. during the operation the VM is down - by locking the VM/template and >>>> its >>>> disks on the engine side, we render leases-like mechanism redundant >>>> >>>> >>>> Eventually we want to protect all operations on storage with sanlock >>>> leases. This is safer and allows for a more distributed approach to >>>> management. Again, the use of leases correctly in host jobs requires >>>> about 5 lines of code. The benefits of standardization far outweigh >>>> any perceived simplification resulting from omitting it. >>>> >>>> >>>> 4. in the worst case - the disk will not be corrupted (only some of the >>>> data >>>> might be removed). >>>> >>>> >>>> Again, the way engine chooses to handle job failures is independent of >>>> the mechanism. Let's separate that from this discussion. >>>> >>>> >>>> So I think that the mechanism for storage jobs is an over-kill for this >>>> case. >>>> We can keep it simple by generalise the V2V-job for other virt-tools >>>> jobs, like >>>> virt-sysprep. >>>> >>>> >>>> I think we ought to standardize on the Host Jobs framework where we >>>> can collaborate on unit tests, standardized locking and error >>>> handling, abort logic, etc. When v2v moves to host jobs then we will >>>> have a unified method of handling ephemeral jobs that are tied to >>>> entities. >>>> >>>> -- >>>> Adam Litke >>>> >>>> >>> >>> -- >>> Adam Litke >> >>

1 0

ovirt-node_ovirt-3.6_create-iso-el7_merged job failed
by Shlomo Ben David 07 Dec '16

07 Dec '16

Hi, The [1] job failed with the following error: *10:21:33 Package qemu-kvm-tools is obsoleted by qemu-kvm-tools-ev, trying to install 10:qemu-kvm-tools-ev-2.6.0-27.1.el7.x86_64 instead 10:21:48 Error creating Live CD : Failed to build transaction : 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires usbredir >= 0.7.1 10:21:48 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires seavgabios-bin >= 1.9.1-4 10:21:48 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires ipxe-roms-qemu >= 20160127-4 10:21:48 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires libusbx >= 1.0.19 10:21:48 ERROR: ISO build failed.* [1] - http://jenkins.ovirt.org/job/ovirt-node_ovirt-3.6_create-iso-el7_merged/171… Best Regards, Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1

2 1

Vdsm tests are 4X times faster on travis
by Nir Soffer 07 Dec '16

07 Dec '16

HI all, Watching vdsm travis builds in the last weeks, it is clear that vdsm tests on travis are about 4X times faster compared with jenkins builds. Here is a typical build: ovirt ci: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/5101/conso… travis ci: https://travis-ci.org/nirs/vdsm/builds/179056079 The build took 4:34 on travis, and 19:34 on ovirt ci. This has a huge impact on vdsm maintainers. Having to wait 20 minutes for each patch means that we must ignore the ci and merge and hope that previous tests without rebasing on master were good enough. The builds are mostly the same, expect: - In travis we don't check if the build system was changed and packages should be built takes 9:18 minutes in ovirt ci. - In travis we don't clean or install anything before the test, we use a container with all the available packages, pulled from dockerhub. takes about 3:52 minutes in ovirt ci - In travis we don't enable coverage. Running the tests with coverage may slow down the tests takes 5:04 minutes in ovirt ci creating the coverage report takes only 15 seconds, not interesting - In travis we don't cleanup anything after the test this takes 34 seconds in ovirt ci The biggest problem is the build system check taking 9:18 minutes. fixing it will cut the build time in half. This is how time is spent in ovirt ci: 1. Starting (1:28) 00:00:00.001 Triggered by Gerrit: https://gerrit.ovirt.org/67338 00:00:00.039 [EnvInject] - Loading node environment variables. 00:00:00.056 Building remotely on fc24-vm15.phx.ovirt.org (phx nested local_disk fc24) in workspace /home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64 2. Installing packages (2:24) 00:01:28.338 INFO: installing package(s): autoconf automake gdb git libguestfs-tools-c libselinux-python3 libvirt-python3 m2crypto make mom openvswitch policycoreutils-python PyYAML python-blivet python-coverage python2-decorator python-devel python-inotify python-ioprocess python-mock python-netaddr python-pthreading python-setuptools python-six python-requests python3-decorator python3-netaddr python3-nose python3-six python3-yaml rpm-build sanlock-python sudo yum yum-utils 3. Setup in check-patch.sh (00:04) 00:03:52.838 + export VDSM_AUTOMATION=1 4. Running the actual tests 00:03:56.670 + ./autogen.sh --system --enable-hooks --enable-vhostmd 5. Installing python-debugifo (00:19) 00:04:15.385 Yum-utils package has been deprecated, use dnf instead. 6. Running make check (05:04) 00:04:30.948 + TIMEOUT=600 00:04:30.948 + make check NOSE_WITH_COVERAGE=1 NOSE_COVER_PACKAGE=/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/vdsm,/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib ... 00:09:33.981 tests: commands succeeded 00:09:33.981 congratulations :) 7. Creating coverage report (00:15) 00:09:34.017 + coverage html -d /home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/exported-artifacts/htmlcov 8. Finding modified files (09:18) 00:09:49.213 + git diff-tree --no-commit-id --name-only -r HEAD 00:09:49.213 + egrep --quiet 'vdsm.spec.in|Makefile.am' 9. Cleaing up (00:27) 00:19:07.994 Took 915 seconds 00:19:34.973 Finished: SUCCESS

3 13

VDSM review & development tools
by Martin Polednik 07 Dec '16

07 Dec '16

Hello developers, this e-mail is mostly aimed at VDSM contributors (current and potential) and should serve as continuation of the idea started at our weekly meeting. We've had a proposal of moving the project to github[1] and using reviewable[2] as a code review due to github's code review interface. I would like to start a *discussion* regarding how we would like to develop VDSM in future. So far, the suggested options were (1st implied): 1) gerrit (stay as we are), 2) github & reviewable, 3) mailing list. What would be your favorite review & development tool and why? Do you hate any of them? Let the flamewar^Wconstructive discussion begin! :) <subjective part> My preferred tool is mailing list with the main tree mirrored to github. Why mailing list in 2016 (almost 2017)? a) stack - We're built on top of libvirt, libvirt is built on top of qemu and qemu utilizes kvm which is a kernel module. Each of this projects uses mailing lists for development. b) tooling - Everyone is free to use tools of choice. Any sane e-mail client can handle mailing list patches, and it's up to reviewers to choose the best way to handle the review. As for sending the patches, there is the universal fallback in the form of git-send-email. c) freedom - It's up to us to decide how would we handle such development. As many system-level projects already use mailing lists (see a), there is enough inspiration for workflow design[3]. d) accessibility - VDSM is in unfortunate position between "cool", high level projects and "boring" low level projects. I believe that we should be more accessible to the developers from below the stack rather than to general public. Having unified workflow that doesn't require additional accounts and is compatible with their workflows makes that easier. </subjective part> [1] https://github.com/ [2] https://reviewable.io/ [3] e.g. https://www.kernel.org/doc/Documentation/SubmittingPatches Regards, mpolednik

3 2

ovirt_4.0_he-system-tests job failed
by Shlomo Ben David 07 Dec '16

07 Dec '16

Hi, The [1] job failed with the following error: 2016-12-07 02:40:43,805::ssh.py::ssh::96::lago.ssh::DEBUG::Command 8b25fff2 on lago-he-basic-suite-4-0-host0 errors: Error: Package: ovirt-hosted-engine-setup-2.0.4.2-0.0.master.20161201083322.gita375924.el7.centos.noarch (alocalsync) Requires: ovirt-engine-sdk-python >= 3.6.3.0 [1] - http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/584/console Best Regards, Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1

3 2

test-repo_ovirt_experimental_4.0
by Shlomo Ben David 07 Dec '16

07 Dec '16

Hi All, The [1] job is failing with the following error: *+ yum install --nogpgcheck -y --downloaddir=/dev/shm ovirt-engine ovirt-log-collector 'ovirt-engine-extension-aaa-ldap*' Error: Package: ovirt-engine-cli-3.6.9.3-0.1.20161202.git61c5da9.el7.centos.noarch (alocalsync) Requires: ovirt-engine-sdk-python >= 3.6.0.0 [0m [36m * [Thread-4] Deploy VM lago-basic-suite-4-0-engine: [31mERROR [0m (in 0:01:51) [0m [31mError while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host host.name <http://host.name>(), RuntimeError: /home/jenkins/workspace/test-repo_ovirt_experimental_4.0/ovirt-system-tests/deployment-basic-suite-4.0/default/scripts/_home_jenkins_workspace_test-repo_ovirt_experimental_4.0_ovirt-system-tests_basic-suite-4.0_.._common_deploy-scripts_setup_engine.sh failed with status 1 on lago-basic-suite-4-0-engine [0m [36m # Deploy environment: [31mERROR [0m (in 0:01:51) [0m [36m@ Deploy oVirt environment: [31mERROR [0m (in 0:01:51) [0m [31mError occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 264, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 187, in do_deploy prefix.deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 67, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 198, in deploy return super(OvirtPrefix, self).deploy() File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1249, in deploy self._deploy_host, self.virt_env.get_vms().values() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 97, in invoke_in_parallel vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host host.name <http://host.name>(), RuntimeError: /home/jenkins/workspace/test-repo_ovirt_experimental_4.0/ovirt-system-tests/deployment-basic-suite-4.0/default/scripts/_home_jenkins_workspace_test-repo_ovirt_experimental_4.0_ovirt-system-tests_basic-suite-4.0_.._common_deploy-scripts_setup_engine.sh failed with status 1 on lago-basic-suite-4-0-engine [0m * [1] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/3416/console Could you please check this issue? Thanks in advanced, Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1

3 6

Re: [ovirt-devel] ovirt-appliance (today) is 1.5G...
by Ryan Barry 06 Dec '16

06 Dec '16

We may also want to do this with Node/RHV-H eventually (since the updates there are already 500MB). I don't think we currently ship deltarpm on RHV-H, but it's a very small addition. The question is whether makedeltarpm can actually do anything meaningful with something which is already a sparsified image wrapped in an RPM, but I'll test... On Sun, Oct 9, 2016 at 6:35 AM, Doron Fediuck <dfediuck(a)redhat.com> wrote: > FYI > > ---------- Forwarded message ---------- > From: Yaniv Kaul <ykaul(a)redhat.com> > Date: Thu, Oct 6, 2016 at 11:13 AM > Subject: [ovirt-devel] ovirt-appliance (today) is 1.5G... > To: devel <devel(a)ovirt.org> > > > I'm sure it was less just days ago, but now[1] it is again quite big. > 4.0's is 'only' 1G in size[2]. > Which anyway raises the question - should we provide delta RPMs for it? > > Thanks, > Y. > > [1] http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm > /el7/noarch/ovirt-engine-appliance-4.1-20160927.1.el7.centos.noarch.rpm > [2] http://resources.ovirt.org/pub/ovirt-4.0-snapshot/rpm/el > 7/noarch/ovirt-engine-appliance-4.0-20161003.1.el7.centos.noarch.rpm > > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel > >

2 2

master/4.1 now requires el7.3 for 4.1 Cluster Level
by Michal Skrivanek 06 Dec '16

06 Dec '16

Dear developers, patch requiring 7.3-based qemu-kvm machine type[1] has been merged to master For using the 4.1 Cluster Level please make sure you have qemu-kvm-rhev 2.6 and libvirt 2.0+ on RHEL 7.3, CentOS 7.3 prerelase repo ([2], I suppose[3] too, and [4]), or Fedora 24+virt-preview repo If you are unable to update your hosts please do use 4.0 compatibility clusters Also note that previous 4.1 or master deployments, including oVirt 4.1 beta from last week, do not properly update the machine type in 4.1 clusters, so please update that manually in the db Thanks, michal [1] https://gerrit.ovirt.org/#/c/65979/ [2] http://buildlogs.centos.org/c7.1611.00/ [3] http://buildlogs.centos.org/c7.1611.01/ [4] https://cbs.centos.org/koji/buildinfo?buildID=13884

2 3

planned jenkins restart
by Evgheni Dereveanchin 05 Dec '16

05 Dec '16

Hi everyone, I am going to restart Jenkins in order to update several plugins. No new builds will be scheduled during this maintenance period. I will announce you once it is finished. Regards, Evgheni Dereveanchin

1 1

master is broken - can't add host - due to vdsm/gluster/__init__.pyc conflicts
by Yaniv Kaul 05 Dec '16

05 Dec '16

Getting: file /usr/lib/python2.7/site-packages/vdsm/gluster/__init__.pyc conflicts between attempted installs of vdsm-python-4.18.999-1059.git63f9e2f.el7.centos.noarch and vdsm-4.18.999-1059.git63f9e2f.el7.centos.x86_64 Thanks, Y.

5 7

[Call to Action] packaging for your preferred distribution
by Sandro Bonazzola 05 Dec '16

05 Dec '16

Hi, we released oVirt 4.1 beta last week providing RPMs for latest Red Hat Enterprise Linux release and its derivatives like CentOS Linux and Scientific Linux and Fedora or similar distributions. Virt support for Fedora is provided as tech preview / best effort being Fedora so fast moving which is too difficult to stay aligned with. We have still 2 months to GA so if you'd like to get oVirt 4.1 on your preferred distribution and you've packaging skills, this is a good chance to start packaging it playing with 4.1 beta released code. You can find some information about porting oVirt to your preferred distribution starting from [1] [1] https://www.ovirt.org/develop/developer-guide/porting-ovirt/ -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0

[Call to Action] oVirt 4.1 translation
by Sandro Bonazzola 05 Dec '16

05 Dec '16

Hi, we released 4.1 beta 1 last week and we're going fast toward the GA currently scheduled for January 24th[1]. We have a scheduled string freeze on December 21st to allow translators to complete the translations but looking at zanata[2] I think it's time to go there and start translating to your preferred language. [1] http://www.ovirt.org/develop/release-management/releases/4.1/release-manage… [2] https://translate.zanata.org/project/view/ovirt?dswid=9807 -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0

Re: [ovirt-devel] Experimental Flow for Master Fails to Run a VM
by Yaniv Kaul 04 Dec '16

04 Dec '16

On Dec 4, 2016 10:57 PM, "Eyal Edri" <eedri(a)redhat.com> wrote: tests are back to stable, but with a cost of not testing 4.1 CL. DC level. But we are not explicitly using any of the 4.1 features for the time being. (I'd like to believe implictly we did, qcow2v3 for example). Iets hope we get centos 7.3 soon. Indeed - but we also need virt-builder image for it. Y. On Dec 4, 2016 22:41, "Yaniv Kaul" <ykaul(a)redhat.com> wrote: > > > On Dec 4, 2016 6:42 PM, "Arik Hadas" <ahadas(a)redhat.com> wrote: > > Yaniv will try to lower the cluster level used in the system-tests to 4.0 > - this is supposed to solve the issue. > > > Done. > Y. > > If it won't help (we will know it in about an hour), we'll add a db-script > that changes the rng device of the blank template only. > > On Sun, Dec 4, 2016 at 3:34 PM, Eyal Edri <eedri(a)redhat.com> wrote: > >> FYI, >> >> I opened a bug [1] to track this issue since I don't see any attempts to >> resolve the issue on the thread, hopefully a bug will get more attention. >> Opened on VDSM since we see the libvirt error there, feel free to move >> product/team. >> >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1401303 >> >> On Sun, Dec 4, 2016 at 1:23 PM, Eyal Edri <eedri(a)redhat.com> wrote: >> >>> Not sure if relevant, but Juan posted a fix for SDK4 last time it >>> happened ( but different failure on log-collector ): >>> >>> https://gerrit.ovirt.org/#/c/67213/ >>> >>> * Added `urandom` to the `RngSource` enumerated type. >>> >>> On Sun, Dec 4, 2016 at 9:17 AM, Eyal Edri <eedri(a)redhat.com> wrote: >>> >>>> And its still failing from Friday, >>>> Since we don't have official Centos 7.3 repos yet ( hopefully we'll >>>> have it this week, but as of this moment its not published yet ) , we have >>>> to either revert the offending patch >>>> or send a quick fix. >>>> >>>> Right now all experimental flows for master are not working and nightly >>>> rpms are not refreshed with new RPMs. >>>> >>>> >>>> >>>> On Fri, Dec 2, 2016 at 9:41 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote: >>>> >>>>> >>>>> >>>>> On Dec 2, 2016 2:11 PM, "Anton Marchukov" <amarchuk(a)redhat.com> wrote: >>>>> >>>>> Hello Martin. >>>>> >>>>> Do by outdated you mean the old libvirt? If so that is that livirt >>>>> available in CentOS 7.2? There is no 7.3 yet. >>>>> >>>>> >>>>> Right, this is the issue. >>>>> Y. >>>>> >>>>> >>>>> Anton. >>>>> >>>>> On Fri, Dec 2, 2016 at 1:07 PM, Martin Polednik <mpolednik(a)redhat.com> >>>>> wrote: >>>>> >>>>>> On 02/12/16 10:55 +0100, Anton Marchukov wrote: >>>>>> >>>>>>> Hello All. >>>>>>> >>>>>>> Engine log can be viewed here: >>>>>>> >>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma >>>>>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh- >>>>>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004 >>>>>>> _basic_sanity.py/lago-basic-suite-master-engine/_var_log_ovi >>>>>>> rt-engine/engine.log >>>>>>> >>>>>>> I see the following exception there: >>>>>>> >>>>>>> 2016-12-02 04:29:24,030-05 DEBUG >>>>>>> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] >>>>>>> (ResponseWorker) [83b6b5d] Message received: {"jsonrpc": "2.0", "id": >>>>>>> "ec254aad-441b-47e7-a644-aebddcc1d62c", "result": true} >>>>>>> 2016-12-02 04:29:24,030-05 ERROR >>>>>>> [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) >>>>>>> [83b6b5d] Not able to update response for >>>>>>> "ec254aad-441b-47e7-a644-aebddcc1d62c" >>>>>>> 2016-12-02 04:29:24,041-05 DEBUG >>>>>>> [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] >>>>>>> (DefaultQuartzScheduler3) [47a31d72] Rescheduling >>>>>>> DEFAULT.org.ovirt.engine.core.bll.gluster.GlusterSyncJob.ref >>>>>>> reshLightWeightData#-9223372036854775775 >>>>>>> as there is no unfired trigger. >>>>>>> 2016-12-02 04:29:24,024-05 DEBUG >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default >>>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Exception: >>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: >>>>>>> VDSGenericException: VDSNetworkException: Timeout during xml-rpc call >>>>>>> at org.ovirt.engine.core.vdsbroke >>>>>>> r.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) >>>>>>> [vdsbroker.jar:] >>>>>>> >>>>>>> ... >>>>>>> >>>>>>> 2016-12-02 04:29:24,042-05 ERROR >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default >>>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Timeout waiting for >>>>>>> VDSM response: Internal timeout occured >>>>>>> 2016-12-02 04:29:24,044-05 DEBUG >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD >>>>>>> SCommand] >>>>>>> (default task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] START, >>>>>>> GetCapabilitiesVDSCommand(HostName = lago-basic-suite-master-host0, >>>>>>> VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', >>>>>>> hostId='5eb7019e-28a3-4f93-9188-685b6c64a2f5', >>>>>>> vds='Host[lago-basic-suite-master-host0,5eb7019e-28a3-4f93-9 >>>>>>> 188-685b6c64a2f5]'}), >>>>>>> log id: 58f448b8 >>>>>>> 2016-12-02 04:29:24,044-05 DEBUG >>>>>>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default >>>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] SEND >>>>>>> destination:jms.topic.vdsm_requests >>>>>>> reply-to:jms.topic.vdsm_responses >>>>>>> content-length:105 >>>>>>> >>>>>>> >>>>>>> Please note that this runs on localhost with local bridge. So it is >>>>>>> not >>>>>>> likely to be network itself. >>>>>>> >>>>>> >>>>>> The main issue I see is that the VM run command has actually failed >>>>>> due to libvirt no accepting /dev/urandom as RNG source[1]. This was >>>>>> done as engine patch and according to git log, posted around Mon Nov >>>>>> 28. Also adding Jakub - this should either not happen from engine's >>>>>> point of view or the lago host is outdated. >>>>>> >>>>>> [1] >>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma >>>>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh- >>>>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004 >>>>>> _basic_sanity.py/lago-basic-suite-master-host0/_var_log_vdsm/vdsm.log >>>>>> >>>>>> >>>>>> Anton. >>>>>>> >>>>>>> On Fri, Dec 2, 2016 at 10:43 AM, Anton Marchukov < >>>>>>> amarchuk(a)redhat.com> >>>>>>> wrote: >>>>>>> >>>>>>> FYI. Experimental flow for master currently fails to run a VM. The >>>>>>>> tests >>>>>>>> times out while waiting for 180 seconds: >>>>>>>> >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ >>>>>>>> master/3838/testReport/(root)/004_basic_sanity/vm_run/ >>>>>>>> >>>>>>>> This is reproducible over 23 runs of this happened tonight, sounds >>>>>>>> like a >>>>>>>> regression to me: >>>>>>>> >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/ >>>>>>>> >>>>>>>> I will update here with additional information once I find it. >>>>>>>> >>>>>>>> Last successful run was with this patch: >>>>>>>> >>>>>>>> https://gerrit.ovirt.org/#/c/66416/ (vdsm: API: move vm parameters >>>>>>>> fixup >>>>>>>> in a method) >>>>>>>> >>>>>>>> Known to start failing around this patch: >>>>>>>> >>>>>>>> https://gerrit.ovirt.org/#/c/67647/ (vdsmapi: fix a typo in string >>>>>>>> formatting) >>>>>>>> >>>>>>>> Please notes that we do not have gating implemented yet, so >>>>>>>> everything >>>>>>>> that was merged in between those patches might have caused this (not >>>>>>>> necessary in vdsm project). >>>>>>>> >>>>>>>> Anton. >>>>>>>> -- >>>>>>>> Anton Marchukov >>>>>>>> Senior Software Engineer - RHEV CI - Red Hat >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Anton Marchukov >>>>>>> Senior Software Engineer - RHEV CI - Red Hat >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>>> Devel mailing list >>>>>>> Devel(a)ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Anton Marchukov >>>>> Senior Software Engineer - RHEV CI - Red Hat >>>>> >>>>> >>>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>> >>>> >>>> >>>> -- >>>> Eyal Edri >>>> Associate Manager >>>> RHV DevOps >>>> EMEA ENG Virtualization R&D >>>> Red Hat Israel >>>> >>>> phone: +972-9-7692018 >>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> >>> >>> >>> >>> -- >>> Eyal Edri >>> Associate Manager >>> RHV DevOps >>> EMEA ENG Virtualization R&D >>> Red Hat Israel >>> >>> phone: +972-9-7692018 >>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>> >> >> >> >> -- >> Eyal Edri >> Associate Manager >> RHV DevOps >> EMEA ENG Virtualization R&D >> Red Hat Israel >> >> phone: +972-9-7692018 >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > > >

1 0

Experimental Flow for Master Fails to Run a VM
by Anton Marchukov 04 Dec '16

04 Dec '16

FYI. Experimental flow for master currently fails to run a VM. The tests times out while waiting for 180 seconds: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3838/testR… This is reproducible over 23 runs of this happened tonight, sounds like a regression to me: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/ I will update here with additional information once I find it. Last successful run was with this patch: https://gerrit.ovirt.org/#/c/66416/ (vdsm: API: move vm parameters fixup in a method) Known to start failing around this patch: https://gerrit.ovirt.org/#/c/67647/ (vdsmapi: fix a typo in string formatting) Please notes that we do not have gating implemented yet, so everything that was merged in between those patches might have caused this (not necessary in vdsm project). Anton. -- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat

6 13

test-repo_ovirt_experimental_master job fails
by Shlomo Ben David 04 Dec '16

04 Dec '16

Hi All, The [1] job is failing since yesterday (Dec 3, 2016 6:29 PM) with the following error: *15:35:52* [36m@ Run test: 004_basic_sanity.py: [31mERROR [0m (in 0:04:44) [0m*15:35:52* [31mError occured, aborting*15:35:52* Traceback (most recent call last):*15:35:52* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 264, in do_run*15:35:52* self.cli_plugins[args.ovirtverb].do_run(args)*15:35:52* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run*15:35:52* self._do_run(**vars(args))*15:35:52* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in wrapper*15:35:52* return func(*args, **kwargs)*15:35:52* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in wrapper*15:35:52* return func(*args, prefix=prefix, **kwargs)*15:35:52* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 105, in do_ovirt_runtest*15:35:52* raise RuntimeError('Some tests failed')*15:35:52* RuntimeError: Some tests failed [0m >From the test results: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 138, in wrapped_test return test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 339, in vm_run lambda: api.vms.get(VM0_NAME).status.state == 'up', File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 231, in assert_true_within_short allowed_exceptions=allowed_exceptions, File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 223, in assert_true_within raise AssertionError('Timed out after %s seconds' % timeout) AssertionError: Timed out after 180 seconds Could you guide me please who should handle this issue? [1] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/ Best Regards, Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1

1 0

VDSM changes Linux memory dirty ratios - why?
by Yaniv Kaul 04 Dec '16

04 Dec '16

It appears that VDSM changes the following params: vm.dirty_ratio = 5 vm.dirty_background_ratio = 2 Any idea why? Because we use cache=none it's irrelevant anyway? TIA, Y.

4 9

ovirt-guest-agent job not building rpms
by Sandro Bonazzola 04 Dec '16

04 Dec '16

Hi, please have a look at http://jenkins.ovirt.org/job/ovirt-guest-agent_master_build-artifacts-el7-x… ovirt-guest-agent job not building rpms Thanks, -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

4 6

Request for oVirt Ansible modules testing feedback
by Ondra Machacek 02 Dec '16

02 Dec '16

Hello all, I would like to kindly ask everyone who is Ansible or oVirt user for testing of the new Ansible oVirt modules. For everyone who is familiar with the Ansible and oVirt, this[1] describes the steps you need to do, to setup oVirt modules library and start using those modules (Most of those modules will be available in Ansible 2.3, some of them are already in 2.2). If you have any issue setting this up, please contact me, I will do the best to help you. If you have an issue, which you think is a bug, please open an issue here[2]. Please note that Ansible is merging it's repositories, so since next week it will actually be stored here[3]. If you are missing anything please open an issue as well, or just contact me, and I will do fix it. You are also very welcome to sent PR with fixes. For those who don't have testing environment which can test against, I've created an Vagrant project which will deploy you the oVirt instance using Ansible playbooks. You can find how to use it here[4]. The repository also contains few examples[5], so you don't have to copy-paste them from the source. Thanks all for reading this and any feedback, Ondra [1] https://github.com/machacekondra/ovirt-tests/releases/tag/0.1 [2] https://github.com/ansible/ansible-modules-extras/issues [3] https://github.com/ansible/ansible [4] https://github.com/machacekondra/ovirt-tests [5] https://github.com/machacekondra/ovirt-tests/tree/master/examples

2 2

oVirt Community Newsletter: November 2016
by Brian Proffitt 02 Dec '16

02 Dec '16

oVirt's development is continuing on pace, as the calendar year draws to a close and we get ready for a new year of development, evangelism, and making virtual machine management a simple process for everyone. Here's what happened in November of 2016: ----------------- Software Releases ----------------- oVirt 4.0.6 Third Release Candidate is now available http://bit.ly/2gOxaDm oVirt 4.1.0 First Beta Release is now available for testing http://bit.ly/2gOtRfa ---------------- In the Community ---------------- Testing ovirt-engine changes without a real cluster http://www.ovirt.org/blog/2016/11/testing-ovirt-changes-without-cluster/ Request for oVirt Ansible modules testing feedback http://bit.ly/2gOByCg ------------------------------------ Deep Dives and Technical Discussions ------------------------------------ Important Open Source Cloud Products [German] http://bit.ly/2gOvWbd Red Hat IT runs OpenShift Container Platform on Red Hat Virtualization and Ansible http://red.ht/2ekHcLV Keynote: Blurring the Lines: The Continuum Between Containers and VMs [Video] http://bit.ly/2gOzutT Quick Guide: How to Plan Your Red Hat Virtualization 4.0 Deployment http://red.ht/2emiQkN A Decade of KVM [Chinese] http://bit.ly/2gOAuyp Expansion of iptables Rules for oVirt 4.0 [Russian] http://bit.ly/2gOBaUf -- Brian Proffitt Principal Community Analyst Open Source and Standards @TheTechScribe 574.383.9BKP

1 0

python path conflicts issues while running python scripts from vdsm verbs
by Ramesh Nachimuthu 02 Dec '16

02 Dec '16

Hi, I am trying to run a python script '/sbin/gluster-eventsapi' in vdsm verb which internally imports some python modules from /usr/lib/python2.7/site-packages/gluster/cliutils/. But it fails with the import error. Following error is seen in the supervdsm log. MainProcess|Thread-1::DEBUG::2016-11-28 16:54:35,130::commands::93::root::(execCmd) FAILED: <err> = 'Traceback (most recent call last):\n File "/sbin/gluster-eventsapi", line 25, in <module>\n from gluster.cliutils.cliutils import (Cmd, execute, node_output_ok, node_output_notok,\nImportError: No module named cliutils.cliutils\n'; <rc> = 1 I think the import statement "from gluster.cliutils.cliutils import (Cmd, execute, node_output_ok, node_output_notok)" in the python script resolves to '/usr/share/vdsm/gluster' instead of /usr/lib/python2.7/site-packages/gluster/cliutils. I see the following in python sys.path while executing a python script from vdsm. ['/usr/libexec/glusterfs', '/usr/share/vdsm', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages'] Looks like '/usr/share/vdsm' take precedence over '/usr/lib64/python2.7/site-packages'. Can someone suggests a way to fix this issue? Note: '/sbin/gluster-eventsapi' works perfectly while running directly from CLI. Related vdsm patch: https://gerrit.ovirt.org/#/c/67168/2/vdsm/gluster/events.py Regards, Ramesh

4 11

Deprecating Defaults for to_all_distros Setting in repoman
by Anton Marchukov 01 Dec '16

01 Dec '16

Hello All. If you use repoman please be aware that it currently has hardcoded list of packages that defaults to being distroless. This is controlled by "to_all_distros" configuration setting, but if you do not set it than the defaults are used. For ovirt experimental we decided that we better have all packages to have distro specified in spec. This also inline with most distro packaging guidelines. But then with current defaults repoman will consider those package to be distroless even when they are not and specifying an empty list in configuration is a silly way to override this. This change provides an incompatibility, but you can still use this feature by specifying to_all_distros package list explicitly in the config. So it is just the change of default values at this point: https://gerrit.ovirt.org/#/c/67662/ If somebody is not ok with defaults change and need time to update configuration please let me know. Also if you are not sure we can go another way and have intermediate repoman version that does not do the change, but issue deprecation warning if defaults are being used, so everybody can test their setups. However I think that it is might not be actually needed in this case as it is a configuration thing. Please feel free to comment on change request. Anton. -- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat

2 2

Gerrit parallel patch handling and CI (Or, why did my code fail post-merge)
by Barak Korren 01 Dec '16

01 Dec '16

Hi there, I would like to address a concernt that had been raised to us by multiple developers, and reach an agreement on how (and if) to remedy it. Lets assume the following situation: We have a Git repo in Gerrit with top commit C0 in master. On time t0 developers Alice and Bob push patches P1 and P2 respectively to master so that we end up with the following situation in git: C0 <= P1 (this is Alice`s patch) C0 <= P2 (this is Bob`s patch) On time t1 CI runs for both patches checking the code as it looks for each patch. Lets assume CI is successful for both. On time t2 Alice submits her patch and Gerrit merges it, resulting in the following situation in master: C0 <= P1 On time t2 Bob submits his patch. Gerrit, seeing master has changed, re-bases the patch and merges it, the resulting situation (If the rebase is successful) is: C0 <= P1 <= P2 This means that the resulting code was never tested in CI. This, in turn, causes various failures to show up post-merge despite having pre-merge CI run successfully. This situation is a result of the way our repos are currently configured. Most repos ATM are configured with the "Rebase If Necessary" submit type. This means that Gerrit tries to automatically rebase patches as mentioned in t2 above. We could, instead, configure the repos to use the "Fast Forward Only" submit type. In that case, when Bob submits on t2, Gerrit refuses to merge and asks Bob to rebase (While offering a convenient button to do it). When he does, a new patch set gets pushed, and subsequently checked by CI. I recommend we switch all projects to use the "Fast Forward Only" submit type. Thoughts? Concerns? -- Barak Korren bkorren(a)redhat.com RHEV-CI Team

7 17

performance improvements and gwt-rpc switch
by Greg Sheremeta 01 Dec '16

01 Dec '16

Hi everyone, For the upcoming oVirt 4.1, the UX team has been focused hard on webadmin performance improvements. There have been some reports [1] [2] of UI sluggishness in both 3.6 and 4.0, usually after the browser had been open some time, and usually in scale environments. After some research, we determined that the primary cause of this sluggishness was memory leaks. We embarked on several weeks of hunting down memory leak bugs and squashing them. Alexander Wels and Vojtech Szocs led this work, and I helped test the performance of each patch as they created them. As they created patches to squash leaks, performance kept getting better and better. Today we've merged the last of our patches [*], and I'm happy to announce that we are now seeing much better UI performance on 4.1-master and 4.0.6. Over the course of several hours with the browser window open, users should see no sluggishness at all. [*] This last patch switches oVirt from using de-rpc to gwt-rpc in the frontend. This improves performance, but it also allows us to upgrade to GWT 2.8. We'd been previously blocked on that. If you're interested in UI performance testing, continue reading. If not, you can stop here :) ..... To verify our performance improvements, we took some simple measurements using selenium webdriver. The tests were unscientific, but very helpful. We ran a webdriver flow through oVirt that clicked some buttons and tabs and refreshed some grids. We did it a few hundred or thousand times. The tests were run using stubbed hosts (ovirt-vdsmfake) so that only the engine and UI were under test. Below are the important takeaways. The x axis is time, and each point on a graph is a loop through the same webdriver flow. The (ms) y axes are response times, and memory is in MB. In this graph, we compare oVirt 4.1 with and without our most impactful patch applied. As you can see, with the patch applied, response time stays flat for 200 loops of my test script over the course of 18 and 43 minutes. Without the patch applied, response time quickly degraded such that 200 loops of my test script took 1 hr 2 minutes vs. 18 minutes with the patch applied -- a 66% improvement! [image: Inline image 1] In this graph [ignore the spike], we tested oVirt hard for 6 hours 25 minutes (2000 loops). As you can see, the response times stay relatively flat over 6 hours! This is a great improvement. Do note that the memory is still growing, albeit much more slowly now. You can see towards the end of this run, maybe around hour 5, that the deviation starts to go up (the line thickens). Takeaway: maybe refresh your browser after many hours of having webadmin open. But, this is a stress test -- I'm betting users won't notice this slowdown after even 6 hours of regular webadmin use or idling. Last, here is a graph that shows gwt-rpc performing slightly better than de-rpc. Memory consumption is about the same -- gwt-rpc is just a faster rpc implementation. Reply with any questions or concerns. Thanks! Best wishes, Greg [1] https://bugzilla.redhat.com/show_bug.cgi?id=1368101 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1388462 -- Greg Sheremeta, MBA Red Hat, Inc. Sr. Software Engineer gshereme(a)redhat.com

6 8