On Tue, Aug 15, 2017 at 4:24 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Tue, Aug 15, 2017 at 5:28 AM, Marc Young <3vilpenguin@gmail.com> wrote:After updating the python sdk:myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ $ rpm -q python-ovirt-engine-sdk4python-ovirt-engine-sdk4-4.1.6-2.20170712git1b99f36.el7.cent os.x86_64 I get more but different errors[1].Then the lago log[2]All engine Logs are even further[3]OK, understood the issue. Your host installation is for some reason very very slow - the test just times out after 15 minutes. It's why the Engine log is a bit cut.It begins here:2017-08-14 21:38:32,143-04 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient] (default task-10) [10728bdc-0cdc-45ba-90f8- 7a78b0b79d6e] Connecting 'root@lago-basic-suite-4-1- host0' And it still installing here:2017-08-14 21:55:01,438-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling. AuditLogDirector] (VdsDeploy) [76cb1954] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Correlation ID: 76cb1954, Call Stack: null, Custom Event ID: -1, Message: Installing Host lago-basic-suite-4-1-host0. Yum install: 239/364: python-netaddr-0.7.5-7.el7. noarch. But ovirt-system-tests already given up. On my laptop it takes ~3 minutes - perhaps you are running on qemu and not kvm, or something similar? It shouldn't be that slow.Y.[1]# add_cluster: Success (in 0:00:03)# add_hosts:* Collect artifacts:- [Thread-5] lago-basic-suite-4-1-host1: ERROR (in 0:00:19)- [Thread-4] lago-basic-suite-4-1-engine: ERROR (in 0:00:20)Error while running threadTraceback (most recent call last):File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()})File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath)File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopathFile "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm. py", line 342, in extract_paths ignore_nopath=ignore_nopath,File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=FalseFile "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path,File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all()File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024)File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout()timeoutError while running threadTraceback (most recent call last):File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()})File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath)File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopathFile "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm. py", line 342, in extract_paths ignore_nopath=ignore_nopath,File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=FalseFile "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path,File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all()File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024)File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout()timeoutError while running threadTraceback (most recent call last):File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()})File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath)File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopathFile "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm. py", line 342, in extract_paths ignore_nopath=ignore_nopath,File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=FalseFile "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path,File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all()File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024)File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout()timeoutError while running threadTraceback (most recent call last):File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()})File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath)File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopathFile "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm. py", line 342, in extract_paths ignore_nopath=ignore_nopath,File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=FalseFile "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path,File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all()File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024)File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout()timeout* Collect artifacts: ERROR (in 0:00:33)[2]# add_hosts: ERROR (in 0:15:52)# Collect artifacts:* [Thread-19] lago-basic-suite-4-1-engine:* [Thread-20] lago-basic-suite-4-1-host1:* [Thread-21] lago-basic-suite-4-1-host0:* [Thread-21] lago-basic-suite-4-1-host0: Success (in 0:00:03)* [Thread-20] lago-basic-suite-4-1-host1: Success (in 0:00:05)* [Thread-19] lago-basic-suite-4-1-engine: Success (in 0:00:16)# Collect artifacts: Success (in 0:00:16)# Results located at /home/myoung/repos/github/ovirt-system-tests/deployment-basi c-suite-4.1/default/002_bootst rap.py.junit.xml @ Run test: 002_bootstrap.py: ERROR (in 0:16:57)Error occured, abortingTraceback (most recent call last):File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args))File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs)File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs)File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed')RuntimeError: Some tests failed+ res=1+ cd -/home/myoung/repos/github/ovirt-system-tests + return 1+ failed=true+ env_collect /home/myoung/repos/github/ovirt-system-tests/test_logs/basic -suite-4.1/post-002_bootstrap. py + local tests_out_dir=/home/myoung/repos/github/ovirt-system-tests/ test_logs/basic-suite-4.1/ post-002_bootstrap.py + echo '#########################'#########################+ [[ -e /home/myoung/repos/github/ovirt-system-tests/test_logs/basic -suite-4.1 ]] + cd /home/myoung/repos/github/ovirt-system-tests/deployment-basi c-suite-4.1/current + lago collect --output /home/myoung/repos/github/ovirt-system-tests/test_logs/basic -suite-4.1/post-002_bootstrap. py @ Collect artifacts:# [Thread-1] lago-basic-suite-4-1-engine:# [Thread-2] lago-basic-suite-4-1-host1:# [Thread-3] lago-basic-suite-4-1-host0:# [Thread-3] lago-basic-suite-4-1-host0: Success (in 0:00:02)# [Thread-2] lago-basic-suite-4-1-host1: Success (in 0:00:03)# [Thread-1] lago-basic-suite-4-1-engine: Success (in 0:00:06)@ Collect artifacts: Success (in 0:00:07)+ cp -a logs /home/myoung/repos/github/ovirt-system-tests/test_logs/basic -suite-4.1/post-002_bootstrap. py/lago_logs + cd -/home/myoung/repos/github/ovirt-system-tests + true+ echo '@@@@ ERROR: Failed running /home/myoung/repos/github/ovirt-system-tests/basic-suite-4. 1/test-scenarios/002_bootstrap .py' @@@@ ERROR: Failed running /home/myoung/repos/github/ovirt-system-tests/basic-suite-4. 1/test-scenarios/002_bootstrap .py + return 1$ cat './test_logs/basic-suite-4.1/post-002_bootstrap.py.2017-08-1 4_21:55:00/lago_logs/lago.log' | fpaste Uploading (1202.8KiB)...[3]$ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log | grep ovirt-engine); do echo $i; cat $i | fpaste; echo ----; done test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt -engine/setup/ovirt-engine- setup-20170814212716-ql8ekq. log Uploading (5544.7KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt -engine/boot.log Uploading (1.5KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt -engine/server.log Uploading (33.9KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt -engine/engine.log Uploading (2697.0KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt -engine-dwh/ovirt-engine-dwhd. log Uploading (2.4KiB)...----On Mon, Aug 14, 2017 at 2:02 PM, Marc Young <3vilpenguin@gmail.com> wrote:rpm -q python-ovirt-engine-sdk4Mine is outdated:Ill update that and run again.$ rpm -q python-ovirt-engine-sdk4python-ovirt-engine-sdk4-4.1.5-2.el7.centos.x86_64 Logs (better to send too many than not enough sometimes). Ovirt engine logs are there.myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ $ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log); do echo $i; cat $i | fpaste; echo ----; done test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/tuned /tuned.log Uploading (3.5KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/audit /audit.log Uploading (401.2KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/yum.l og Uploading (24.3KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/setup -ds.log Uploading (37.4KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt -engine/setup/ovirt-engine-set up-20170814074645-vzih31.log Uploading (5544.4KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt -engine/boot.log Uploading (1.3KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt -engine/server.log Uploading (34.1KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt -engine/engine.log Uploading (2724.0KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt -engine-dwh/ovirt-engine-dwhd. log Uploading (2.4KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/snmpt rapd.log Uploading (0.1KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host1/_var_log/tuned/ tuned.log Uploading (1.5KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host1/_var_log/audit/ audit.log Uploading (183.1KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host1/_var_log/yum.lo g Uploading (18.3KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host0/_var_log/tuned/ tuned.log Uploading (1.5KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host0/_var_log/audit/ audit.log Uploading (179.9KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host0/_var_log/yum.lo g Uploading (17.4KiB)...----test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs/la go.log Uploading (1198.1KiB)...----On Mon, Aug 14, 2017 at 1:11 PM, Eyal Edri <eedri@redhat.com> wrote:On Mon, Aug 14, 2017 at 7:18 PM, Marc Young <3vilpenguin@gmail.com> wrote:You should have all logs needed @/home/myoung/repos/github/ovirt-system-tests/test_logs/ basic-suite-4.1/ post-002_bootstrap.py/lago_logs The one I pasted into fpaste was the only one in that directory, if there are more logs Im just unsure where they live.Heres all logs for basic suite 4.1 that I know how to find:myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ The engine and hosts logs are usually under the test_logs, which should be in the same subdir as lago_logs directory.Also, can you verify you have the latest python v4 SDK installled?rpm -q python-ovirt-engine-sdk4If you don't have it installed, you can get it directly from the build job ( which acts also as yum repo ):fc26 build is available only for master [1] but should also work.There are also fc24/25 builds, just follow the same pattern to find them on Jenkins.$ for i in $(find . -name 'lago_logs' | grep basic-suite-4.1); do cat $i/*.log | fpaste; doneUploading (1355.0KiB)...Uploading (1154.0KiB)...Uploading (1377.8KiB)...Uploading (1176.7KiB)...Uploading (1397.1KiB)...Uploading (1198.1KiB)...On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul@redhat.com> wrote:On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin@gmail.com> wrote:Full log is just as uninteresting: https://paste.fedoraproject.org/paste/ZaZMcT5 sg9SAGQ7OKC3-EA I was actually hoping for the engine log.Y.On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul@redhat.com> wrote:On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:Thanks for clarifying, that makes sense in hindsight since its testing things in their entirety. This is more of a rabbit hole for me as Im really just trying to learn Lago by using the OST project (since I'll be duplicating a lot of the setup).stdout logs from a run: https://pastebin.com/KBDaCCYp Can you send engine log so we'll understand why the host installation failed?You should have all logs needed @/home/myoung/repos/github/ovirt-system-tests/test_logs/basi c-suite-4.1/post-002_bootstrap .py/lago_logs Y.On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> wrote:On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:Ill try to get some detailed log files later, but fwiw I'm not running the hc suites (afaik, Im still getting inundated with the system tests/lago). The link I used for Jenkins was just to try to pull the latest 'passing' hash for ovirt-system-testsI see, its a bit more complicated than that I'm afraid, the HASH you see is probably on for OST code itself, while the suites can fail on any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than just a single hash, but rather a list of RPMs and their versions, together with hash of OST ( for the tests themselves ) and also OS updates. The errors have all been from master branch on the basic suite for 4.1 via:$ ./run_suite.sh basic-suite-4.1/OK, then we need to investigate it and understand why add host fails on 4.1, please provide logs when possible.BTW, you can also try running the manual job [1], but you'll need to provide it with custom yum repo URL with your built artifacts to test your code.There is a section on it on the OST readthedocs page.On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> wrote:I see you run the 'hc' suite, which means hyperconverged. This suite is running an hosted engine on gluster storage, It's a more complex suite than the basic one, and prone to more errors, however it should still work, but if you don't require it specifically, I would recommend running the basic suite, which should be easier to debug and also quicker to run.I'm also adding hc maintainer so she can check, can you share the link to the Jenkins job you're running? Or the log files if you're running locally.On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote:Actually I spoke too soon, still fails:+ lago ovirt runtest /home/myoung/repos/github/ovirt-system-tests/vagrant/test-sc enarios/002_bootstrap.py @ Run test: 002_bootstrap.py:nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']# print_api_ver:# print_api_ver: Success (in 0:00:00)# add_dc:# add_dc: Success (in 0:00:43)# add_cluster:# add_cluster: Success (in 0:00:03)# add_hosts:dd* Collect artifacts:* Collect artifacts: Success (in 0:01:14)# add_hosts: Success (in 0:16:36)# Results located at /home/myoung/repos/github/ovirt-system-tests/deployment-vagr ant/default/002_bootstrap.py.j unit.xml @ Run test: 002_bootstrap.py: Success (in 0:17:26)Error occured, abortingTraceback (most recent call last):File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args))File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs)File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs)File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed')RuntimeError: Some tests failedOn Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com> wrote:Edit: reply-allIt does, every time.I got it to pass by using the last passing revision from Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas ter/15/consoleFull Not sure if it's a true race condition, that build in Jenkins has failed consistently since #15On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> wrote:On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com> wrote:What's the fix for this for those of us using ovirt-system-tests?Im trying to adapt some of the code for testing a third party tool, but master is still failing on 002_bootstrapIt fails consistently?AFAIK this is a race condition that happens maybe once a week, can you share your logs? is it also failing on add_secondary_storage?On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com> wrote:I'm not sure it's related since the LSM test runs much later, in 004On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> wrote:Adding Allon & Benny.Is it possible to verify this is related to the LSM issue we've been handling in [1]?If this is the case, we agreed to disable the test next time it fails, as the current workarounds with sleep isn't enough.Can you confirm this is the case, and so we'll have to disable this test until one of the RFEs described in [1] is merged?--On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> wrote:On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote:
>
>
> On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com> wrote:
>>
>> Test failed: [ 002_bootstrap.add_secondary_storage_domains ] Because it is the only patch participating in the test.
>>
>> Link to suspected patches:
>> https://gerrit.ovirt.org/#/c/79974
>
>
> This patch adds missing log when resizing an online disk - why do you think
> it is related to the failure?
(The test was equivalent to running the manual job with just this patch)
Then again this may also one of the usual SD testing race conditions.
Since the code in the patch seemed to be related to storage I didn't
want to just assume that.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
