- Devel - oVirt List Archives

[monitoring][collectd] the collectd virt plugin is now on par with Vdsm needs
by Francesco Romani 27 Feb '17

27 Feb '17

Hello everyone, in the last weeks I've been submitting PRs to collectd upstream, to bring the virt plugin up to date with Vdsm and oVirt needs. Previously, the collectd virt plugin reported only a subset of metrics oVirt uses. In current collectd master, the collectd virt plugin provides all the data Vdsm (thus Engine) needs. This means that it is now possible for Vdsm or Engine to query collectd, not Vdsm/libvirt, and have the same data. There are only two caveats: 1. it is yet to be seen which version of collectd will ship all those enhancements 2. collectd *intentionally* report metrics as rates, not as absolute values as Vdsm does. This may be one issue in presence of restarts/data loss in the link between collectd and the metrics store. Please keep reading for more details: How to get the code? -------------------------------- This somehow tricky until we get one official release. If one is familiar with the RPM build process, it is easy to build one custom packages from a snapshot from collectd master (https://github.com/collectd/collectd) and a recent 5.7.1 RPM (like https://koji.fedoraproject.org/koji/buildinfo?buildID=835669) How to configure it? ------------------------------ Most thing work out of the box. One currently in progress Vdsm patch ships the recommended configuration https://gerrit.ovirt.org/#/c/71176/6/static/etc/collectd.d/virt.conf The meaning of the configuration option is documented in man 5 collectd.conf How it looks like? -------------------------- Let me post one "screenshot" :) $ collectdctl listval | grep a0 a0/virt/disk_octets-hdc a0/virt/disk_octets-vda a0/virt/disk_ops-hdc a0/virt/disk_ops-vda a0/virt/disk_time-hdc a0/virt/disk_time-vda a0/virt/if_dropped-vnet0 a0/virt/if_errors-vnet0 a0/virt/if_octets-vnet0 a0/virt/if_packets-vnet0 a0/virt/memory-actual_balloon a0/virt/memory-rss a0/virt/memory-total a0/virt/ps_cputime a0/virt/total_requests-flush-hdc a0/virt/total_requests-flush-vda a0/virt/total_time_in_ms-flush-hdc a0/virt/total_time_in_ms-flush-vda a0/virt/virt_cpu_total a0/virt/virt_vcpu-0 a0/virt/virt_vcpu-1 How to consume the data? ----------------------------------------- Among the ways to query collectd, the two most popular (and most fitting for oVirt use case) ways are perhaps the network protocol (https://collectd.org/wiki/index.php/Binary_protocol) and the plain text protocol (https://collectd.org/wiki/index.php/Plain_text_protocol) The first could be used by Engine to get the data directly, or to consolidate the metrics in one database (e.g to run any kind of query, for historical series...). The latter will be used by Vdsm to keep reporting the metrics (again https://gerrit.ovirt.org/#/c/71176/6) Please note that the performance of the plain text protocol are known to be lower than the binary protocol What about the unresponsive hosts? ------------------------------------------------------- We know from experience that hosts may become unresponsive, and this can disrupt monitoring. however, we do want to keep monitoring the responsive hosts, avoiding that one rogue hosts makes us lose all the monitoring data. To cope with this need, the virt plugin gained support for "partition tag". With this, we can group VMs together using one arbitrary tag. This is completely transparent to collectd, and also completely optional. oVirt can use this tag to group VMs per-storage-domain, or however it sees fit, trying to minimize the disruption should one host become unresponsive. Read the full docs here: https://github.com/collectd/collectd/commit/999efc28d8e2e96bc15f535254d412a… What about the collectd-ovirt plugin? -------------------------------------------------------- Some time ago I implemented one out-of-tree collectd plugin leveraging the libvirt bulk stats: https://github.com/fromanirh/collectd-ovirt This plugin is meant to be a modern, drop-in replacement for the existing virt plugin. The development of that out of tree plugin is now halted, because we have everything we need in the upstream collectd plugin. Future work ------------------ We believe we have reached feature parity, so we are looking for bugixes/performance tuning in the near term future. I'll be happy to provide more patches/PRs about that. Thanks and bests, -- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani

5 6

[ OST Failure Report ] [ oVirt master ] [ 26.02.2017 ] [test-repo_ovirt_experimental_master]
by Shlomo Ben David 26 Feb '17

26 Feb '17

Hi, Test failed: [ test-repo_ovirt_experimental_master ] Link to Job: [1] Link to all logs: [2] Link to error log: [3] [1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538 [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538/artif… [3] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538/artif… Error snippet from the log: <error> 2017-02-26 05:35:53,340-0500 ERROR (jsonrpc/3) [storage.TaskManager.Task] (Task='22828901-d87f-4607-9690-a106c474ebe4') Unexpected error (task:871) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 878, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 52, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3200, in teardownImage dom.deactivateImage(imgUUID) File "/usr/share/vdsm/storage/blockSD.py", line 1278, in deactivateImage lvm.deactivateLVs(self.sdUUID, volUUIDs) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 1304, in deactivateLVs _setLVAvailability(vgName, toDeactivate, "n") File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 845, in _setLVAvailability raise error(str(e)) CannotDeactivateLogicalVolume: Cannot deactivate Logical Volume: ('General Storage Exception: ("5 [] [\' Logical volume 1dd0ee2a-f26a-423c-90f9-79703343aa1e/8323e511-eb93-4b1c-a9fa-ad66409994e7 in use.\', \' Logical volume 1dd0ee2a-f26a-423c-90f9-79703343aa1e/99662dfa-acf2-4392-8ab8-106412c2afa5 in use.\']\\n1dd0ee2a-f26a-423c-90f9-79703343aa1e/[\'99662dfa-acf2-4392-8ab8-106412c2afa5\', \'8323e511-eb93-4b1c-a9fa-ad66409994e7\']",)',) 2017-02-26 05:35:53,347-0500 INFO (jsonrpc/3) [storage.TaskManager.Task] (Task='22828901-d87f-4607-9690-a106c474ebe4') aborting: Task is aborted: 'Cannot deactivate Logical Volume' - code 552 (task:1176) 2017-02-26 05:35:53,348-0500 ERROR (jsonrpc/3) [storage.Dispatcher] {'status': {'message': 'Cannot deactivate Logical Volume: (\'General Storage Exception: ("5 [] [\\\' Logical volume 1dd0ee2a-f26a-423c-90f9-79703343aa1e/8323e511-eb93-4b1c-a9fa-ad66409994e7 in use.\\\', \\\' Logical volume 1dd0ee2a-f26a-423c-90f9-79703343aa1e/99662dfa-acf2-4392-8ab8-106412c2afa5 in use.\\\']\\\\n1dd0ee2a-f26a-423c-90f9-79703343aa1e/[\\\'99662dfa-acf2-4392-8ab8-106412c2afa5\\\', \\\'8323e511-eb93-4b1c-a9fa-ad66409994e7\\\']",)\',)', 'code': 552}} (dispatcher:78) 2017-02-26 05:35:53,349-0500 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Image.teardown failed (error 552) in 19.36 seconds (__init__:552) </error> Best Regards, Shlomi Ben-David | Software Engineer | Red Hat ISRAEL RHCSA | RHCVA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1

4 4

[REQUEST FOR COMMENTS] oVirt features tracking and documentation
by Sandro Bonazzola 26 Feb '17

26 Feb '17

Hi, I'm facing a new challenge today, I see new features getting pushed to oVirt gerrit intentionally out of bugzilla. The specific case is not really relevant, but just for reference: https://gerrit.ovirt.org/#/c/65761/ . Looks like we'll see features getting merged without any RFE opened in bugzilla for them. With current workflow, auto-generating release notes from bugzilla doc-texts. this means they won't ever be documented. I'd like to open this public discussion getting comments about how things will be handled. Thanks, -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

5 8

[HEADS UP] ovift-engine: repository closure check on check-patch
by Sandro Bonazzola 26 Feb '17

26 Feb '17

Hi, with the following commits: - https://gerrit.ovirt.org/71553 - https://gerrit.ovirt.org/71623 a minimal repository closure check is added to check-patches.sh. Instead of performing a full blown repository closure, I limited the check to the built rpms in oreder to reduce the risk of hitting repository closure errors due to different projects. This should help dropping the long time failing repository closure jobs which are struggling to keep the pace with the changes we introduce in master. It should also help avoiding to merge patches which will break the repository and the plain rpm install flow. The second patch switch from ovirt-master-release to experimental repo. Here I need infra input, not sure experimental is the desired repo, deisgned to replace ovirt-master-snapshot. Thanks, -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

2 4

Vdsm build failure
by Piotr Kliczewski 24 Feb '17

24 Feb '17

Nir, I saw this issue in [1]: 10:20:12 FAIL: test_send_receive (storage_mailbox_test.TestMailbox) 10:20:12 ---------------------------------------------------------------------- 10:20:12 Traceback (most recent call last): 10:20:12 File "/home/jenkins/workspace/vdsm_master_check-patch-fc25-x86_64/vdsm/tests/storage_mailbox_test.py", line 132, in test_send_receive 10:20:12 self.assertFalse(expired, 'message was not processed on time') 10:20:12 AssertionError: message was not processed on time it is something known? Thanks, Piotr [1] http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc25-x86_64/1541/conso…

2 4

planned Jenkins restart
by Evgheni Dereveanchin 24 Feb '17

24 Feb '17

Hi everyone, I'll be performing a planned Jenkins restart within the next hour. No new builds will be scheduled during this maintenance period. I will announce you once it is finished. Regards, Evgheni Dereveanchin

1 1

[ OST Failure Report ] [ oVirt master ] [ 22/02/2017 ] [add_secondary_storage_domains]
by Barak Korren 23 Feb '17

23 Feb '17

Test failed: [ add_secondary_storage_domains ] Note: - This may or may not be related to https://bugzilla.redhat.com/show_bug.cgi?id=1421945 The BZ talks about sporadic failures, while this seems to be happening consistently (for 6 runs so far) Link to suspected patches: - https://gerrit.ovirt.org/70415 - https://gerrit.ovirt.org/69157 Link to Job: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5487/ Link to all logs: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5487/artif… Error snippet from log: <error> RequestError: status: 400 reason: Bad Request detail: Cannot import Virtual Disk: Storage Domain cannot be accessed. -Please check that at least one Host is operational and Data Center state is up. </error> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5487/testR… -- Barak Korren bkorren(a)redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/

4 13

oVirt Engine 4.1.1 has been branched out
by Tal Nisan 23 Feb '17

23 Feb '17

Hi everyone, Since we've moved to blocker/exception only mode we've branched oVirt engine 4.1.1 today from the ovirt-engine-4.1 branch to allow work on 4.1.3 bugs. >From this moment on 4.1.1 engine bugs have to be merged to: master ovirt-engine-4.1 ovirt-engine-4.1.1.z For 4.1.3 only the first 2 branches. Thanks.

5 8

[ OST Failure Report ] [ oVirt master ] [ 22/2/2017 ] [prepare_migration_attachments_ipv4]
by Barak Korren 23 Feb '17

23 Feb '17

Test failed: [ prepare_migration_attachments_ipv4 ] Link to suspected patches: - Link to Job: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5480/ Link to all logs: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5480/artif… Error snippet from log: <error> status: 409 reason: Conflict detail: Cannot setup Networks. Operation can be performed only when Host status is Maintenance, Up, NonOperational. </error> -- Barak Korren bkorren(a)redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/

2 2

[Important!][Action required!] CI-whitelist re-activated
by Barak Korren 23 Feb '17

23 Feb '17

TL;DR: If you are a patch author, please ask your project maintainer to have you added to the CI whitelist. The oVirt CI system tries to be a useful and powerful tool for patch authors and project maintainers. With the current CI-standards, power is placed in the hands of patch authors to make the system do almost anything. We try to be an "open" Open-Source project, so permission is given to anyone to open a Gerrit account and submit patches. The above opens the door to the CI system being maliciously exploited, so some countermeasure was needed. The builders of the CI system foresaw this and have put in place a white-list mechanism that makes the CI system only run jobs for patches that come from listed authors. In recent years the CI system had been re-engineered with the push towards the CI standards, and the whitelist mechanism was rendered non-active. Finally realizing the potential for harm, we acted quickly to re-enable the whitelist mechanism. We made an effort to include current authors and maintainers, but we do not know everyone. People not included in the whitelist will see the following message when submitting patches: To avoid overloading the infrastructure, a whitelist for running gerrit triggered jobs has been set in place, if you feel like you should be in it, please contact infra at ovirt dot org If you come across this message, please ask the project maintainer to send a message to the infra team asking for you to added to the CI whitelist. We'd rather not receive direct messages from individual contributors, because we do not know everyone and cannot verify sources. We (the infra team) know that the current mechanism can sound draconian and be inconvenient. But we had to put something in place quickly. Please join the discussion at [1] to improve it. [1]: https://ovirt-jira.atlassian.net/browse/OVIRT-1154 -- Barak Korren bkorren(a)redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/

4 6