- Devel - oVirt List Archives

Proposal: Hystrix for realtime command monitoring
by Roman Mohr 15 Dec '15

15 Dec '15

Hi All, I am contributing to the engine for three months now. While I dug into the code I started to wonder how to visualize what the engine is actually doing. To get better insights I added hystrix[1] to the engine. Hystrix is a circuit breaker library which was developed by Netflix and has one pretty interesting feature: Real time metrics for commands. In combination with hystrix-dashboard[2] it allows very interesting insights. You can easily get an overview of commands involved in operations, their performance and complexity. Look at [2] and the attachments in [5] and [6] for screenshots to get an Impression. I want to propose to integrate hystrix permanently because from my perspective the results were really useful and I also had some good experiences with hystrix in past projects. A first implementation can be found on gerrit[3]. # Where is it immediately useful? During development and QA. An example: I tested the hystrix integration on /api/vms and /api/hosts rest endpoints and immediately saw that the number of command exectuions grew lineary whit the number of vms and hosts. The bug reports [5] and [6] are the result. # How to monitor the engine? It is as easy as starting a hystrix-dashboard [2] with $ git clone https://github.com/Netflix/Hystrix.git $ cd Hystrix/hystrix-dashboard $ ../gradlew jettyRun and point the dashboard to https://<customer.engine.ip>/ovirt-engine/hystrix.stream. # Other possible benefits? * Live metrics at customer site for admins, consultants and support. * Historical metrics for analysis in addition to the log files. The metrics information is directly usable in graphite [7]. Therefore it would be possible to collect the json stream for a certain time period and analyze them later like in [4]. To do that someone just has to run curl --user admin@internal:engine http://localhost:8080/ovirt-engine/api/hystrix.stream > hystrix.stream for as long as necessary. The results can be analyzed later. # Possible architectural benefits? In addition to the live metrics we might also have use for the real hystrix features: * Circuit Breaker * Bulk execution of commands * De-dublication of commands (Caching) * Synchronous and asynchronous execution support * ... Our commands do already have a lot of features, so I don't think that there are some quick wins, but maybe there are interesting opportunities for infra. # Overhead? In [5] the netflix employees describe their results regarding the overhead of wrapping every command into a new instance of a hystrix command. They ran their tests on a standard 4-core Amazon EC2 server with a load of 60 request per second. When using threadpools they measured a mean overhead of less than one millisecond (so negligible). At the 90th percentile they measured an overhead of 3 ms. At the 99th percentile of about 9 ms. When configuring the hystrix commands to use semaphores instead of threadpools they are even faster. # How to integrate? A working implementation can be found on gerrit[3]. These patch sets wrap a hystrix command around every VdcAction, every VdcQuery and every VDSCommand. This just required four small modifications in the code base. # Security? In the provided patches the hystrix-metrics-servlet is accessible at /ovirt-engine/api/hystrix.stream. It is protected by basic auth but accessible for everyone who can authenticate. We should probably restrict it to admins. # Todo? 1) We do report failed actions with return values. Hystrix expects failing commands to throw an exception. So on the dashboard almost every command looks like a success. To overcome this, it would be pretty easy to throw an exception inside the command and catch it immediately after it leaves the hystrix wrapper. 2) Finetuning Do we want semaphores or a thread pool. When the thread pool, what size do we want? 3) Three unpackaged dependencies: archaius, hystrix-core, hystrix-contrib # References [1] https://github.com/Netflix/Hystrix [2] https://github.com/Netflix/Hystrix/tree/master/hystrix-dashboard [3] https://gerrit.ovirt.org/#/q/topic:hystrix [4] http://www.nurkiewicz.com/2015/02/storing-months-of-historical-metrics.html [5] https://github.com/Netflix/Hystrix/wiki/FAQ#what-is-the-processing-overhead… [5] https://bugzilla.redhat.com/show_bug.cgi?id=1268216 [6] https://bugzilla.redhat.com/show_bug.cgi?id=1268224 [7] http://graphite.wikidot.com

3 5

Debian support for vdsm and subpackages
by Yaniv Bronheim 14 Dec '15

14 Dec '15

We currently manage debian packaging files under https://gerrit.ovirt.org/#/q/project:releng-tools instead of in the project itself. imo it makes it harder, as developer won't update the debian folder while changing spec parts , which makes much more observation work for Simone.. But if we keep to manage debian that way we should remove the debian folder from cpopen, vdsm, pthreading to avoid duplication. Simone can you explain more how you manage to flow code changes and update the debian support currently? should I indeed drop the current debian code that we have in vdsm? -- *Yaniv Bronhaim.*

4 4

suggestion to merge ovirt-engine unit-tests in master to standard ci
by Eyal Edri 14 Dec '15

14 Dec '15

FYI, As part of moving all jobs to the standard ci [1], I'd like to propose to move the following into 'check-patch' and 'check-merged' : 1. unit tests 2. find-bugs Currently these run as separate jobs and not even yamelized. My only concern is populating junit & findbugs results into jenkins UI to enable reviewing the errors and continue to enforce logic from the original jobs, like '0' high/med/low bugs on find-bugs. Thoughts/Concerns? -- Eyal Edri Supervisor, RHEV CI EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

4 4

About jenkins load
by David Caro 14 Dec '15

14 Dec '15

--HnQK338I3UIa/qiP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable While trying to figure out a solution to the current jenkins load issue, I = got some stats on the current build execution in jenkins, and just so we all are aware of the numbers. In the last month, the number of builds per day has been multiplied by 1750= %, that is, from ~500 build/day tops to more than 8750 build/day The percentage of failures has gone down too (around 5% in the last week). = That does not mean that the false negatives is going down too, but for sure it m= eans that it's not raising a lot if raising at all. For the last 7 days (including weekend) we ran 25858 builds, from which only 928 were failures (combined real and false negatives, no easy way to know y= et) The daily average build time is at around 10min, that does not include the = time waiting in the queue though. For the queues, we had a big peak of >750 queued jobs on tuesday (lasting u= ntil thursday), with 4 smaller peaks of 300-200 jobs on last week's monday,tuesd= ay and wednesday and on this week's monday. Will keep digging into this, cheers! --=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --HnQK338I3UIa/qiP Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJWaswuAAoJEEBxx+HSYmnDjKgH/0gZgR1gla/kod6p+S+jInKX SSebg/BU/5PQ9/f1MDq0p09eyQZDMCtRzY4xXm9KtJTVYnL5+P1w7ptiy7h2sdOD pd4Z69vo7KY0G+m+Wg4+G7Eg3bfDA3t6TJj21WIAGsL2mxfcWe+J6EPHS4ZH3lMj M1JY94rMZEdF9Cl5UeNv9N4r8n4W8aTKbYqOFonQdvNrj86GAtU9KXrbkALnwlyc +cuDXhEVCazFMcTDw0T9XBBBYSBjrkEY8eR/nUVxLdjeHorfbKXO58LcKrv+hOU/ 26ISR7/JLQ4jn8eAQHswZNOZeapPct5LSRgGfKtFJiCWHRWuiC13mUOP+R/qrxc= =PPLc -----END PGP SIGNATURE----- --HnQK338I3UIa/qiP--

2 1

[vdsm] branching out 3.6.1 monday 20151214
by Francesco Romani 14 Dec '15

14 Dec '15

Hi all, After last 3.6.1-RC3 build, I plan to branch out 3.6.1 from branch 3.6 next monday 2015/12/14. vdsm +- ovirt-3.6 +- ovirt-3.6.1 Hopefully we will not to use it ever; should we require another RC before final, it will be from that branch, so please remember to backport 3.6.1 patches on: - ovirt-3.6 (as before) - ovirt-3.6.1 At the moment only urgent fixes, targeted for 3.6.1, are being merged into ovirt-3.6 VDSM branch. After we branch out 3.6.1, I'll reopen the floodgates, merging patches targeted for 3.6.2. Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani

1 1

Master building issue
by Fred Rolland 13 Dec '15

13 Dec '15

Hi , I have issues building the engine on master. The build finishes quick with this error : packaging/setup/plugins/ovirt-engine-setup/ovirt-engine-common/distro-rpm/packages.py:188:9: E301 expected 1 blank line, found 0 Makefile:316: recipe for target 'validations' failed Is there a fix for it ? Thanks, Fred

3 2

[ANN] oVirt 3.6.1 Fourth Release Candidate is now available for testing
by Sandro Bonazzola 11 Dec '15

11 Dec '15

The oVirt Project is pleased to announce the availability of the Fourth Release Candidate of oVirt 3.6.1 for testing, as of December 10th, 2015. This release is available now for Fedora 22, Red Hat Enterprise Linux 6.7, CentOS Linux 6.7 (or similar) and Red Hat Enterprise Linux >= 7.1, CentOS Linux >= 7.1 (or similar). This release supports Hypervisor Hosts running Red Hat Enterprise Linux >= 7.1, CentOS Linux >= 7.1 (or similar) and Fedora 22. Highly experimental support for Debian 8.2 Jessie has been added too. This release candidate includes updated packages for: - ovirt-engine - ovirt-engine-extension-aaa-ldap This release of oVirt 3.6.1 includes numerous bug fixes. See the release notes [1] for an initial list of the new features and bugs fixed. Please refer to release notes [1] for Installation / Upgrade instructions. A new oVirt Live ISO will be available soon[2]. Please note that mirrors[3] may need usually one day before being synchronized. Please refer to the release notes for known issues in this release. [1] http://www.ovirt.org/OVirt_3.6.1_Release_Notes [2] http://resources.ovirt.org/pub/ovirt-3.6-pre/iso/ [3] http://www.ovirt.org/Repository_mirrors#Current_mirrors -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

3 3

oVirt engine 4.0 will require WildFly 10 / EAP 7
by Martin Perina 11 Dec '15

11 Dec '15

Hi, we are going to merge patches which drops EAP 6 support [1] and requires WildFly 10 / EAP 7 [2] for oVirt engine master on Monday December 14th at 10:00 CET. This is a huge step as we will finally be able to use all new features provided by WildFly 10 (J2EE 7, Java 8, RESTEasy 3, ActiveMQ Artemis, Hibernate 5, ...). Upgrade of code base to use those new features will be done incrementally, for example this patch [3] cleans up our CDI related code and upgrades to CDI 1.2. So on Monday please upgrade WildFly packages on your development machines to: ovirt-engine-wildfly-10.0.0 ovirt-engine-wildfly-overlay-10.0.0 and rebase your patches on top of latest master. To ease backporting of patches to oVirt 3.6 we have already merged patches which made oVirt 3.6 compatible with WildFly 10, but please bear in mind that oVirt 3.6 have to be compatible with EAP 6 and there are no plans to upgrade it to WildFly 10! Please contact me or other members of infra team if you have any issue with this upgrade. Thanks Martin Perina [1] https://gerrit.ovirt.org/48208 [2] https://gerrit.ovirt.org/48209 [3] https://gerrit.ovirt.org/48305

2 2

Re: [ovirt-devel] Question about https://gerrit.ovirt.org/#/c/49860/
by Allon Mureinik 10 Dec '15

10 Dec '15

Adding devel list, which was dropped by mistake. On Thu, Dec 10, 2015 at 1:56 PM, Allon Mureinik <amureini(a)redhat.com> wrote: > > > On Wed, Dec 9, 2015 at 6:51 PM, Vojtech Szocs <vszocs(a)redhat.com> wrote: > >> Hi Allon, >> >> not sure if you saw my email (below) but I was wondering >> what's your opinion about test method naming conventions. >> > Generally agree with that, and I tried excluding them where possible - if > I missed any, please say so. > > > >> >> Also, regarding patch https://gerrit.ovirt.org/#/c/49910/ >> why not simply exclude all Java files under src/test/java >> for all projects? (What's the advantage of having per-file >> exclude list of test classes?) >> > I hat overly-broad exclusions. > There are a ton of badly named method in test classes, and I don't want to > encourage any more. > Although if you think this is getting out of hand, we could just do that... > > > >> >> To address other files, like "Other GWT-related classes" >> in [1] -- did you consider adding some source-level Java >> annotation like @DontCheckMethodNames or similar? (Again, >> avoid per-file exclude list of other classes.) >> > Yeah, but I don't want to have to write my own checker - this is a > standard checkstyle check... > > >> >> [1] >> https://gerrit.ovirt.org/#/c/49910/4/build-tools-root/checkstyles/src/main/… >> >> Thanks, >> Vojtech >> >> >> ----- Forwarded Message ----- >> From: "Vojtech Szocs" <vszocs(a)redhat.com> >> To: "devel" <devel(a)ovirt.org> >> Sent: Monday, December 7, 2015 8:00:36 PM >> Subject: [ovirt-devel] Question about https://gerrit.ovirt.org/#/c/49860/ >> >> Hi Allon! >> >> patch in $subject modifies test method name from: >> >> testGetLocaleFromStringString_withDefault >> >> to: >> >> testGetLocaleFromStringStringWithDefault >> >> I've always thought that Java test method names should be >> as much readable as possible, because in Java tests, the >> method name is used to describe the particular test case. >> >> (Unfortunately, Java test frameworks don't do a good job >> in this area.. please correct me if I'm wrong.) >> >> So with JS/Jasmine we can do this: >> >> describe('LocaleUtils', function () { >> it('should extract locale from given String', function () { >> // actual test code >> }); >> }); >> >> but in Java we usually do this: >> >> // <methodName>_<extraInfoForSpecificTestCase> >> void testGetLocaleFromStringString_withDefault >> >> which is already far from readable, and this: >> >> testGetLocaleFromStringStringWithDefault >> >> makes it even worse.. Even though it complies with Java >> method naming convention, it totally obfuscates what is >> that particular test case about, which means people who >> want to maintain such tests need to spend time studying >> what the test is actually doing.. >> >> What's your take on this? >> >> Regards, >> Vojtech >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > >

1 0

unable to add host ~ InitVdsOnUpCommand parameters specifies storage domain ID when creating host.
by Martin Mucha 10 Dec '15

10 Dec '15

Hi, 'm unable to add host to engine. I have fresh engine build, cleaned host, dropped and recreated DB. I'm adding host using rest¹. Command InitVdsOnUpCommand is called during operation, having getStoragePoolId() equal to 00000001-0001-0001-0001-0000000000c1. However, there's no storage domain in system — it's empty installation, and I'm only creating host yet (so I cannot have any storage domain). Because of that, following code: StorageDomain masterDomain = getStorageDomainDao().getStorageDomains(storagePoolId, StorageDomainType.Master).get(0); in method: org.ovirt.engine.core.bll.InitVdsOnUpCommand#runConnectHostToPoolEvent produces NPE, which is probably cause of host marked as nonoperational. ——— If I'm doing something wrong, plese advice. Martin. [1] curl -v -u "admin@internal:adminadmin" -H "Content-type: application/xml" -d " <host> <name>$NAME</name> <address>10.34.63.167</address> <root_password>qum5net</root_password> <cluster> <name>$CLUSTERNAME</name> </cluster> </host> " -X POST http://localhost:8080/ovirt-engine/api/hosts

1 0