AppErrors cleanup
by Allon Mureinik
Hi all,
A recent bug [1] reported as part of the translation effort alerted me to the fact that we have a lot (and I mean a LOT - over 100 per file) of deprecated, unused keys in the various AppErrors files that serve no purpose and just take up space and waste translators time when they examine them.
To make a long story short - I've just merged a patch to remove all these useless messages, and enforce via unit tests that EVERY key there should have a corresponding constant in the EngineMessage or EngineError enums.
Many thanks to my reviewers!
I know this was an tedious patch that couldn't have been too fun to review.
-Allon
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1244766
8 years, 4 months
Lago testing environment contribution
by Yaniv Bronheim
Hi list,
If you're not aware yet about Lago you should. This our engine to test
functional flows over variant distributions for ovirt-engine and vdsm
This project is currently maintained by dcaro but it will love to have more
contribution in reviews, tests and more.
If you want to follow it I encourage to join the mailing list
lago-devel(a)ovirt.org and help with reviews if possible ([2])
You can also checkout following links for more info
In [3] you can find full readme about how to use it to run ovirt-engine
tests. should be intuitive and quick. If not, please raise
[1] https://gerrit.ovirt.org/#/admin/projects/lago
[2] https://gerrit.ovirt.org/#/q/project:+lago+status:+open
[3] https://github.com/ovirt/lago
Hope you'll find it interesting and Lago will get more attention by our
community
Greetings,
--
*Yaniv Bronhaim.*
8 years, 4 months
Proposal: Hystrix for realtime command monitoring
by Roman Mohr
Hi All,
I am contributing to the engine for three months now. While I dug into the
code I
started to wonder how to visualize what the engine is actually doing.
To get better insights I added hystrix[1] to the engine. Hystrix is a
circuit
breaker library which was developed by Netflix and has one pretty
interesting
feature: Real time metrics for commands.
In combination with hystrix-dashboard[2] it allows very interesting
insights.
You can easily get an overview of commands involved in operations, their
performance and complexity. Look at [2] and the attachments in [5] and [6]
for
screenshots to get an Impression.
I want to propose to integrate hystrix permanently because from my
perspective
the results were really useful and I also had some good experiences with
hystrix
in past projects.
A first implementation can be found on gerrit[3].
# Where is it immediately useful?
During development and QA.
An example: I tested the hystrix integration on /api/vms and /api/hosts rest
endpoints and immediately saw that the number of command exectuions grew
lineary whit the number of vms and hosts. The bug reports [5] and [6] are
the
result.
# How to monitor the engine?
It is as easy as starting a hystrix-dashboard [2] with
$ git clone https://github.com/Netflix/Hystrix.git
$ cd Hystrix/hystrix-dashboard
$ ../gradlew jettyRun
and point the dashboard to
https://<customer.engine.ip>/ovirt-engine/hystrix.stream.
# Other possible benefits?
* Live metrics at customer site for admins, consultants and support.
* Historical metrics for analysis in addition to the log files.
The metrics information is directly usable in graphite [7]. Therefore it
would be
possible to collect the json stream for a certain time period and
analyze them
later like in [4]. To do that someone just has to run
curl --user admin@internal:engine
http://localhost:8080/ovirt-engine/api/hystrix.stream > hystrix.stream
for as long as necessary. The results can be analyzed later.
# Possible architectural benefits?
In addition to the live metrics we might also have use for the real hystrix
features:
* Circuit Breaker
* Bulk execution of commands
* De-dublication of commands (Caching)
* Synchronous and asynchronous execution support
* ...
Our commands do already have a lot of features, so I don't think that there
are
some quick wins, but maybe there are interesting opportunities for infra.
# Overhead?
In [5] the netflix employees describe their results regarding the overhead
of
wrapping every command into a new instance of a hystrix command.
They ran their tests on a standard 4-core Amazon EC2 server with a load of
60
request per second.
When using threadpools they measured a mean overhead of less than one
millisecond (so negligible). At the 90th percentile they measured an
overhead
of 3 ms. At the 99th percentile of about 9 ms.
When configuring the hystrix commands to use semaphores instead of
threadpools
they are even faster.
# How to integrate?
A working implementation can be found on gerrit[3]. These patch sets wrap a
hystrix command around every VdcAction, every VdcQuery and every VDSCommand.
This just required four small modifications in the code base.
# Security?
In the provided patches the hystrix-metrics-servlet is accessible at
/ovirt-engine/api/hystrix.stream. It is protected by basic auth but
accessible
for everyone who can authenticate. We should probably restrict it to admins.
# Todo?
1) We do report failed actions with return values. Hystrix expects failing
commands to throw an exception. So on the dashboard almost every command
looks
like a success. To overcome this, it would be pretty easy to throw an
exception inside the command and catch it immediately after it leaves the
hystrix wrapper.
2) Finetuning
Do we want semaphores or a thread pool. When the thread pool, what size do
we want?
3) Three unpackaged dependencies: archaius, hystrix-core, hystrix-contrib
# References
[1] https://github.com/Netflix/Hystrix
[2] https://github.com/Netflix/Hystrix/tree/master/hystrix-dashboard
[3] https://gerrit.ovirt.org/#/q/topic:hystrix
[4]
http://www.nurkiewicz.com/2015/02/storing-months-of-historical-metrics.html
[5]
https://github.com/Netflix/Hystrix/wiki/FAQ#what-is-the-processing-overhe...
[5] https://bugzilla.redhat.com/show_bug.cgi?id=1268216
[6] https://bugzilla.redhat.com/show_bug.cgi?id=1268224
[7] http://graphite.wikidot.com
8 years, 4 months
Debian support for vdsm and subpackages
by Yaniv Bronheim
We currently manage debian packaging files under
https://gerrit.ovirt.org/#/q/project:releng-tools instead of in the project
itself. imo it makes it harder, as developer won't update the debian folder
while changing spec parts , which makes much more observation work for
Simone..
But if we keep to manage debian that way we should remove the debian folder
from cpopen, vdsm, pthreading to avoid duplication.
Simone can you explain more how you manage to flow code changes and update
the debian support currently? should I indeed drop the current debian code
that we have in vdsm?
--
*Yaniv Bronhaim.*
8 years, 4 months
suggestion to merge ovirt-engine unit-tests in master to standard ci
by Eyal Edri
FYI,
As part of moving all jobs to the standard ci [1], I'd like to propose to
move the following into
'check-patch' and 'check-merged' :
1. unit tests
2. find-bugs
Currently these run as separate jobs and not even yamelized.
My only concern is populating junit & findbugs results into jenkins UI to
enable reviewing
the errors and continue to enforce logic from the original jobs, like '0'
high/med/low bugs on find-bugs.
Thoughts/Concerns?
--
Eyal Edri
Supervisor, RHEV CI
EMEA ENG Virtualization R&D
Red Hat Israel
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
8 years, 4 months
About jenkins load
by David Caro
--HnQK338I3UIa/qiP
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
While trying to figure out a solution to the current jenkins load issue, I =
got
some stats on the current build execution in jenkins, and just so we all are
aware of the numbers.
In the last month, the number of builds per day has been multiplied by 1750=
%,
that is, from ~500 build/day tops to more than 8750 build/day
The percentage of failures has gone down too (around 5% in the last week). =
That
does not mean that the false negatives is going down too, but for sure it m=
eans
that it's not raising a lot if raising at all.
For the last 7 days (including weekend) we ran 25858 builds, from which only
928 were failures (combined real and false negatives, no easy way to know y=
et)
The daily average build time is at around 10min, that does not include the =
time
waiting in the queue though.
For the queues, we had a big peak of >750 queued jobs on tuesday (lasting u=
ntil
thursday), with 4 smaller peaks of 300-200 jobs on last week's monday,tuesd=
ay
and wednesday and on this week's monday.
Will keep digging into this, cheers!
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
IRC: dcaro|dcaroest@{freenode|oftc|redhat}
Web: www.redhat.com
RHT Global #: 82-62605
--HnQK338I3UIa/qiP
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJWaswuAAoJEEBxx+HSYmnDjKgH/0gZgR1gla/kod6p+S+jInKX
SSebg/BU/5PQ9/f1MDq0p09eyQZDMCtRzY4xXm9KtJTVYnL5+P1w7ptiy7h2sdOD
pd4Z69vo7KY0G+m+Wg4+G7Eg3bfDA3t6TJj21WIAGsL2mxfcWe+J6EPHS4ZH3lMj
M1JY94rMZEdF9Cl5UeNv9N4r8n4W8aTKbYqOFonQdvNrj86GAtU9KXrbkALnwlyc
+cuDXhEVCazFMcTDw0T9XBBBYSBjrkEY8eR/nUVxLdjeHorfbKXO58LcKrv+hOU/
26ISR7/JLQ4jn8eAQHswZNOZeapPct5LSRgGfKtFJiCWHRWuiC13mUOP+R/qrxc=
=PPLc
-----END PGP SIGNATURE-----
--HnQK338I3UIa/qiP--
8 years, 4 months
[vdsm] branching out 3.6.1 monday 20151214
by Francesco Romani
Hi all,
After last 3.6.1-RC3 build, I plan to branch out 3.6.1 from
branch 3.6 next monday 2015/12/14.
vdsm
+- ovirt-3.6
+- ovirt-3.6.1
Hopefully we will not to use it ever; should we require another
RC before final, it will be from that branch, so please remember to
backport 3.6.1 patches on:
- ovirt-3.6 (as before)
- ovirt-3.6.1
At the moment only urgent fixes, targeted for 3.6.1,
are being merged into ovirt-3.6 VDSM branch.
After we branch out 3.6.1, I'll reopen the floodgates,
merging patches targeted for 3.6.2.
Bests,
--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
8 years, 4 months
Master building issue
by Fred Rolland
Hi ,
I have issues building the engine on master.
The build finishes quick with this error :
packaging/setup/plugins/ovirt-engine-setup/ovirt-engine-common/distro-rpm/packages.py:188:9:
E301 expected 1 blank line, found 0
Makefile:316: recipe for target 'validations' failed
Is there a fix for it ?
Thanks,
Fred
8 years, 4 months
