- Devel - oVirt List Archives

Evaluating Errai JAX-RS for oVirt
by Martin Betak 25 Nov '14

25 Nov '14

Hi all, I've been exploring the possibilities of utilizing Errai JAX-RS for REST client in our current GWT-based frontend. This would be a complement to Vojtech's oVirt.js library which is very native to javascript and utilizes dynamic discovery of resources and actions. Errai JAX-RS would enable us to reuse our restapi definitions of resource interfaces and entities and generate proxies automatically thus eliminating the need for extensive manual code generation. [errai jax-rs] https://docs.jboss.org/author/display/ERRAI/Errai+JAX-RS Unfortunately in my quest I stumbled upon some issues I couldn't exactly google or solve and I would like to ask Mark or Christian, whether the below is somehow possible with Errai: 1) Adding custom header to every request (some form of request interceptors) You, see our api returns by default XML and we of course want to consume JSON. Thus adding 'Content-Type: application/json' as well as other custom authentication headers is necessary for our usage. 2) Using custom JSONProvider/ObjectMapper instead of the default Jackson one. In our JSONProvider [1] we use custom ObjectMapper [2] mainly to map JAX-B annotations to json. Without this for example errai generates marshaller for the VMs resource public class VMs { @XmlElement(name = "vm") protected List<VM> vMs; } that expects json field called "vMs" instead of the JAX-B specification "vm" which is the actual format returned from the API. There may be other differencies such as [3] so I think the best solution would be if we could tell errai to use specific JSONProvider for generating the marshallers. This are the issues that so far seem to be the most blocking ones, others may arise along the way. Thank you for any help provided. Best regards, Martin [1] https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/r… [2] https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/r… [3] https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/r… [resource schema] https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/r…

3 5

mom-0.4.3
by Adam Litke 25 Nov '14

25 Nov '14

Hi all, A recent commit [1] to vdsm (a99bacf) introduced a dependency on mom-0.4.3 but mom-0.4.3 does not yet exist. To work around this problem you may build a pre-release src.rpm [2] of mom-0.4.3 that includes the needed functionality. Once we have enough content for a mom point release I'll build the official upstream packages and release an update. Sorry for the inconvenience. [1] http://gerrit.ovirt.org/#/c/35407/ [2] http://people.redhat.com/~alitke/mom-0.4.3-0.0.aglpre.fc20.src.rpm -- Adam Litke

1 0

[QE][ACTION REQUIRED] oVirt 3.5.1 RC status - postponed
by Sandro Bonazzola 25 Nov '14

25 Nov '14

Hi, We'were supposed to start composing oVirt 3.5.1 RC today *2014-11-25 08:00 UTC* from 3.5 branch. We have still blockers for oVirt 3.5.1 RC release so we need to postpone it untill they'll be fixed. The new tentative date for 3.5.1 RC composition is set to *2014-12-02 08:00 UTC The bug tracker [1] shows 4 open blocker: Bug ID Whiteboard Status Summary 1164218 POST glfs_set_volfile_server() method causes segmentation fault when bad arguments are passed. 1159839 storage POST New FC LUNs are not detected on hypervisor without a reboot 1162640 gluster POST supervdsm segfault in libgfapi while querying volume status detail 1160846 sla NEW Can't add disk to VM without specifying disk profile when the storage domain has more than one disk profile The following bugs have been keyworded as Regression and not marked as blockers: Bug ID Whiteboard Status Summary 1159314 sla NEW Error accessing ha metadata while deploying second host 1160846 sla NEW Can't add disk to VM without specifying disk profile when the storage domain has more than one disk profile 1138144 storage NEW [BLOCKED]Failed to autorecover storage domain after unblocking connection with host 1118349 storage NEW [vdsm] Creating DataCenter 3.5 using master domain V1 fails with InquireNotSupportedError 1165336 virt ASSIGNED FC20 qemu needs kvmclock bugfixes In order to stabilize the release a new branch ovirt-engine-3.5.1 will be created from the same git hash used for composing the RC. Maintainers: - Please be sure that 3.5 snapshot allow to create VMs before *2014-12-01 15:00 UTC* - Please be sure that no pending patches are going to block the release before *2014-12-01 15:00 UTC* - If any patch must block the RC release please raise the issue as soon as possible. There are still 103 bugs [2] targeted to 3.5.1. Excluding node and documentation bugs we still have 76 bugs [3] targeted to 3.5.1. Maintainers / Assignee: - Please review bugs marked as Regression and add to blocker if they're real regressions. - Please add the bugs to the tracker if you think that 3.5.1 should not be released without them fixed. - Please update the target to 3.5.2 or later for bugs that won't be in 3.5.1: it will ease gathering the blocking bugs for next releases. - Please fill release notes, the page has been created here [4] Community: - If you're testing oVirt 3.5 nightly snapshot, please add yourself to the test page [5] [1] http://bugzilla.redhat.com/1155170 [2] http://goo.gl/7G0PDV [3] http://goo.gl/6gUbVr [4] http://www.ovirt.org/OVirt_3.5.1_Release_Notes [5] http://www.ovirt.org/Testing/oVirt_3.5.1_Testing -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

3 4

Add a license
by 穆骏 25 Nov '14

25 Nov '14

------=_Part_137231_1211145132.1416792045826 Content-Type: text/plain; charset=GBK Content-Transfer-Encoding: 7bit How to add a license to ovirt engine, where can i start from? ------=_Part_137231_1211145132.1416792045826 Content-Type: text/html; charset=GBK Content-Transfer-Encoding: 7bit <div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial">How to add a license to ovirt engine, where can i start from?</div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span> ------=_Part_137231_1211145132.1416792045826--

2 5

where can i start reading ovirt-engine source code ?
by 穆骏 25 Nov '14

25 Nov '14

------=_Part_142630_2147045644.1416792693087 Content-Type: text/plain; charset=GBK Content-Transfer-Encoding: 7bit where can i start reading ovirt-engine source code ? ------=_Part_142630_2147045644.1416792693087 Content-Type: text/html; charset=GBK Content-Transfer-Encoding: 7bit <div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial">where  can i start reading ovirt-engine source code ?</div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span> ------=_Part_142630_2147045644.1416792693087--

2 1

[VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
by Francesco Romani 24 Nov '14

24 Nov '14

Hi everyone, I was doing JSON-RPC investigation recently, running VDSM on RHEL6.{5,6}, and while some initial profile work, I discovered a (little) performance degradation about pure JSONRPC coding/encoding, *only* on the above platforms. Here's why: - the JSON codec module in the python stdlib is based on simplejson - (thus) simplejson API is identical to stdlib's json module API - python 2.6 JSON codec is (based on) simplejson 1.9 - python 2.7 JSON codec is (based on) simplejson 2.0.9, with significant speedups. See https://docs.python.org/2/whatsnew/2.7.html#new-and-improved-modules and https://bugs.python.org/issue4136 - RHEL6.x/Centos 6.x includes python 2.6, so here (and only here) JSON codec is unnecessarily slower. - RHEL6.x (surely) and CentOS 6.x (need to check, don't see why not) includes anyway simplejson 2.0.9 as external package So, it seems to me that we can get python 2.7 speed even on stable platform with a five line patch: === cut here === diff --git a/lib/yajsonrpc/__init__.py b/lib/yajsonrpc/__init__.py index b3fd590..6682fc3 100644 --- a/lib/yajsonrpc/__init__.py +++ b/lib/yajsonrpc/__init__.py @@ -12,7 +12,10 @@ # You should have received a copy of the GNU General Public # License along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA -import json +try: + import simplejson as json +except ImportError + import json import logging from functools import partial from Queue import Queue === cut here === Here's an excerpt of the effects of this patch, using an initial still rough bencmark - once again 100 VMs boot, is the beginning of every testing routine from mine, so nothing new here. Let's consider two initial profiles. Please note that 1) the amount of calls is not the same (error from mine in profiling, need more precise bench) but still 2) using 2.0.9 module JSON just vanishes from the profile alltogether Vanilla VDSM, stdlib json, top 20 expensive calls ncalls tottime percall cumtime percall filename:lineno(function) 19/100 127.216 6.696 9.974 0.100 /usr/lib/python2.6/site-packages/mom/GuestMonitor.py:51(GuestMonitor.run) 1555/1635 41.180 0.026 58.566 0.036 /usr/lib64/python2.6/threading.py:481(Thread.run) 11708/1553160 11.967 0.001 37.921 0.000 /usr/lib64/python2.6/copy.py:144(deepcopy) 1268613 6.154 0.000 9.209 0.000 /usr/lib64/python2.6/copy.py:261(_keep_alive) 4890/130093 6.149 0.001 37.596 0.000 /usr/lib64/python2.6/copy.py:251(_deepcopy_dict) 497858/2070078 5.224 0.000 11.102 0.000 /usr/lib64/python2.6/json/encoder.py:284(JSONEncoder._iterencode) 498440/1333474 4.187 0.000 10.368 0.000 /usr/lib64/python2.6/json/encoder.py:213(JSONEncoder._iterencode_dict) 12/100 4.115 0.343 4.130 0.041 /usr/share/vdsm/virt/sampling.py:424(VmStatsThread.run) 43826 3.692 0.000 6.516 0.000 /usr/lib64/python2.6/xml/dom/expatbuilder.py:743(ExpatBuilderNS.start_element_handler) 7345 3.657 0.000 11.047 0.002 /usr/share/vdsm/virt/vm.py:2264(Vm._getRunningVmStats) 9666/320508 3.568 0.000 3.572 0.000 /usr/lib64/python2.6/xml/dom/minidom.py:305(_get_elements_by_tagName_helper) 274994/275000 3.339 0.000 5.593 0.000 /usr/lib64/python2.6/StringIO.py:208(StringIO.write) 90217 1.823 0.000 1.870 0.000 /usr/lib64/python2.6/xml/dom/minidom.py:349(Attr.__init__) 44025/44036 1.666 0.000 2.470 0.000 /usr/share/vdsm/storage/lvm.py:217(makeLV) 144212/144226 1.287 0.000 1.292 0.000 /usr/lib/python2.6/site-packages/vdsm/utils.py:285(convertToStr) 201 1.285 0.006 7.278 0.036 /usr/share/vdsm/storage/lvm.py:411(LVMCache._reloadlvs) 109880 1.235 0.000 1.283 0.000 /usr/lib64/python2.6/xml/dom/minidom.py:281(Document._append_child) 57874 1.163 0.000 1.799 0.000 /usr/lib64/python2.6/xml/dom/expatbuilder.py:274(ExpatBuilderNS.character_data_handler_cdata) 1130859 1.160 0.000 1.160 0.000 /usr/lib64/python2.6/copy.py:197(_deepcopy_atomic) 3021/3022 1.121 0.000 2.554 0.001 /usr/lib64/python2.6/inspect.py:247(getmembers) patch applied, top 20 expensive calls 24/100 117.222 4.884 13.372 0.134 /usr/lib/python2.6/site-packages/mom/GuestMonitor.py:51(GuestMonitor.run) 1209/1302 60.812 0.050 64.963 0.050 /usr/lib64/python2.6/threading.py:481(Thread.run) 9243/1235834 10.259 0.001 33.047 0.000 /usr/lib64/python2.6/copy.py:144(deepcopy) 13/100 7.007 0.539 7.281 0.073 /usr/share/vdsm/virt/sampling.py:424(VmStatsThread.run) 4116/103669 5.536 0.001 32.779 0.000 /usr/lib64/python2.6/copy.py:251(_deepcopy_dict) 9670/320560 4.437 0.000 4.443 0.000 /usr/lib64/python2.6/xml/dom/minidom.py:305(_get_elements_by_tagName_helper) 5653 3.232 0.001 9.443 0.002 /usr/share/vdsm/virt/vm.py:2264(Vm._getRunningVmStats) 43836 3.117 0.000 7.304 0.000 /usr/lib64/python2.6/xml/dom/expatbuilder.py:743(ExpatBuilderNS.start_element_handler) 274967/275000 2.955 0.000 6.833 0.000 /usr/lib64/python2.6/StringIO.py:208(StringIO.write) 1100/28500 2.376 0.002 12.881 0.000 /usr/share/vdsm/virt/vmxml.py:489(Element.__hacked_writexml) 43745/44036 1.729 0.000 2.736 0.000 /usr/share/vdsm/storage/lvm.py:217(makeLV) 90217/90221 1.695 0.000 1.953 0.000 /usr/lib64/python2.6/xml/dom/minidom.py:349(Attr.__init__) 201 1.549 0.008 7.725 0.038 /usr/share/vdsm/storage/lvm.py:411(LVMCache._reloadlvs) 57888 1.441 0.000 2.216 0.000 /usr/lib64/python2.6/xml/dom/expatbuilder.py:274(ExpatBuilderNS.character_data_handler_cdata) 2351/2354 1.262 0.001 2.750 0.001 /usr/lib64/python2.6/inspect.py:247(getmembers) 992794 1.232 0.000 8.027 0.000 /usr/lib64/python2.6/copy.py:261(_keep_alive) 4827 1.150 0.000 1.368 0.000 /usr/lib/python2.6/site-packages/mom/Policy/spark.py:211(Parser.buildState) 108040 1.119 0.000 1.119 0.000 /usr/lib/python2.6/site-packages/vdsm/utils.py:285(convertToStr) 336771/422122 1.106 0.000 2.575 0.000 /usr/share/vdsm/protocoldetector.py:108(MultiProtocolAcceptor._process_events) 883021 0.987 0.000 0.987 0.000 /usr/lib64/python2.6/copy.py:197(_deepcopy_atomic) Indeed this is a microbenchmark, but my point is we *recover* some speed at nearly zero cost and risk. Now, the question is: Do we want this patch? Do we want it on master? On 3.5 branch? Only on RHEL/Centos 6 platforms? -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani

7 13

[VDSM][JSON RPC] new vdsClient?
by Francesco Romani 23 Nov '14

23 Nov '14

Hi everyone, the recent JSONRPC discussion and my last work around vdsClient reminded me that since a long time there it was vague plans for a vdsClient overhaul or at least I haven't ever found a detailed plan :) Since we have to add jsonrpc support anyway, it could be a good chance to bootstrap a new vdsClient, and incrementally enhance it as needs arises; the old vdsClient will not go away very soon, I guess. At least, we can to put together a wishlist for a new vdsClient and add it to the wiki. My one, so far: * client library package: we should have a program-friendly python package to access VDSM, in order * JSON for input: I's like to open logs, grab a line including a command and feed vdsClient with the very same line Thoughts? -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani

5 8

[URGENT] failures while building ovirt-engine from 3.5 branch
by Sandro Bonazzola 22 Nov '14

22 Nov '14

Hi, can you please take a look to the following errors while building ovirt-engine from 3.5 branch? On Fedora 19: http://jenkins.ovirt.org/view/Stable%20branches%20per%20project/view/ovirt-… 12:33:13 [ERROR] COMPILATION ERROR : 12:33:13 [INFO] ------------------------------------------------------------- 12:33:13 [ERROR] /home/jenkins/workspace/ovirt-engine_3.5_create-rpms_merged/label/fc19/rpmbuild/BUILD/ovirt-engine-3.5.1/backend/manager/modules/dal/src/main/java/org/ovirt/engine/core/dal/dbbroker/DbFacadeLocator.java:[175,52] error: cannot access ShellLikeConfd 12:33:13 [INFO] 1 error On EL6: http://jenkins.ovirt.org/view/Stable%20branches%20per%20project/view/ovirt-… 12:43:47 Results : 12:43:47 12:43:47 Tests in error: 12:43:47 initializationError(org.ovirt.engine.api.restapi.resource.BackendResourceInfoDetailTest): Bad <init> method call from inside of a branch 12:43:47 Exception Details: 12:43:47 Location: 12:43:47 org/ovirt/engine/api/restapi/resource/AbstractBackendResourceLoggingTest.<init>(Lorg/powermock/core/IndicateReloadClass;)V @40: invokespecial 12:43:47 Reason: 12:43:47 Error exists in the bytecode 12:43:47 Bytecode: 12:43:47 0000000: 2a2b 4e4d 1210 b800 1604 bd00 0d59 032d 12:43:47 0000010: 5312 a6b8 001b b800 213a 0519 05b2 0025 12:43:47 0000020: a500 0e2a 01c0 0027 b700 2aa7 000a 2c2d 12:43:47 0000030: b700 2a01 57b1 12:43:47 Stackmap Table: 12:43:47 full_frame(@46,{UninitializedThis,Object[#39],UninitializedThis,Object[#39],Top,Object[#13]},{}) 12:43:47 full_frame(@53,{Object[#2],Object[#39],Object[#2],Object[#39],Top,Object[#13]},{}) 12:43:47 12:43:47 initializationError(org.ovirt.engine.api.restapi.resource.BackendResourceDebugDetailTest): Bad <init> method call from inside of a branch 12:43:47 Exception Details: 12:43:47 Location: 12:43:47 org/ovirt/engine/api/restapi/resource/AbstractBackendResourceLoggingTest.<init>(Lorg/powermock/core/IndicateReloadClass;)V @40: invokespecial 12:43:47 Reason: 12:43:47 Error exists in the bytecode 12:43:47 Bytecode: 12:43:47 0000000: 2a2b 4e4d 1210 b800 1604 bd00 0d59 032d 12:43:47 0000010: 5312 a6b8 001b b800 213a 0519 05b2 0025 12:43:47 0000020: a500 0e2a 01c0 0027 b700 2aa7 000a 2c2d 12:43:47 0000030: b700 2a01 57b1 12:43:47 Stackmap Table: 12:43:47 full_frame(@46,{UninitializedThis,Object[#39],UninitializedThis,Object[#39],Top,Object[#13]},{}) 12:43:47 full_frame(@53,{Object[#2],Object[#39],Object[#2],Object[#39],Top,Object[#13]},{}) 12:43:47 12:43:47 12:43:47 Tests run: 1524, Failures: 0, Errors: 2, Skipped: 74 Also for some reason it seems that the host is using German as locale. -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

2 1

contEIOVMs regression?
by Adam Litke 21 Nov '14

21 Nov '14

Hi list, I am taking a look at Bug 1157421 [1] which describes a situation where VM's that are paused with an -EIO error are not automatically resumed after the problem with storage has been corrected. I have some patches [2] on gerrit that resolve the problem. Since this appears to be a regression I am looking at a non-intrusive way to fix it in the 3.5 branch. There is some disagreement on the proper way to fix this so I am hoping we can arrive at a solution through an open discussion. The main issue at hand is with the Event/Callback mechanism we use to call clientIF.contEIOVMs. According to my experiments and this online discussion [3] weakref does not work for instance methods such as clientIF.contEIOVMs. Our Event class uses weakref to prevent it from holding references to registered callback functions. I see two easy ways to fix the regression: 1) Treat clientIF as a singleton class (which it is) and make contEIOVMs a module-level method which gets the clientIF instance and calls it's bound contEIOVMs method. See my patches [2] for the code behind this idea. 2) Allow Event to maintain a strong reference on the bound clientIF.contEIOVMs method. This will allow the current code to work as designed but will change the Event implementation to accomodate this specific use case. Since no one else appears to be using this code, it should have no functional impact. Are there any other ideas I'm missing? I am aware of plans to refactor this code for 3.6 but I am more interested in a short-term, practical solution to address the current regression. Thanks for offering your insight on this problem. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1157421 [2] http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:bu… -- Adam Litke

2 5

Transactions handling in commands
by Yair Zaslavsky 21 Nov '14

21 Nov '14

Hello all, During some chasing after some locks issues, I have encountered some commands that are not using the transaction framework properly (IMHO). For example, ExportRepoImageCommand - has no annotation indicating it should not be run in transaction. It contains IMHO a classic flow of - a. get from db (and potential update) b. send a call to VDSM -> this may be long !!! c. update results in DB. IMHO, ExportRepoImageCommand should be annotated with @NonTransactive(forceCompensation=true) and should include compensation code. If I am wrong with the specific command, please add a nice comment indicating why this was not done, so the next person who reads the code and deals with locks, transactions and all these interesting issues should be aware. Transactions should be kept as short as possible, as they are an expensive , limited resource. This is not the first time I encountered this issue with developers in the last weeks, and maybe I sound angry or something - this is not the intention, I can't say I know all engine code myself (and even around the infra side), and I think that one of the good things in the way we work is we can learn from each other. If any of of you feel like they need some "reminding" on our transactions framework (and yes, I believe there is room to improvement in it, I already spoke with Oved about at least one idea) - feel free to address me, with enough audience I might even run a session when I visit TLV (I would have done it form home, but my internet condition is pretty bad, I'm afraid). Cheers, Yair

1 0