January 2016 - Devel - oVirt List Archives

Ensure processes death by terminating decorator - https://gerrit.ovirt.org/51407
by Yaniv Bronheim 25 Jan '16

25 Jan '16

Hi guys, Following the work to omit deathSignal attribute from our cpopen implementation we posted https://gerrit.ovirt.org/51407 which is ready for use. Currently locations that should use it are: (I wrote above who I expect to check the area and post a patch for that - we'll discuss it during next vdsm-sync to follow the work) shavivi: vdsm/v2v.py - in _start_virt_v2v you return aysnProc that should call kill() on fail fromani: vdsm_hooks/checkimages/before_vm_start.py - in checkImage - the code looks ok, but check if not better to use the terminating decorator.. I think it will be nicer nsoffer: vdsm/storage/mount.py - good looks ok, I prefer to use terminator there vdsm/storage/iscsiadm.py vdsm/storage/imageSharing.py vdsm/storage/hba.py -good handling, use terminator vdsm/storage/blockSD.py please check your usage with the returned process and see that you're not depending on deathSignal for it to die properly on crush some places define deathSignal for no reason, the call is sync - please remove those places: nsoffer: lib/vdsm/qemuimg.py vdsm/storage/curlImgWrap.py vdsm/storage/storage_mailbox.py vdsm/storage/misc.py fromani: lib/vdsm/virtsparsify.py ybronhei: vdsm/API.py If you can't get to it in a reasonable time, add the task to the list [1] and someone else will be it up. Please try to go over before the sync call. [1] - https://docs.google.com/spreadsheets/d/180F-C1jU54ajUn7TuR-NwrKRZY1IiZI1Z8U… -- *Yaniv Bronhaim.*

3 4

Re: [ovirt-devel] No more "vds group" in master code , instead please use "cluster"
by Eli Mesika 24 Jan '16

24 Jan '16

Hi all Please note that there is still a problem on engine-setup I have already a fixing draft [1] Meanwhile, until this is verified and merged please do before engine-setup from dbscripts/upgrade dir psql -U engine -f 04_00_0210_add_vds_group_view.sql <db> Thanks and sorry for that, this will be fixed ASAP and reported in this thread. Eli [1] https://gerrit.ovirt.org/#/c/52454/ ----- Original Message ----- > From: "Idan Shaby" <ishaby(a)redhat.com> > To: "Eli Mesika" <emesika(a)redhat.com> > Sent: Tuesday, January 19, 2016 10:10:17 AM > Subject: Re: [ovirt-devel] No more "vds group" in master code , instead please use "cluster" > > Hi Eli, > > I've fetched and rebased master, and got this error while running > engine-setup: > > [ INFO ] Stage: Setup validation > [WARNING] Cannot validate host name settings, reason: resolved host does > not match any of the local addresses > [ ERROR ] Failed to execute stage 'Setup validation': relation "vds_groups" > does not exist LINE 2: SELECT compatibility_version FROM > vds_groups... ^ > [ INFO ] Stage: Clean up > Log file is located at > /home/ishaby/ovirt-engine/var/log/ovirt-engine/setup/ovirt-engine-setup-20160119100829-tebscp.log > [ INFO ] Generating answer file > '/home/ishaby/ovirt-engine/var/lib/ovirt-engine/setup/answers/20160119100830-setup.conf' > [ INFO ] Stage: Pre-termination > [ INFO ] Stage: Termination > [ ERROR ] Execution of setup failed > > Any idea? > > > Regards, > Idan > > On Sun, Jan 17, 2016 at 4:50 PM, Eli Mesika <emesika(a)redhat.com> wrote: > > > > > > > ----- Original Message ----- > > > From: "Eli Mesika" <emesika(a)redhat.com> > > > To: "Jakub Niedermertl" <jniederm(a)redhat.com> > > > Cc: "devel" <devel(a)ovirt.org> > > > Sent: Sunday, January 17, 2016 1:32:27 PM > > > Subject: Re: [ovirt-devel] No more "vds group" in master code , instead > > please use "cluster" > > > > > > > > > > > > ----- Original Message ----- > > > > From: "Jakub Niedermertl" <jniederm(a)redhat.com> > > > > To: "Eli Mesika" <emesika(a)redhat.com> > > > > Cc: "devel" <devel(a)ovirt.org> > > > > Sent: Thursday, January 14, 2016 7:48:12 PM > > > > Subject: Re: [ovirt-devel] No more "vds group" in master code , > > instead > > > > please use "cluster" > > > > > > > > Hi Eli, > > > > > > > > thank you, it helped. Now `engine-setup` works fine. However engine > > itself > > > > reports following in a few seconds after start: > > > > > > > > ERROR: relation "clusters_storage_domain" does not exist > > > > > > > > Relevant part of engine log: http://pastebin.test.redhat.com/340672 > > > > Tested on current master (commit 8561bb69560686d0) and clean db. > > > > > > You are right, sorry for that ... > > > This patch fixes that issue > > > https://gerrit.ovirt.org/#/c/51929 > > > > Patch merged please re-base on master and try again > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > From: "Eli Mesika" <emesika(a)redhat.com> > > > > > To: "Jakub Niedermertl" <jniederm(a)redhat.com> > > > > > Cc: "devel" <devel(a)ovirt.org> > > > > > Sent: Thursday, January 14, 2016 1:10:42 AM > > > > > Subject: Re: [ovirt-devel] No more "vds group" in master code , > > instead > > > > > please use "cluster" > > > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > From: "Jakub Niedermertl" <jniederm(a)redhat.com> > > > > > > To: "Eli Mesika" <emesika(a)redhat.com> > > > > > > Cc: "devel" <devel(a)ovirt.org> > > > > > > Sent: Wednesday, January 13, 2016 9:18:57 PM > > > > > > Subject: Re: [ovirt-devel] No more "vds group" in master code , > > instead > > > > > > please use "cluster" > > > > > > > > > > > > Hi Eli, > > > > > > > > > > > > are db migration scripts supposed to handle this change? My > > > > > > `engine-setup` > > > > > > failed with following message in log: > > > > > > > > > > Hi Jakub > > > > > > > > > > Please re-base on latest master, there was an issue with that and it > > was > > > > > solved in https://gerrit.ovirt.org/#/c/51784/ > > > > > > > > > > > > > > > > > > > > > > """ > > > > > > 2016-01-13 20:02:57 DEBUG > > > > > > otopi.ovirt_**FILTERED**_setup.**FILTERED**_common.database > > > > > > database.execute:172 Database: 'None', Statement: ' > > > > > > SELECT compatibility_version FROM cluster;; > > > > > > ', args: {} > > > > > > 2016-01-13 20:02:57 DEBUG > > > > > > otopi.ovirt_**FILTERED**_setup.**FILTERED**_common.database > > > > > > database.execute:177 Creating own connection > > > > > > 2016-01-13 20:02:57 DEBUG otopi.context context._executeMethod:156 > > > > > > method > > > > > > exception > > > > > > Traceback (most recent call last): > > > > > > File "/usr/lib/python2.7/site-packages/otopi/context.py", line > > 146, > > > > > > in > > > > > > _executeMethod > > > > > > method['method']() > > > > > > File > > > > > > > > "/home/jakub/target/share/ovirt-**FILTERED**/setup/bin/../plugins/ovirt-**FILTERED**-setup/ovirt-**FILTERED**/db/schema.py", > > > > > > line 218, in _validation > > > > > > self._checkSupportedVersionsPresent() > > > > > > File > > > > > > > > "/home/jakub/target/share/ovirt-**FILTERED**/setup/bin/../plugins/ovirt-**FILTERED**-setup/ovirt-**FILTERED**/db/schema.py", > > > > > > line 180, in _checkSupportedVersionsPresent > > > > > > transaction=False, > > > > > > File > > > > > > > > "/home/jakub/target/share/ovirt-**FILTERED**/setup/ovirt_**FILTERED**_setup/**FILTERED**_common/database.py", > > > > > > line 196, in execute > > > > > > args, > > > > > > ProgrammingError: relation "cluster" does not exist > > > > > > LINE 2: SELECT compatibility_version FROM cluster;; > > > > > > ^ > > > > > > > > > > > > 2016-01-13 20:02:57 ERROR otopi.context context._executeMethod:165 > > > > > > Failed > > > > > > to > > > > > > execute stage 'Setup validation': relation "cluster" does not exist > > > > > > LINE 2: SELECT compatibility_version FROM cluster;; > > > > > > ^ > > > > > > """ > > > > > > > > > > > > Thank you for answer. > > > > > > > > > > > > Best regards > > > > > > Jakub > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > From: "Eli Mesika" <emesika(a)redhat.com> > > > > > > > To: "devel" <devel(a)ovirt.org> > > > > > > > Sent: Wednesday, January 13, 2016 12:00:50 AM > > > > > > > Subject: [ovirt-devel] No more "vds group" in master code , > > instead > > > > > > > please > > > > > > > use "cluster" > > > > > > > > > > > > > > Hi > > > > > > > > > > > > > > I have merged today this patch[1] to master. > > > > > > > > > > > > > > The code from historical reasons uses both "vds group" and > > "cluster" > > > > > > > for > > > > > > > a > > > > > > > cluster entity. > > > > > > > This makes the code not-clear, non-readable and hard for > > beginners > > > > > > > (to > > > > > > > find > > > > > > > for example SPs that handle clusters , or all code related to a > > > > > > > cluster > > > > > > > operation) > > > > > > > This also makes our logging and error messages using sometimes > > "vds > > > > > > > group" > > > > > > > and sometimes "cluster" to relate to the same entity. > > > > > > > Worse than that, new code written sometimes introduce a mix of > > both > > > > > > > terms. > > > > > > > > > > > > > > Patch[1] renames "vds group" to cluster all over the code. > > > > > > > This renaming covers all engine code including > > > > > > > Class names > > > > > > > Variables > > > > > > > Comments > > > > > > > Logging > > > > > > > Error messages > > > > > > > Database tables,views, columns and SPs including all kinds of > > keys > > > > > > > and > > > > > > > constrains > > > > > > > > > > > > > > Please do not use from now on the term "vds group" (all its > > variants > > > > > > > (VdsGroup, vdsGroup, vds_group etc.) > > > > > > > Instead , cluster and all its variants should be used > > > > > > > > > > > > > > If you have some written code that is not merged yet, you will > > > > > > > probably > > > > > > > have > > > > > > > a little work on rebase on top this patch, as far as I know those > > > > > > > should > > > > > > > be > > > > > > > trivial patches and if you have any question, please ask. > > > > > > > > > > > > > > Possible affects on other products are minor and were > > communicated to > > > > > > > the > > > > > > > relevant product people. > > > > > > > > > > > > > > [1] https://gerrit.ovirt.org/#/c/51109/ > > > > > > > > > > > > > > Thanks > > > > > > > Eli Mesika > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Devel mailing list > > > > > > > Devel(a)ovirt.org > > > > > > > http://lists.ovirt.org/mailman/listinfo/devel > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Devel mailing list > > > Devel(a)ovirt.org > > > http://lists.ovirt.org/mailman/listinfo/devel > > > > > _______________________________________________ > > Devel mailing list > > Devel(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/devel > > >

1 1

Installing vdsm? Watch your weight.
by Fabian Deutsch 23 Jan '16

23 Jan '16

Hey, recently I looked at how many (indirect) dependencies vdsm is pulling in. I did the comparisong using the Node Next squashfs image. What Node Next is doing, it uses the CentOS 7 @core group, and installs vdsm on-top of this. The conclusion is: without-vdsm with-vdsm Ratio Size of squashfs [MB] 292072 450384 1.542 Package count 297 604 2.034 Disk usage [MB] 839596 1382576 1.647 (Without vdsm: @core group + lvm + efi bits) So we see that vdsm doubles the amount of packages, and increases the disk-space requirements by 64%. Two take aways: CentOS @core could be smaller, and we can take another look at vdsm's direct and indirect dependencies. - fabian

2 1

Vdsm leak in multiprocessing / supervdsm interaction
by Milan Zamazal 22 Jan '16

22 Jan '16

FYI, when I performed my simple Vdsm leak test some time ago, I've found a leak in Python multiprocessing module, where multiprocessing.util._afterfork_registry dictionary grows on each supervdsm call. It's not much dramatic from the point of memory usage but maybe the possibly large dictionary can cause other performance problems. It's better to have this fixed and it looks like a Python bug, reported here: https://bugs.python.org/issue26180.

1 0

[RFC] Proposal for dropping FC22 jenkins tests on master branch
by Sandro Bonazzola 22 Jan '16

22 Jan '16

Hi, can we drop FC22 testing in jenkins now that FC23 jobs are up and running? it will reduce jenkins load. If needed we can keep FC22 builds, just dropping the check jobs. Comments? -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

3 7

Missing admin permissions.
by Alexander Wels 21 Jan '16

21 Jan '16

Hi, Somewhere during master upgrades somehow my admin@internal did not get permissions to create VMs. Its is compiling about no permissions to assign a CPU profile. Its been a while since I tried creating a VM. Can any one point me to what is missing and how to fix it? This is the error in the log: 2016-01-18 15:59:46,843 INFO [org.ovirt.engine.core.bll.AddVmCommand] (default task-56) [a6fd12b] Lock Acquired to object 'EngineLock: {exclusiveLocks='[xxxx=<VM_NAME, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='[00000000-0000-0000-0000-000000000000=<TEMPLATE, ACTION_TYPE_FAILED_TEMPLATE_IS_USED_FOR_CREATE_VM$VmName xxxx>]'}' 2016-01-18 15:59:46,893 WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-56) [] Validation of action 'AddVm' failed for user admin@internal. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,ACTION_TYPE_NO_PERMISSION_TO_ASSIGN_CPU_PROFILE, $cpuProfileId f9a05b39-9f57-4655-aab2-2846fe6519f6,$cpuProfileName DEV35 2016-01-18 15:59:46,894 INFO [org.ovirt.engine.core.bll.AddVmCommand] (default task-56) [] Lock freed to object 'EngineLock: {exclusiveLocks='[xxxx=<VM_NAME, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='[00000000-0000-0000-0000-000000000000=<TEMPLATE, ACTION_TYPE_FAILED_TEMPLATE_IS_USED_FOR_CREATE_VM$VmName xxxx>]'}'

4 4

[ATN] RESTAPI specification moving to a separate git repository
by Juan Hernández 21 Jan '16

21 Jan '16

Hello, The specification of the RESTAPI (a.k.a. the model) and the tools that process it (a.k.a. the metamodel) have been moved to separate git repositories: Model: git clone https://gerrit.ovirt.org/ovirt-engine-api-model https://github.com/oVirt/ovirt-engine-api-metamodel Metamodel: git clone https://gerrit.ovirt.org/ovirt-engine-api-metamodel https://github.com/oVirt/ovirt-engine-api-model Currently I'm manually keeping these repositories in sync with the ovirt-engine repository, but I will very soon remove all this code from the ovirt-engine repository: restapi: Move model to separate repository https://gerrit.ovirt.org/51519 This means that once the above patch is merged, which will be very soon, probably this week, when you need to do changes to the specification of the RESTAPI, you will need first to prepare a patch for the ovirt-engine-api-model, submit it, review it, etc. Once that patch is merged I will do a new release of the model. Then you will need to update the root POM of the engine to use the new version of the model, for example: - <model.version>4.0.1</model.version> + <model.version>4.0.2</model.version> And adjust the engine to work with the new version of the model. As you will probably want to do these changes and test them before publishing anything, or asking for any review, I'd suggest the following process: 1. Checkout the model project: $ git clone git://gerrit.ovirt.org/ovirt-engine-api-model 2. Check the version number in the root POM. It should be a SNAPSHOT version, unless you explicitly checkout from a tag. For example, currently it is 4.0.2-SNAPSHOT. 3. Make your changes to the model, and then install it to your local Maven repository: $ mvn clean install 4. Add to your $HOME/.m2/settings.xml a profile that is activated automatically and that changes the value of the "model.version" property used by the engine: <activeProfiles> <activeProfile>myprofile</activeProfile> </activeProfiles> <profiles> <profile> <id>myprofile</id> <properties> <model.version>4.0.2-SNAPSHOT</model.version> </properties> </profile> </profiles> 5. Make your changes to the engine, and build it as usual, it will your modified version of the model. 6. Publish/review the changes to the model. 7. Wait for the new model release. 8. Publish/review the changes to the engine, including the change of the model version in the root POM. Note that you can publish/review these changes without waiting for the new model release, but the Jenkins test will fail. Note also that changes to the model will need to be properly documented in order to be accepted. There are some instructions on how to document the model here: https://github.com/oVirt/ovirt-engine-api-model/blob/master/README.md All these changes affect only the master branch, nothing changed in these regards in the 3.6 branch. Regards, Juan Hernandez -- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.

2 2

Alpha version of the Ruby SDK ready for use
by Juan Hernández 20 Jan '16

20 Jan '16

Hello, Version 4 of oVirt will include a new Ruby SDK. It is currently in "alpha" status, but I think it is already usable. It is available from "rubygems.org": https://rubygems.org/gems/ovirt-engine-sdk You can install it as usual: $ gem install ovirt-engine-sdk There are examples of how to use it in the source code itself: https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/sdk/README.md https://github.com/oVirt/ovirt-engine-sdk-ruby/tree/master/sdk/examples Any feedback is welcome, via this list, reporting bugs: https://bugzilla.redhat.com/enter_bug.cgi?product=ovirt-engine-sdk-ruby Or sending patches to gerrit: $ git clone git://gerrit.ovirt.org/ovirt-engine-sdk-ruby.git Regards, Juan Hernandez -- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.

1 0

Vdsm sync 19/1
by Yaniv Bronheim 20 Jan '16

20 Jan '16

Hi all, (Participated: Piotr, Francesco, Adam, Nir, Edy and I) We discussed about the following topics: Python3 - terminating contextmanager is ready https://gerrit.ovirt.org/51407 - this allows us to remove deathSignal usages and safely kill processes on failures. Francesco already covered one location (https://gerrit.ovirt.org/#/c/52349/) this week I hope we'll *start* the work to cover the rest of the code as follow: vdsm/storage/mount.py vdsm/storage/iscsiadm.py vdsm/storage/imageSharing.py vdsm/storage/hba.py vdsm/storage/blockSD.py vdsm/v2v.py If any gaps or dependencies are raised I asked to state it in python3 work sheet ( https://docs.google.com/spreadsheets/d/180F-C1jU54ajUn7TuR-NwrKRZY1IiZI1Z8U…) and we'll discuss it in next call. next will be to remove all deathSignal usages - currently there are redundant place where its being used that can be removed today: lib/vdsm/qemuimg.py vdsm/storage/curlImgWrap.py vdsm/storage/storage_mailbox.py vdsm/storage/misc.py vdsm/API.py Hopefully patches for that will be posted during the week. Next step will be to use Popen where cpopen is not available ( https://gerrit.ovirt.org/48384) and we'll move forward with migrate more tests to python3. - Vdsm communicate - we did a little summary about what was explored and what we want to have. It was only a discussion to see where we're heading. External Broker - In the past we checked Cupid (IMQP) and ActiveMQ (STOMP). In cupid we found many bugs and gaps, we thought about using only the client of it for point to point communication, but then found it too complex. actionMQ is in java and found to be too slow and consume a lot of memory so we thought about having only one instance in cluster, or maybe run it in external host that doesn't run vdsm, or put it with the engine host - for that we will need to change all "host lifecycle" that we have today so we left it as well (piotr, fill me up if I missed something) We have implementation for "mini borker" as part of vdsm which perform what we need to run it as external process to vdsm and forward messages to clients. there are alternatives as ZeroMQ that we can explore. Bottom line - we want to approve the communicate inside the host, between vdsm to other services such as supervdsm, mom and maybe later we'll split the current implementation of vdsm to more services that will run in parallel (such as vm monitoring, vdsm-storage and so on. For that we can use dbus, multiprocessing(uds), or some kind of a broker. This leaded us to talk about service separation which we want to design soon. - Vdsm contract - Piotr sent yaml schema plan in https://gerrit.ovirt.org/#/c/52404 - please ack that you agree with the concept. Piotr moves on with that and start to migrate all verbs to that form. For next version we shell have both types of schema available, and start to add new verbs and events structures in the new yaml format. - Exceptions - Nir raises that we should define Virt specific exceptions - Francesco can elaborate about the plans next week. - Network updates - Edy checks how to improve network configuration time. Currently vdsm modifies ifcfg files, which after changing them it takes time for the update to catch. Interesting call guys, I encourage more developers to participate, listen and influence vdsm 4.0 directions. See you next week. -- *Yaniv Bronhaim.*

1 0

vdsm] tag and branch ovirt 3.6.2 RC3
by Francesco Romani 19 Jan '16

19 Jan '16

Hi, Sorry for the short notice of this email. the state of branch ovirt-3.6 is as follows tag sha subject bug target release ---------------------------------------------------------------------------------------------------------------------- de54e59 virt: Don't expose GuestAgent.guestInfo directly 1295428 3.6.3 83c4ac8 supervdsm: failed validateAccess leaves pipes open 1271575 3.6.3 37bda50 vm: safer handling of conf in restore 1296936 3.6.3 57b1814 virt: safer handling of migration parameters 1296936 3.6.3 9b29ddd lun: Serial attr should not passed to libvirt for lun disks. 1291930 3.6.2 <<<< 7b94531 migration: use context manager for semaphore 1296936 3.6.3 508b2fd virt: Correct epoll unregistration usage in vmchannels 1226911 3.6.3 68a1b69 virt: Set cloexec flag on channel sockets 1226911 3.6.3 4.17.17 505026d spec: require libvirt to fix hotplugging So, for the next tag the plan is to 1. branch out ovirt-3.6.2 from tag 4.17.17. This branch is expected to be short-lived (basically only for this RC). 2. apply commit 9b29ddd only 3. tag 4.17.18 from branch ovirt-3.6.2 I see we have only two patches in queue for ovirt-3.6 https://gerrit.ovirt.org/#/c/52346/ https://gerrit.ovirt.org/#/c/52348/ Do we need them in this build? Should we require another 3.6.2 build, backports will be needed ALSO on branch ovirt-3.6.2. Will start implement the plan before lunch if noone objects. Thanks -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani

1 1