Ensure processes death by terminating decorator - https://gerrit.ovirt.org/51407
by Yaniv Bronheim
Hi guys,
Following the work to omit deathSignal attribute from our cpopen
implementation we posted https://gerrit.ovirt.org/51407 which is ready for
use.
Currently locations that should use it are:
(I wrote above who I expect to check the area and post a patch for that -
we'll discuss it during next vdsm-sync to follow the work)
shavivi:
vdsm/v2v.py - in _start_virt_v2v you return aysnProc that should call
kill() on fail
fromani:
vdsm_hooks/checkimages/before_vm_start.py - in checkImage - the code looks
ok, but check if not better to use the terminating decorator.. I think it
will be nicer
nsoffer:
vdsm/storage/mount.py - good looks ok, I prefer to use terminator there
vdsm/storage/iscsiadm.py
vdsm/storage/imageSharing.py
vdsm/storage/hba.py -good handling, use terminator
vdsm/storage/blockSD.py
please check your usage with the returned process and see that you're not
depending on deathSignal for it to die properly on crush
some places define deathSignal for no reason, the call is sync - please
remove those places:
nsoffer:
lib/vdsm/qemuimg.py
vdsm/storage/curlImgWrap.py
vdsm/storage/storage_mailbox.py
vdsm/storage/misc.py
fromani:
lib/vdsm/virtsparsify.py
ybronhei:
vdsm/API.py
If you can't get to it in a reasonable time, add the task to the list [1]
and someone else will be it up.
Please try to go over before the sync call.
[1] -
https://docs.google.com/spreadsheets/d/180F-C1jU54ajUn7TuR-NwrKRZY1IiZI1Z...
--
*Yaniv Bronhaim.*
8 years, 2 months
Re: [ovirt-devel] No more "vds group" in master code , instead please use "cluster"
by Eli Mesika
Hi all
Please note that there is still a problem on engine-setup I have already a fixing draft [1]
Meanwhile, until this is verified and merged please do before engine-setup from dbscripts/upgrade dir
psql -U engine -f 04_00_0210_add_vds_group_view.sql <db>
Thanks and sorry for that, this will be fixed ASAP and reported in this thread.
Eli
[1] https://gerrit.ovirt.org/#/c/52454/
----- Original Message -----
> From: "Idan Shaby" <ishaby(a)redhat.com>
> To: "Eli Mesika" <emesika(a)redhat.com>
> Sent: Tuesday, January 19, 2016 10:10:17 AM
> Subject: Re: [ovirt-devel] No more "vds group" in master code , instead please use "cluster"
>
> Hi Eli,
>
> I've fetched and rebased master, and got this error while running
> engine-setup:
>
> [ INFO ] Stage: Setup validation
> [WARNING] Cannot validate host name settings, reason: resolved host does
> not match any of the local addresses
> [ ERROR ] Failed to execute stage 'Setup validation': relation "vds_groups"
> does not exist LINE 2: SELECT compatibility_version FROM
> vds_groups... ^
> [ INFO ] Stage: Clean up
> Log file is located at
> /home/ishaby/ovirt-engine/var/log/ovirt-engine/setup/ovirt-engine-setup-20160119100829-tebscp.log
> [ INFO ] Generating answer file
> '/home/ishaby/ovirt-engine/var/lib/ovirt-engine/setup/answers/20160119100830-setup.conf'
> [ INFO ] Stage: Pre-termination
> [ INFO ] Stage: Termination
> [ ERROR ] Execution of setup failed
>
> Any idea?
>
>
> Regards,
> Idan
>
> On Sun, Jan 17, 2016 at 4:50 PM, Eli Mesika <emesika(a)redhat.com> wrote:
>
> >
> >
> > ----- Original Message -----
> > > From: "Eli Mesika" <emesika(a)redhat.com>
> > > To: "Jakub Niedermertl" <jniederm(a)redhat.com>
> > > Cc: "devel" <devel(a)ovirt.org>
> > > Sent: Sunday, January 17, 2016 1:32:27 PM
> > > Subject: Re: [ovirt-devel] No more "vds group" in master code , instead
> > please use "cluster"
> > >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Jakub Niedermertl" <jniederm(a)redhat.com>
> > > > To: "Eli Mesika" <emesika(a)redhat.com>
> > > > Cc: "devel" <devel(a)ovirt.org>
> > > > Sent: Thursday, January 14, 2016 7:48:12 PM
> > > > Subject: Re: [ovirt-devel] No more "vds group" in master code ,
> > instead
> > > > please use "cluster"
> > > >
> > > > Hi Eli,
> > > >
> > > > thank you, it helped. Now `engine-setup` works fine. However engine
> > itself
> > > > reports following in a few seconds after start:
> > > >
> > > > ERROR: relation "clusters_storage_domain" does not exist
> > > >
> > > > Relevant part of engine log: http://pastebin.test.redhat.com/340672
> > > > Tested on current master (commit 8561bb69560686d0) and clean db.
> > >
> > > You are right, sorry for that ...
> > > This patch fixes that issue
> > > https://gerrit.ovirt.org/#/c/51929
> >
> > Patch merged please re-base on master and try again
> >
> >
> > >
> > > >
> > > > ----- Original Message -----
> > > > > From: "Eli Mesika" <emesika(a)redhat.com>
> > > > > To: "Jakub Niedermertl" <jniederm(a)redhat.com>
> > > > > Cc: "devel" <devel(a)ovirt.org>
> > > > > Sent: Thursday, January 14, 2016 1:10:42 AM
> > > > > Subject: Re: [ovirt-devel] No more "vds group" in master code ,
> > instead
> > > > > please use "cluster"
> > > > >
> > > > >
> > > > >
> > > > > ----- Original Message -----
> > > > > > From: "Jakub Niedermertl" <jniederm(a)redhat.com>
> > > > > > To: "Eli Mesika" <emesika(a)redhat.com>
> > > > > > Cc: "devel" <devel(a)ovirt.org>
> > > > > > Sent: Wednesday, January 13, 2016 9:18:57 PM
> > > > > > Subject: Re: [ovirt-devel] No more "vds group" in master code ,
> > instead
> > > > > > please use "cluster"
> > > > > >
> > > > > > Hi Eli,
> > > > > >
> > > > > > are db migration scripts supposed to handle this change? My
> > > > > > `engine-setup`
> > > > > > failed with following message in log:
> > > > >
> > > > > Hi Jakub
> > > > >
> > > > > Please re-base on latest master, there was an issue with that and it
> > was
> > > > > solved in https://gerrit.ovirt.org/#/c/51784/
> > > > >
> > > > >
> > > > > >
> > > > > > """
> > > > > > 2016-01-13 20:02:57 DEBUG
> > > > > > otopi.ovirt_**FILTERED**_setup.**FILTERED**_common.database
> > > > > > database.execute:172 Database: 'None', Statement: '
> > > > > > SELECT compatibility_version FROM cluster;;
> > > > > > ', args: {}
> > > > > > 2016-01-13 20:02:57 DEBUG
> > > > > > otopi.ovirt_**FILTERED**_setup.**FILTERED**_common.database
> > > > > > database.execute:177 Creating own connection
> > > > > > 2016-01-13 20:02:57 DEBUG otopi.context context._executeMethod:156
> > > > > > method
> > > > > > exception
> > > > > > Traceback (most recent call last):
> > > > > > File "/usr/lib/python2.7/site-packages/otopi/context.py", line
> > 146,
> > > > > > in
> > > > > > _executeMethod
> > > > > > method['method']()
> > > > > > File
> > > > > >
> > "/home/jakub/target/share/ovirt-**FILTERED**/setup/bin/../plugins/ovirt-**FILTERED**-setup/ovirt-**FILTERED**/db/schema.py",
> > > > > > line 218, in _validation
> > > > > > self._checkSupportedVersionsPresent()
> > > > > > File
> > > > > >
> > "/home/jakub/target/share/ovirt-**FILTERED**/setup/bin/../plugins/ovirt-**FILTERED**-setup/ovirt-**FILTERED**/db/schema.py",
> > > > > > line 180, in _checkSupportedVersionsPresent
> > > > > > transaction=False,
> > > > > > File
> > > > > >
> > "/home/jakub/target/share/ovirt-**FILTERED**/setup/ovirt_**FILTERED**_setup/**FILTERED**_common/database.py",
> > > > > > line 196, in execute
> > > > > > args,
> > > > > > ProgrammingError: relation "cluster" does not exist
> > > > > > LINE 2: SELECT compatibility_version FROM cluster;;
> > > > > > ^
> > > > > >
> > > > > > 2016-01-13 20:02:57 ERROR otopi.context context._executeMethod:165
> > > > > > Failed
> > > > > > to
> > > > > > execute stage 'Setup validation': relation "cluster" does not exist
> > > > > > LINE 2: SELECT compatibility_version FROM cluster;;
> > > > > > ^
> > > > > > """
> > > > > >
> > > > > > Thank you for answer.
> > > > > >
> > > > > > Best regards
> > > > > > Jakub
> > > > > >
> > > > > >
> > > > > > ----- Original Message -----
> > > > > > > From: "Eli Mesika" <emesika(a)redhat.com>
> > > > > > > To: "devel" <devel(a)ovirt.org>
> > > > > > > Sent: Wednesday, January 13, 2016 12:00:50 AM
> > > > > > > Subject: [ovirt-devel] No more "vds group" in master code ,
> > instead
> > > > > > > please
> > > > > > > use "cluster"
> > > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > I have merged today this patch[1] to master.
> > > > > > >
> > > > > > > The code from historical reasons uses both "vds group" and
> > "cluster"
> > > > > > > for
> > > > > > > a
> > > > > > > cluster entity.
> > > > > > > This makes the code not-clear, non-readable and hard for
> > beginners
> > > > > > > (to
> > > > > > > find
> > > > > > > for example SPs that handle clusters , or all code related to a
> > > > > > > cluster
> > > > > > > operation)
> > > > > > > This also makes our logging and error messages using sometimes
> > "vds
> > > > > > > group"
> > > > > > > and sometimes "cluster" to relate to the same entity.
> > > > > > > Worse than that, new code written sometimes introduce a mix of
> > both
> > > > > > > terms.
> > > > > > >
> > > > > > > Patch[1] renames "vds group" to cluster all over the code.
> > > > > > > This renaming covers all engine code including
> > > > > > > Class names
> > > > > > > Variables
> > > > > > > Comments
> > > > > > > Logging
> > > > > > > Error messages
> > > > > > > Database tables,views, columns and SPs including all kinds of
> > keys
> > > > > > > and
> > > > > > > constrains
> > > > > > >
> > > > > > > Please do not use from now on the term "vds group" (all its
> > variants
> > > > > > > (VdsGroup, vdsGroup, vds_group etc.)
> > > > > > > Instead , cluster and all its variants should be used
> > > > > > >
> > > > > > > If you have some written code that is not merged yet, you will
> > > > > > > probably
> > > > > > > have
> > > > > > > a little work on rebase on top this patch, as far as I know those
> > > > > > > should
> > > > > > > be
> > > > > > > trivial patches and if you have any question, please ask.
> > > > > > >
> > > > > > > Possible affects on other products are minor and were
> > communicated to
> > > > > > > the
> > > > > > > relevant product people.
> > > > > > >
> > > > > > > [1] https://gerrit.ovirt.org/#/c/51109/
> > > > > > >
> > > > > > > Thanks
> > > > > > > Eli Mesika
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > Devel mailing list
> > > > > > > Devel(a)ovirt.org
> > > > > > > http://lists.ovirt.org/mailman/listinfo/devel
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > _______________________________________________
> > > Devel mailing list
> > > Devel(a)ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/devel
> > >
> > _______________________________________________
> > Devel mailing list
> > Devel(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
> >
>
8 years, 2 months
Installing vdsm? Watch your weight.
by Fabian Deutsch
Hey,
recently I looked at how many (indirect) dependencies vdsm is pulling in.
I did the comparisong using the Node Next squashfs image.
What Node Next is doing, it uses the CentOS 7 @core group, and
installs vdsm on-top of this.
The conclusion is:
without-vdsm with-vdsm Ratio
Size of squashfs [MB] 292072 450384 1.542
Package count 297 604 2.034
Disk usage [MB] 839596 1382576 1.647
(Without vdsm: @core group + lvm + efi bits)
So we see that vdsm doubles the amount of packages, and increases the
disk-space requirements by 64%.
Two take aways: CentOS @core could be smaller, and we can take another
look at vdsm's direct and indirect dependencies.
- fabian
8 years, 2 months
Vdsm leak in multiprocessing / supervdsm interaction
by Milan Zamazal
FYI, when I performed my simple Vdsm leak test some time ago, I've found
a leak in Python multiprocessing module, where
multiprocessing.util._afterfork_registry dictionary grows on each
supervdsm call. It's not much dramatic from the point of memory usage
but maybe the possibly large dictionary can cause other performance
problems.
It's better to have this fixed and it looks like a Python bug, reported
here: https://bugs.python.org/issue26180.
8 years, 2 months
[RFC] Proposal for dropping FC22 jenkins tests on master branch
by Sandro Bonazzola
Hi,
can we drop FC22 testing in jenkins now that FC23 jobs are up and running?
it will reduce jenkins load. If needed we can keep FC22 builds, just
dropping the check jobs.
Comments?
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
8 years, 2 months
Missing admin permissions.
by Alexander Wels
Hi,
Somewhere during master upgrades somehow my admin@internal did not get
permissions to create VMs. Its is compiling about no permissions to assign a
CPU profile. Its been a while since I tried creating a VM. Can any one point me
to what is missing and how to fix it?
This is the error in the log:
2016-01-18 15:59:46,843 INFO [org.ovirt.engine.core.bll.AddVmCommand]
(default task-56) [a6fd12b] Lock Acquired to object 'EngineLock:
{exclusiveLocks='[xxxx=<VM_NAME, ACTION_TYPE_FAILED_OBJECT_LOCKED>]',
sharedLocks='[00000000-0000-0000-0000-000000000000=<TEMPLATE,
ACTION_TYPE_FAILED_TEMPLATE_IS_USED_FOR_CREATE_VM$VmName xxxx>]'}'
2016-01-18 15:59:46,893 WARN [org.ovirt.engine.core.bll.AddVmCommand]
(default task-56) [] Validation of action 'AddVm' failed for user
admin@internal. Reasons:
VAR__ACTION__ADD,VAR__TYPE__VM,ACTION_TYPE_NO_PERMISSION_TO_ASSIGN_CPU_PROFILE,
$cpuProfileId f9a05b39-9f57-4655-aab2-2846fe6519f6,$cpuProfileName DEV35
2016-01-18 15:59:46,894 INFO [org.ovirt.engine.core.bll.AddVmCommand]
(default task-56) [] Lock freed to object 'EngineLock:
{exclusiveLocks='[xxxx=<VM_NAME, ACTION_TYPE_FAILED_OBJECT_LOCKED>]',
sharedLocks='[00000000-0000-0000-0000-000000000000=<TEMPLATE,
ACTION_TYPE_FAILED_TEMPLATE_IS_USED_FOR_CREATE_VM$VmName xxxx>]'}'
8 years, 2 months
[ATN] RESTAPI specification moving to a separate git repository
by Juan Hernández
Hello,
The specification of the RESTAPI (a.k.a. the model) and the tools that
process it (a.k.a. the metamodel) have been moved to separate git
repositories:
Model:
git clone https://gerrit.ovirt.org/ovirt-engine-api-model
https://github.com/oVirt/ovirt-engine-api-metamodel
Metamodel:
git clone https://gerrit.ovirt.org/ovirt-engine-api-metamodel
https://github.com/oVirt/ovirt-engine-api-model
Currently I'm manually keeping these repositories in sync with the
ovirt-engine repository, but I will very soon remove all this code from
the ovirt-engine repository:
restapi: Move model to separate repository
https://gerrit.ovirt.org/51519
This means that once the above patch is merged, which will be very soon,
probably this week, when you need to do changes to the specification of
the RESTAPI, you will need first to prepare a patch for the
ovirt-engine-api-model, submit it, review it, etc. Once that patch is
merged I will do a new release of the model. Then you will need to
update the root POM of the engine to use the new version of the model,
for example:
- <model.version>4.0.1</model.version>
+ <model.version>4.0.2</model.version>
And adjust the engine to work with the new version of the model.
As you will probably want to do these changes and test them before
publishing anything, or asking for any review, I'd suggest the following
process:
1. Checkout the model project:
$ git clone git://gerrit.ovirt.org/ovirt-engine-api-model
2. Check the version number in the root POM. It should be a SNAPSHOT
version, unless you explicitly checkout from a tag. For example,
currently it is 4.0.2-SNAPSHOT.
3. Make your changes to the model, and then install it to your local
Maven repository:
$ mvn clean install
4. Add to your $HOME/.m2/settings.xml a profile that is activated
automatically and that changes the value of the "model.version" property
used by the engine:
<activeProfiles>
<activeProfile>myprofile</activeProfile>
</activeProfiles>
<profiles>
<profile>
<id>myprofile</id>
<properties>
<model.version>4.0.2-SNAPSHOT</model.version>
</properties>
</profile>
</profiles>
5. Make your changes to the engine, and build it as usual, it will your
modified version of the model.
6. Publish/review the changes to the model.
7. Wait for the new model release.
8. Publish/review the changes to the engine, including the change of the
model version in the root POM. Note that you can publish/review these
changes without waiting for the new model release, but the Jenkins test
will fail.
Note also that changes to the model will need to be properly documented
in order to be accepted. There are some instructions on how to document
the model here:
https://github.com/oVirt/ovirt-engine-api-model/blob/master/README.md
All these changes affect only the master branch, nothing changed in
these regards in the 3.6 branch.
Regards,
Juan Hernandez
--
Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
3ºD, 28016 Madrid, Spain
Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.
8 years, 2 months
Alpha version of the Ruby SDK ready for use
by Juan Hernández
Hello,
Version 4 of oVirt will include a new Ruby SDK. It is currently in
"alpha" status, but I think it is already usable. It is available from
"rubygems.org":
https://rubygems.org/gems/ovirt-engine-sdk
You can install it as usual:
$ gem install ovirt-engine-sdk
There are examples of how to use it in the source code itself:
https://github.com/oVirt/ovirt-engine-sdk-ruby/blob/master/sdk/README.md
https://github.com/oVirt/ovirt-engine-sdk-ruby/tree/master/sdk/examples
Any feedback is welcome, via this list, reporting bugs:
https://bugzilla.redhat.com/enter_bug.cgi?product=ovirt-engine-sdk-ruby
Or sending patches to gerrit:
$ git clone git://gerrit.ovirt.org/ovirt-engine-sdk-ruby.git
Regards,
Juan Hernandez
--
Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
3ºD, 28016 Madrid, Spain
Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.
8 years, 3 months
Vdsm sync 19/1
by Yaniv Bronheim
Hi all,
(Participated: Piotr, Francesco, Adam, Nir, Edy and I)
We discussed about the following topics:
Python3 - terminating contextmanager is ready
https://gerrit.ovirt.org/51407 - this allows us to remove deathSignal
usages and safely kill processes on failures.
Francesco already covered one location (https://gerrit.ovirt.org/#/c/52349/)
this week I hope we'll *start* the work to cover the rest of the code as
follow:
vdsm/storage/mount.py
vdsm/storage/iscsiadm.py
vdsm/storage/imageSharing.py
vdsm/storage/hba.py
vdsm/storage/blockSD.py
vdsm/v2v.py
If any gaps or dependencies are raised I asked to state it in python3 work
sheet (
https://docs.google.com/spreadsheets/d/180F-C1jU54ajUn7TuR-NwrKRZY1IiZI1Z...)
and we'll discuss it in next call.
next will be to remove all deathSignal usages - currently there are
redundant place where its being used that can be removed today:
lib/vdsm/qemuimg.py
vdsm/storage/curlImgWrap.py
vdsm/storage/storage_mailbox.py
vdsm/storage/misc.py
vdsm/API.py
Hopefully patches for that will be posted during the week.
Next step will be to use Popen where cpopen is not available (
https://gerrit.ovirt.org/48384) and we'll move forward with migrate more
tests to python3.
- Vdsm communicate - we did a little summary about what was explored and
what we want to have. It was only a discussion to see where we're heading.
External Broker - In the past we checked Cupid (IMQP) and ActiveMQ (STOMP).
In cupid we found many bugs and gaps, we thought about using only the
client of it for point to point communication, but then found it too
complex. actionMQ is in java and found to be too slow and consume a lot of
memory so we thought about having only one instance in cluster, or maybe
run it in external host that doesn't run vdsm, or put it with the engine
host - for that we will need to change all "host lifecycle" that we have
today so we left it as well (piotr, fill me up if I missed something)
We have implementation for "mini borker" as part of vdsm which perform what
we need to run it as external process to vdsm and forward messages to
clients. there are alternatives as ZeroMQ that we can explore.
Bottom line - we want to approve the communicate inside the host, between
vdsm to other services such as supervdsm, mom and maybe later we'll split
the current implementation of vdsm to more services that will run in
parallel (such as vm monitoring, vdsm-storage and so on. For that we can
use dbus, multiprocessing(uds), or some kind of a broker.
This leaded us to talk about service separation which we want to design
soon.
- Vdsm contract - Piotr sent yaml schema plan in
https://gerrit.ovirt.org/#/c/52404 - please ack that you agree with the
concept. Piotr moves on with that and start to migrate all verbs to that
form. For next version we shell have both types of schema available, and
start to add new verbs and events structures in the new yaml format.
- Exceptions - Nir raises that we should define Virt specific exceptions -
Francesco can elaborate about the plans next week.
- Network updates - Edy checks how to improve network configuration time.
Currently vdsm modifies ifcfg files, which after changing them it takes
time for the update to catch.
Interesting call guys, I encourage more developers to participate, listen
and influence vdsm 4.0 directions.
See you next week.
--
*Yaniv Bronhaim.*
8 years, 3 months
vdsm] tag and branch ovirt 3.6.2 RC3
by Francesco Romani
Hi,
Sorry for the short notice of this email.
the state of branch ovirt-3.6 is as follows
tag sha subject bug target release
----------------------------------------------------------------------------------------------------------------------
de54e59 virt: Don't expose GuestAgent.guestInfo directly 1295428 3.6.3
83c4ac8 supervdsm: failed validateAccess leaves pipes open 1271575 3.6.3
37bda50 vm: safer handling of conf in restore 1296936 3.6.3
57b1814 virt: safer handling of migration parameters 1296936 3.6.3
9b29ddd lun: Serial attr should not passed to libvirt for lun disks. 1291930 3.6.2 <<<<
7b94531 migration: use context manager for semaphore 1296936 3.6.3
508b2fd virt: Correct epoll unregistration usage in vmchannels 1226911 3.6.3
68a1b69 virt: Set cloexec flag on channel sockets 1226911 3.6.3
4.17.17 505026d spec: require libvirt to fix hotplugging
So, for the next tag the plan is to
1. branch out ovirt-3.6.2 from tag 4.17.17. This branch is expected to be short-lived (basically only for this RC).
2. apply commit 9b29ddd only
3. tag 4.17.18 from branch ovirt-3.6.2
I see we have only two patches in queue for ovirt-3.6
https://gerrit.ovirt.org/#/c/52346/
https://gerrit.ovirt.org/#/c/52348/
Do we need them in this build?
Should we require another 3.6.2 build, backports will be needed ALSO on branch ovirt-3.6.2.
Will start implement the plan before lunch if noone objects.
Thanks
--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
8 years, 3 months
