From gang.wei at intel.com  Sat Dec  1 16:47:53 2012
From: gang.wei at intel.com (Wei, Gang)
Date: Sat, 1 Dec 2012 16:47:53 +0000
Subject: [Engine-devel] Trusted Compute Pools
In-Reply-To: <50B89105.4010006@redhat.com>
References: <D0B11485C64D4B47B66902F8A4E901BE3FCDA2@SHSMSX101.ccr.corp.intel.com>
	<50B89105.4010006@redhat.com>
Message-ID: <D0B11485C64D4B47B66902F8A4E901BE421966@SHSMSX101.ccr.corp.intel.com>

Thanks a lot for such a detailed reply with good questions & suggestions.
Some comments below.

Jimmy

Itamar Heim wrote on?2012-11-30:
> On 11/20/2012 08:06 AM, Wei, Gang wrote:
>> Hi,
>> 
>> I am an engineer working in Intel Open Source Technology Center,
>> interested in integrating Intel initiated OpenAttestation(OAT) project
>> (https://github.com/OpenAttestation/OpenAttestation.git) into oVirt to
>> provide a way for Administrator to deploy VMs on trusted hosts hardened
>> with H/W-based security features, such as Intel TXT.
>> 
>> I made a draft feature page for this:
>> http://wiki.ovirt.org/wiki/Trusted_compute_pools
>> 
>> My draft idea is to provide trust_level requirement while doing vm
creation
>> like below:
>> 
>> curl -v -u "vdcadmin at qa.lab.tlv.redhat.com"
>>      -H "Content-type: application/xml"
>>      -d '<vm><name>my_new_vm</name>
>> <cluster id="99408929-82cf-4dc7-a532-9d998063fa95" />
>> <template id="00000000-0000-0000-0000-000000000000"/>
>> <trust_level>trusted</trust_level></vm>'
>>      'http://10.35.1.1/rhevm-api/vms'
>> Then oVirt Engine should query attestation server built with OAT via
RESTful
>> API to get all trusted hosts and select one to create the VM.
>> 
>> Attestation server performs host verification through following steps:
>> 1. Hosts boot with Intel TXT technology enabled
>> 2. The hosts' BIOS, hypervisor and OS are measured
> 
> a feature page on collecting bios info was just posted, please review it
> for your needs: http://wiki.ovirt.org/wiki/Features/Design/HostBiosInfo
> 
>> 3. These measured data is sent to Attestation server when challenged by
>> attestation server
> 
> does this has to be live or can be cached?

This has to be live.

>> 4. Attestation server verifies those measurements against good/known
>> database to determine hosts' trustworthiness
> 
> btw, you can do a uiplugin which adds a subtab or a field to an existing
> subtab so when standing on an intel host, user would see attestation
> status of the host from the attestation service.

Good suggestion.

>> Hosts need to be installed with OAT host agent to report host integrity
to
>> attestation server.
> 
> iiuc, this has nothing to do with vdsm, so this would be a plugin to the
> new ovirt-host-deploy host bootstrap script to install this rpm on hosts?

Yes, it has nothing to do with vdsm by far. I will look into how to update
ovirt-host-deploy host bootstrap script to install it.

>> By far, I am still in process of getting familiar with oVirt code and not
>> get solid idea yet on how the oVirt Engine should be modified to support
>> this feature.
>> 
>> Any kind of comments or suggestions will be highly appreciated.
> 
> several questions (apologies if covered in the feature page, i'm offline)
> 
> 1. is the attestation server open source and intended to be
>     packaged/deployed together with engine, or assumption is it is an
>     external service engine only needs to integrate with?
>     same question on anything needed to be deployed on host side.

Attestation server is open source, but it is expected to be deployed as a
separate service outside of ovirt engine. Ovirt engine need to integrate
with it. Host side need to deploy a host agent for attestation.

> 2. depending on first question - amount of configuration needed for
>     secure communication with the attestation service.

Yes.

I will think about your suggested approaches below throughout and then come
back to you.

> 3. I see two main approaches to implementing such a feature in engine
>     (regardless of the bundling/packaging/deployment aspects mentioned
>     above)
> a. fully integrated
> - adding a field to vm scheme with this field.
> - adding a set of config options for attestation service url and
>    authentication info
> - adding to RunVm command a method to check attestation aspects.
> 
> b. pluggable
> - this would be somewhat like the custom hooks in vdsm
> - you would add a custom property to vm for this field (no need for db
> scheme change, just a config change for this custom property. if the
> plugin is packaged with ovirt, it could be a predefined one, so user
> does not need to do anything. if the plugin is optional, then the plugin
> will configure this config key as well at deployment.
> - you can use a uiplugin to make this custom property appear as a main
> field in the vm dialog, but i think a custom property can be a good
> enough start. we have feature which are only implemented as custom
> properties (i.e., advanced options).
> - you will add an engine plugin to do the check (engine plugins do not
> exist yet).
> 
> more notes/thoughts:
> - Gilad is looking into making the scheduler pluggable and will send his
> thoughts on this. this means you could extend the scheduler with the
> check you want to do.
> - you can add a cluster level policy on allowed values for this field at
> vm level, this should also be possible via cluster level custom
> properties and plugins (when we'll have them).
> - should engine call attestation service for each host each time, or can
> just load a cache every X minutes for all hosts?
> 
> since there are several options for the final approach, i suggest you
> start with a POC based on the simplest option and check it is working,
> then hopefully the other tangents will converge or will revisit the best
> solution.
> 
> a POC patch would be a branch in your repo or the gerrit, flagged as WIP
> (work in progress) so it can be reviewed/commented/tested, but not
> merged till we revisit best approach for the feature (which may be to
> merge it and enhance at a later time as well, pending the other factors)
> 
> the POC should allow you to have a working solution based on private
> builds from the branch to test the integration.
> 
> for a POC patch i suggest: 1. use vm custom level property as simpler
> for first phase then changing the db scheme. 2. use bare minimum engine
> config support for service location/authentication. 3. assume no plugins
> and just add the code to the relevant part in RunVM which filters the
> relevant hosts:
> 
> relevant logic is in this class:
> ./backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/Vds
> S elector.java
> 
> the getVdsToRunOn[1] method has a loop to filter irrelevant hosts, then
> to choose the best host.
> sounds like you need to add a hostValidator which will call the class
> actually communicating with the attestation service (or hopefully, will
> check against the cache this class maintains).
> 
> hope this makes things a bit more clearer on the available options.
> 
> Thanks,
>     Itamar
> [1] private Guid getVdsToRunOn(Iterable<VDS> vdss, boolean isMigrate) {
>          StringBuilder sb = new StringBuilder();
>          final List<VDS> readyToRun = new ArrayList<VDS>();
>          for (VDS curVds : vdss) {
>              if (validateHostIsReadyToRun(curVds, sb, isMigrate) ==
> ValidationResult.VALID) {
>                  readyToRun.add(curVds);
>              }
>          }
> ...
>      }
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 8586 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121201/92e9d549/attachment.p7s>

From mpastern at redhat.com  Sun Dec  2 09:10:49 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Sun, 02 Dec 2012 11:10:49 +0200
Subject: [Engine-devel] [ANNOUNCE] ovirt-java-sdk
Message-ID: <50BB1B19.9070507@redhat.com>

Hi All,

I'm happy to announce the general availability of the ovirt-java-sdk,
it's yet another fully auto-generated client based on our rest-api
RSDL (RESTful Service Description Language),

More details can be found at [1], you more than welcome joining me
by contributing to this project,

And as always, further feedbacks are most welcome!

[1] http://www.ovirt.org/Java-sdk

-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From mpastern at redhat.com  Sun Dec  2 09:14:57 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Sun, 02 Dec 2012 11:14:57 +0200
Subject: [Engine-devel] ovirt-python-sdk wiki has moved
In-Reply-To: <50BB1B19.9070507@redhat.com>
References: <50BB1B19.9070507@redhat.com>
Message-ID: <50BB1C11.7010205@redhat.com>


As part of adding more SDKs to oVirt project we moved python-sdk wiki here [1].

[1] http://www.ovirt.org/Python-sdk

-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From lpeer at redhat.com  Sun Dec  2 11:28:39 2012
From: lpeer at redhat.com (Livnat Peer)
Date: Sun, 02 Dec 2012 13:28:39 +0200
Subject: [Engine-devel] Proposal to add Shireesh Anjal as maintainer to
 engine-gluster
In-Reply-To: <50B89873.8030404@redhat.com>
References: <50B89873.8030404@redhat.com>
Message-ID: <50BB3B67.8090105@redhat.com>

On 30/11/12 13:28, Itamar Heim wrote:
> Shireesh has been working on the engine gluster support for the past
> year, covering patches and leading reviews of the various engine gluster
> patches.
> 
> I'd like to propose a new sub-component of the engine for the gluster
> module, adding shireesh as its maintainer.
> 

+1,
I had the pleasure of working with shireesh in the past year, In
addition to the obvious contribution he did (patches and reviews), I
liked his fresh POV in the various discussions.

Thanks, Livnat


> Thanks,
>    Itamar
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel



From thomas at parker.iz34.com  Mon Dec  3 08:55:05 2012
From: thomas at parker.iz34.com (thomas at parker.iz34.com)
Date: Mon, 03 Dec 2012 09:55:05 +0100
Subject: [Engine-devel] Change in ovirt-engine[master]: webadmin: Show
	PM confirmation only	in Virt Cluster
In-Reply-To: <201212030853.qB38rAqK029785@gerrit.ovirt.org>
References: <201212030853.qB38rAqK029785@gerrit.ovirt.org>
Message-ID: <201212030853.qB38rAqK029785@gerrit.ovirt.org>

Gilad Chaplik has submitted this change and it was merged.

Change subject: webadmin: Show PM confirmation only in Virt Cluster
.....................................................................


webadmin: Show PM confirmation only in Virt Cluster

While adding a new host to a cluster, PM confirmation dialog
will be shown only if the cluster supports Virt.

Change-Id: I37f3d9bff9542f90444a16db8ede62675eee6c03
Signed-off-by: Kanagaraj M <kmayilsa at redhat.com>
---
M frontend/webadmin/modules/uicommonweb/src/main/java/org/ovirt/engine/ui/uicommonweb/models/hosts/HostListModel.java
1 file changed, 1 insertion(+), 2 deletions(-)

Approvals:
  Gilad Chaplik: Verified; Looks good to me, approved


--
To view, visit http://gerrit.ovirt.org/9347
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I37f3d9bff9542f90444a16db8ede62675eee6c03
Gerrit-PatchSet: 3
Gerrit-Project: ovirt-engine
Gerrit-Branch: master
Gerrit-Owner: Kanagaraj M <kmayilsa at redhat.com>
Gerrit-Reviewer: Daniel Erez <derez at redhat.com>
Gerrit-Reviewer: Gilad Chaplik <gchaplik at redhat.com>
Gerrit-Reviewer: Kanagaraj M <kmayilsa at redhat.com>
Gerrit-Reviewer: Shireesh Anjal <sanjal at redhat.com>
_______________________________________________
Engine-commits mailing list
Engine-commits at ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-commits
-------------- next part --------------
A non-text attachment was scrubbed...
Name: originalBody.txt
Type: application/octet-stream
Size: 1408 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121203/76eecd6a/attachment.obj>

From mkublin at redhat.com  Mon Dec  3 09:19:48 2012
From: mkublin at redhat.com (Michael Kublin)
Date: Mon, 3 Dec 2012 04:19:48 -0500 (EST)
Subject: [Engine-devel] Task cancelation feature
Message-ID: <2070626308.3115040.1354526388119.JavaMail.root@redhat.com>

Hi, I created a wiki page with design of task cancellation feature.
The url is : http://www.ovirt.org/Features/TaskManagerCancelTask
I can not call these design, I have not any requirements , except a name of the feature,
so my wiki doesn't contains anything except open questions.
Also, I think that it is impossible to make a good feature based on very problematic infrastructure,
I think before we should fix all our infrastructure problems, and after that to add any cancellation task
feature will be a meter of couple hours of work

Regards Michael


From gchaplik at redhat.com  Mon Dec  3 10:16:11 2012
From: gchaplik at redhat.com (Gilad Chaplik)
Date: Mon, 3 Dec 2012 05:16:11 -0500 (EST)
Subject: [Engine-devel] default config value (vdc_options; general)
In-Reply-To: <2030495798.3142039.1354529672067.JavaMail.root@redhat.com>
Message-ID: <2052447568.3142250.1354529771954.JavaMail.root@redhat.com>

hi all,

In engine config table (vdc_options), when an option has a general as its version, and we fetch by version,
we aren't getting the general value, rather the default one (hard coded in the config value enum)

Is this the wanted behavior and why?
According to it general is some-kind of a version number instead of a configurable fallback to a missing version.

NOTE: currently there isn't any configurable default; in case the hard coded default != your configurable default, 
you need to specify it for each version.

I will be glad to fix it.

Thanks, 
Gilad.



From alonbl at redhat.com  Mon Dec  3 10:19:53 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Mon, 3 Dec 2012 05:19:53 -0500 (EST)
Subject: [Engine-devel] default config value (vdc_options; general)
In-Reply-To: <2052447568.3142250.1354529771954.JavaMail.root@redhat.com>
Message-ID: <1084703712.2852155.1354529993246.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Gilad Chaplik" <gchaplik at redhat.com>
> To: "engine-devel" <engine-devel at ovirt.org>
> Sent: Monday, December 3, 2012 12:16:11 PM
> Subject: [Engine-devel] default config value (vdc_options; general)
> 
> hi all,
> 
> In engine config table (vdc_options), when an option has a general as
> its version, and we fetch by version,
> we aren't getting the general value, rather the default one (hard
> coded in the config value enum)
> 
> Is this the wanted behavior and why?
> According to it general is some-kind of a version number instead of a
> configurable fallback to a missing version.
> 
> NOTE: currently there isn't any configurable default; in case the
> hard coded default != your configurable default,
> you need to specify it for each version.
> 
> I will be glad to fix it.
> 
> Thanks,
> Gilad.

I am about to rewrite this whole configuration subsystem.

Defaults will not be stored in database nor in code.
There will be default per version.

Regards,
Alon.


From dfediuck at redhat.com  Mon Dec  3 12:32:55 2012
From: dfediuck at redhat.com (Doron Fediuck)
Date: Mon, 3 Dec 2012 07:32:55 -0500 (EST)
Subject: [Engine-devel] default config value (vdc_options; general)
In-Reply-To: <1084703712.2852155.1354529993246.JavaMail.root@redhat.com>
Message-ID: <524926007.39282395.1354537975497.JavaMail.root@redhat.com>

----- Original Message -----
> From: "Alon Bar-Lev" <alonbl at redhat.com>
> To: "Gilad Chaplik" <gchaplik at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Monday, December 3, 2012 12:19:53 PM
> Subject: Re: [Engine-devel] default config value (vdc_options; general)
> 
> 
> 
> ----- Original Message -----
> > From: "Gilad Chaplik" <gchaplik at redhat.com>
> > To: "engine-devel" <engine-devel at ovirt.org>
> > Sent: Monday, December 3, 2012 12:16:11 PM
> > Subject: [Engine-devel] default config value (vdc_options; general)
> > 
> > hi all,
> > 
> > In engine config table (vdc_options), when an option has a general
> > as
> > its version, and we fetch by version,
> > we aren't getting the general value, rather the default one (hard
> > coded in the config value enum)
> > 
> > Is this the wanted behavior and why?
> > According to it general is some-kind of a version number instead of
> > a
> > configurable fallback to a missing version.
> > 
> > NOTE: currently there isn't any configurable default; in case the
> > hard coded default != your configurable default,
> > you need to specify it for each version.
> > 
> > I will be glad to fix it.
> > 
> > Thanks,
> > Gilad.
> 
> I am about to rewrite this whole configuration subsystem.
> 
> Defaults will not be stored in database nor in code.
> There will be default per version.
> 
> Regards,
> Alon.

Please explain 'default per version';

- Can you have more than one configuration for the same version?
- What happens if I need a general configuration which isn't relevant for a specific version?


From alonbl at redhat.com  Mon Dec  3 12:35:52 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Mon, 3 Dec 2012 07:35:52 -0500 (EST)
Subject: [Engine-devel] default config value (vdc_options; general)
In-Reply-To: <524926007.39282395.1354537975497.JavaMail.root@redhat.com>
Message-ID: <1054053490.2870340.1354538152213.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Doron Fediuck" <dfediuck at redhat.com>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, "Gilad Chaplik" <gchaplik at redhat.com>
> Sent: Monday, December 3, 2012 2:32:55 PM
> Subject: Re: [Engine-devel] default config value (vdc_options; general)
> 
> ----- Original Message -----
> > From: "Alon Bar-Lev" <alonbl at redhat.com>
> > To: "Gilad Chaplik" <gchaplik at redhat.com>
> > Cc: "engine-devel" <engine-devel at ovirt.org>
> > Sent: Monday, December 3, 2012 12:19:53 PM
> > Subject: Re: [Engine-devel] default config value (vdc_options;
> > general)
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Gilad Chaplik" <gchaplik at redhat.com>
> > > To: "engine-devel" <engine-devel at ovirt.org>
> > > Sent: Monday, December 3, 2012 12:16:11 PM
> > > Subject: [Engine-devel] default config value (vdc_options;
> > > general)
> > > 
> > > hi all,
> > > 
> > > In engine config table (vdc_options), when an option has a
> > > general
> > > as
> > > its version, and we fetch by version,
> > > we aren't getting the general value, rather the default one (hard
> > > coded in the config value enum)
> > > 
> > > Is this the wanted behavior and why?
> > > According to it general is some-kind of a version number instead
> > > of
> > > a
> > > configurable fallback to a missing version.
> > > 
> > > NOTE: currently there isn't any configurable default; in case the
> > > hard coded default != your configurable default,
> > > you need to specify it for each version.
> > > 
> > > I will be glad to fix it.
> > > 
> > > Thanks,
> > > Gilad.
> > 
> > I am about to rewrite this whole configuration subsystem.
> > 
> > Defaults will not be stored in database nor in code.
> > There will be default per version.
> > 
> > Regards,
> > Alon.
> 
> Please explain 'default per version';
> 
> - Can you have more than one configuration for the same version?

No, nothing will be changed, you will have one default per version with fallback to the generic version default.

> - What happens if I need a general configuration which isn't relevant
> for a specific version?

You will get the default of the generic version.

Alon.


From mkolesni at redhat.com  Mon Dec  3 12:37:50 2012
From: mkolesni at redhat.com (Mike Kolesnik)
Date: Mon, 3 Dec 2012 07:37:50 -0500 (EST)
Subject: [Engine-devel] default config value (vdc_options; general)
In-Reply-To: <1054053490.2870340.1354538152213.JavaMail.root@redhat.com>
Message-ID: <598694144.3198096.1354538270457.JavaMail.root@redhat.com>

----- Original Message -----
> 
> 
> ----- Original Message -----
> > From: "Doron Fediuck" <dfediuck at redhat.com>
> > To: "Alon Bar-Lev" <alonbl at redhat.com>
> > Cc: "engine-devel" <engine-devel at ovirt.org>, "Gilad Chaplik"
> > <gchaplik at redhat.com>
> > Sent: Monday, December 3, 2012 2:32:55 PM
> > Subject: Re: [Engine-devel] default config value (vdc_options;
> > general)
> > 
> > ----- Original Message -----
> > > From: "Alon Bar-Lev" <alonbl at redhat.com>
> > > To: "Gilad Chaplik" <gchaplik at redhat.com>
> > > Cc: "engine-devel" <engine-devel at ovirt.org>
> > > Sent: Monday, December 3, 2012 12:19:53 PM
> > > Subject: Re: [Engine-devel] default config value (vdc_options;
> > > general)
> > > 
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Gilad Chaplik" <gchaplik at redhat.com>
> > > > To: "engine-devel" <engine-devel at ovirt.org>
> > > > Sent: Monday, December 3, 2012 12:16:11 PM
> > > > Subject: [Engine-devel] default config value (vdc_options;
> > > > general)
> > > > 
> > > > hi all,
> > > > 
> > > > In engine config table (vdc_options), when an option has a
> > > > general
> > > > as
> > > > its version, and we fetch by version,
> > > > we aren't getting the general value, rather the default one
> > > > (hard
> > > > coded in the config value enum)
> > > > 
> > > > Is this the wanted behavior and why?
> > > > According to it general is some-kind of a version number
> > > > instead
> > > > of
> > > > a
> > > > configurable fallback to a missing version.
> > > > 
> > > > NOTE: currently there isn't any configurable default; in case
> > > > the
> > > > hard coded default != your configurable default,
> > > > you need to specify it for each version.
> > > > 
> > > > I will be glad to fix it.
> > > > 
> > > > Thanks,
> > > > Gilad.
> > > 
> > > I am about to rewrite this whole configuration subsystem.
> > > 
> > > Defaults will not be stored in database nor in code.
> > > There will be default per version.
> > > 
> > > Regards,
> > > Alon.
> > 
> > Please explain 'default per version';
> > 
> > - Can you have more than one configuration for the same version?
> 
> No, nothing will be changed, you will have one default per version
> with fallback to the generic version default.
> 
> > - What happens if I need a general configuration which isn't
> > relevant
> > for a specific version?
> 
> You will get the default of the generic version.
> 

I think it would be wise to check out http://commons.apache.org/configuration/ if re-evaluating the current configuration system.

> Alon.
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From alonbl at redhat.com  Mon Dec  3 12:39:08 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Mon, 3 Dec 2012 07:39:08 -0500 (EST)
Subject: [Engine-devel] default config value (vdc_options; general)
In-Reply-To: <598694144.3198096.1354538270457.JavaMail.root@redhat.com>
Message-ID: <116336073.2870836.1354538348220.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Mike Kolesnik" <mkolesni at redhat.com>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, "Doron Fediuck" <dfediuck at redhat.com>
> Sent: Monday, December 3, 2012 2:37:50 PM
> Subject: Re: [Engine-devel] default config value (vdc_options; general)
> 
> ----- Original Message -----
> > 
> > 
> > ----- Original Message -----
> > > From: "Doron Fediuck" <dfediuck at redhat.com>
> > > To: "Alon Bar-Lev" <alonbl at redhat.com>
> > > Cc: "engine-devel" <engine-devel at ovirt.org>, "Gilad Chaplik"
> > > <gchaplik at redhat.com>
> > > Sent: Monday, December 3, 2012 2:32:55 PM
> > > Subject: Re: [Engine-devel] default config value (vdc_options;
> > > general)
> > > 
> > > ----- Original Message -----
> > > > From: "Alon Bar-Lev" <alonbl at redhat.com>
> > > > To: "Gilad Chaplik" <gchaplik at redhat.com>
> > > > Cc: "engine-devel" <engine-devel at ovirt.org>
> > > > Sent: Monday, December 3, 2012 12:19:53 PM
> > > > Subject: Re: [Engine-devel] default config value (vdc_options;
> > > > general)
> > > > 
> > > > 
> > > > 
> > > > ----- Original Message -----
> > > > > From: "Gilad Chaplik" <gchaplik at redhat.com>
> > > > > To: "engine-devel" <engine-devel at ovirt.org>
> > > > > Sent: Monday, December 3, 2012 12:16:11 PM
> > > > > Subject: [Engine-devel] default config value (vdc_options;
> > > > > general)
> > > > > 
> > > > > hi all,
> > > > > 
> > > > > In engine config table (vdc_options), when an option has a
> > > > > general
> > > > > as
> > > > > its version, and we fetch by version,
> > > > > we aren't getting the general value, rather the default one
> > > > > (hard
> > > > > coded in the config value enum)
> > > > > 
> > > > > Is this the wanted behavior and why?
> > > > > According to it general is some-kind of a version number
> > > > > instead
> > > > > of
> > > > > a
> > > > > configurable fallback to a missing version.
> > > > > 
> > > > > NOTE: currently there isn't any configurable default; in case
> > > > > the
> > > > > hard coded default != your configurable default,
> > > > > you need to specify it for each version.
> > > > > 
> > > > > I will be glad to fix it.
> > > > > 
> > > > > Thanks,
> > > > > Gilad.
> > > > 
> > > > I am about to rewrite this whole configuration subsystem.
> > > > 
> > > > Defaults will not be stored in database nor in code.
> > > > There will be default per version.
> > > > 
> > > > Regards,
> > > > Alon.
> > > 
> > > Please explain 'default per version';
> > > 
> > > - Can you have more than one configuration for the same version?
> > 
> > No, nothing will be changed, you will have one default per version
> > with fallback to the generic version default.
> > 
> > > - What happens if I need a general configuration which isn't
> > > relevant
> > > for a specific version?
> > 
> > You will get the default of the generic version.
> > 
> 
> I think it would be wise to check out
> http://commons.apache.org/configuration/ if re-evaluating the
> current configuration system.

Right. Still did not start anything... Next on my queue :)

Alon.


From amureini at redhat.com  Mon Dec  3 14:03:21 2012
From: amureini at redhat.com (Allon Mureinik)
Date: Mon, 3 Dec 2012 09:03:21 -0500 (EST)
Subject: [Engine-devel] Revisiting Java7
In-Reply-To: <691858204.9577947.1354542010739.JavaMail.root@redhat.com>
Message-ID: <1482018768.9611429.1354543401912.JavaMail.root@redhat.com>

Hi guys,

Earlier today, Java6 compatibility was broken (http://gerrit.ovirt.org/#/c/9430/). 
This was picked up on pretty quickly, and easily fixed (http://gerrit.ovirt.org/#/c/9666/).

However, I think this is a good opportunity to revisit our policy towards Java 7.
Currently, we have an "odd" setup, where we recommend running a compiling /with/ Java 7 [1] but comply to Java 6 language level [2] and JDK [3].
Inevitably, mistakes like the that happened today will happen.

I know we're holding back due to GWT issues, but looking forward to oVirt 4.0, is Java 7 on our roadmap? 

-Allon


[1] http://wiki.ovirt.org/Building_oVirt_engine#Installing_OpenJDK
[2] maven-compiler-plugin section in ${OVIRT_GIT}/pom.xml
[3] http://jenkins.ovirt.org/job/ovirt_engine_animal_sniffer_check/

P.S.
If you want to check that you aren't breaking Java/JDK 6 compatibility locally without installing java 6, you can run "mvn animal-sniffer:check". Note that animal sniffer analyzes binaries, so this has to be done /after/ the project was built. Of course, you can do this in a single line "mvn install animal-sniffer:check"


From lhornyak at redhat.com  Mon Dec  3 14:36:42 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Mon, 3 Dec 2012 09:36:42 -0500 (EST)
Subject: [Engine-devel] Revisiting Java7
In-Reply-To: <1482018768.9611429.1354543401912.JavaMail.root@redhat.com>
Message-ID: <1180046676.5062327.1354545402880.JavaMail.root@redhat.com>

Hi,


----- Original Message -----
> From: "Allon Mureinik" <amureini at redhat.com>
> To: "engine-devel" <engine-devel at ovirt.org>
> Sent: Monday, December 3, 2012 3:03:21 PM
> Subject: [Engine-devel] Revisiting Java7
> 
> Hi guys,
> 
> Earlier today, Java6 compatibility was broken
> (http://gerrit.ovirt.org/#/c/9430/).
> This was picked up on pretty quickly, and easily fixed
> (http://gerrit.ovirt.org/#/c/9666/).
> 
> However, I think this is a good opportunity to revisit our policy
> towards Java 7.
> Currently, we have an "odd" setup, where we recommend running a
> compiling /with/ Java 7 [1] but comply to Java 6 language level [2]
> and JDK [3].
> Inevitably, mistakes like the that happened today will happen.
> 
> I know we're holding back due to GWT issues, but looking forward to
> oVirt 4.0, is Java 7 on our roadmap?

Last time I checked the latest checkstyle, it did not work with java 7 syntax unfortunately...
Theoretically checkstyle already supports java 7, probably they only forgot about the maven plugin.

I think this is blocking the way from using java 7 in the code that we do not share with GWT.

> 
> -Allon
> 
> 
> [1] http://wiki.ovirt.org/Building_oVirt_engine#Installing_OpenJDK
> [2] maven-compiler-plugin section in ${OVIRT_GIT}/pom.xml
> [3] http://jenkins.ovirt.org/job/ovirt_engine_animal_sniffer_check/
> 
> P.S.
> If you want to check that you aren't breaking Java/JDK 6
> compatibility locally without installing java 6, you can run "mvn
> animal-sniffer:check". Note that animal sniffer analyzes binaries,
> so this has to be done /after/ the project was built. Of course, you
> can do this in a single line "mvn install animal-sniffer:check"

Nice :)

> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From smizrahi at redhat.com  Mon Dec  3 17:15:07 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 3 Dec 2012 12:15:07 -0500 (EST)
Subject: [Engine-devel] Task cancelation feature
In-Reply-To: <2070626308.3115040.1354526388119.JavaMail.root@redhat.com>
Message-ID: <1520134129.13217619.1354554907074.JavaMail.root@redhat.com>

VDSM tasks are changing to something completely different.
It's still under discussion but the general direction is that:
- TaskIDs will be decided by the caller.
- VDSM can start tasks on it's own
- There will be no distinction between async tasks and sync tasks. Everything is always async.
- There will be no cleanTask() when tasks are done they return result to the caller and disappear immediately.

Also, some stuff you consider tasks will no longer be tasks any more.
For instance, copying and image will finish successfully once VDSM registers the operation for with the storage subsystem and creates the image handle.
After that the status of the copy is bound to the status of the new image and is tracked that way.
This means that the thing you track when you do copyImage() is actually the creation of the image handle and the metadata to make it usable.
After that is done any host can query the state of the new image by using the image ID and not the task Id which was deprecated.
This will be true for all storage operations.


----- Original Message -----
> From: "Michael Kublin" <mkublin at redhat.com>
> To: "engine-devel" <engine-devel at ovirt.org>
> Sent: Monday, December 3, 2012 4:19:48 AM
> Subject: [Engine-devel] Task cancelation feature
> 
> Hi, I created a wiki page with design of task cancellation feature.
> The url is : http://www.ovirt.org/Features/TaskManagerCancelTask
> I can not call these design, I have not any requirements , except a
> name of the feature,
> so my wiki doesn't contains anything except open questions.
> Also, I think that it is impossible to make a good feature based on
> very problematic infrastructure,
> I think before we should fix all our infrastructure problems, and
> after that to add any cancellation task
> feature will be a meter of couple hours of work
> 
> Regards Michael
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From thomas at parker.iz34.com  Mon Dec  3 17:35:03 2012
From: thomas at parker.iz34.com (thomas at parker.iz34.com)
Date: Mon, 03 Dec 2012 18:35:03 +0100
Subject: [Engine-devel] Change in ovirt-host-deploy[master]: core: misc:
	add space after dot	for confirmation
In-Reply-To: <201212031730.qB3HUL0K027376@gerrit.ovirt.org>
References: <201212031730.qB3HUL0K027376@gerrit.ovirt.org>
Message-ID: <201212031730.qB3HUL0K027376@gerrit.ovirt.org>

Alon Bar-Lev has submitted this change and it was merged.

Change subject: core: misc: add space after dot for confirmation
.....................................................................


core: misc: add space after dot for confirmation

Change-Id: I62e20cc4702eaef424d0856aebdd0ebda70cc6ea
Signed-off-by: Alon Bar-Lev <alonbl at redhat.com>
---
M src/plugins/ovirt-host-deploy/core/misc.py
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Alon Bar-Lev: Verified; Looks good to me, approved


--
To view, visit http://gerrit.ovirt.org/9676
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I62e20cc4702eaef424d0856aebdd0ebda70cc6ea
Gerrit-PatchSet: 1
Gerrit-Project: ovirt-host-deploy
Gerrit-Branch: master
Gerrit-Owner: Alon Bar-Lev <alonbl at redhat.com>
Gerrit-Reviewer: Alon Bar-Lev <alonbl at redhat.com>
_______________________________________________
Engine-commits mailing list
Engine-commits at ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-commits
-------------- next part --------------
A non-text attachment was scrubbed...
Name: originalBody.txt
Type: application/octet-stream
Size: 1058 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121203/ea91ec9e/attachment.obj>

From smizrahi at redhat.com  Mon Dec  3 18:07:04 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 3 Dec 2012 13:07:04 -0500 (EST)
Subject: [Engine-devel] Task cancelation feature
In-Reply-To: <1520134129.13217619.1354554907074.JavaMail.root@redhat.com>
Message-ID: <162351084.13236672.1354558024805.JavaMail.root@redhat.com>

I just noticed I only implied the implications to task cancellation but didn't give a concrete example.

For storage at least, there will be a difference between canceling for now (hibernating) and canceling completely.
When you will cancel a copy VDSM task, for example, what will happen is that VDSM will stop the current copy but persist some information so it can continue the operation on a later time.
This means you are actually "hibernating" the copy operation. To actually cancel the copy completely you need to stop any operation on the target image and remove it.

This means that relevant vdsm task IDs can change in the middle of the operation and that one task in the engine can be N sequential tasks in VDSM.
It also means that some operations can be "hibernated" and some can't.

----- Original Message -----
> From: "Saggi Mizrahi" <smizrahi at redhat.com>
> To: "Michael Kublin" <mkublin at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Monday, December 3, 2012 12:15:07 PM
> Subject: Re: [Engine-devel] Task cancelation feature
> 
> VDSM tasks are changing to something completely different.
> It's still under discussion but the general direction is that:
> - TaskIDs will be decided by the caller.
> - VDSM can start tasks on it's own
> - There will be no distinction between async tasks and sync tasks.
> Everything is always async.
> - There will be no cleanTask() when tasks are done they return result
> to the caller and disappear immediately.
> 
> Also, some stuff you consider tasks will no longer be tasks any more.
> For instance, copying and image will finish successfully once VDSM
> registers the operation for with the storage subsystem and creates
> the image handle.
> After that the status of the copy is bound to the status of the new
> image and is tracked that way.
> This means that the thing you track when you do copyImage() is
> actually the creation of the image handle and the metadata to make
> it usable.
> After that is done any host can query the state of the new image by
> using the image ID and not the task Id which was deprecated.
> This will be true for all storage operations.
> 
> 
> ----- Original Message -----
> > From: "Michael Kublin" <mkublin at redhat.com>
> > To: "engine-devel" <engine-devel at ovirt.org>
> > Sent: Monday, December 3, 2012 4:19:48 AM
> > Subject: [Engine-devel] Task cancelation feature
> > 
> > Hi, I created a wiki page with design of task cancellation feature.
> > The url is : http://www.ovirt.org/Features/TaskManagerCancelTask
> > I can not call these design, I have not any requirements , except a
> > name of the feature,
> > so my wiki doesn't contains anything except open questions.
> > Also, I think that it is impossible to make a good feature based on
> > very problematic infrastructure,
> > I think before we should fix all our infrastructure problems, and
> > after that to add any cancellation task
> > feature will be a meter of couple hours of work
> > 
> > Regards Michael
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > 
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From agl at us.ibm.com  Mon Dec  3 20:30:21 2012
From: agl at us.ibm.com (Adam Litke)
Date: Mon, 3 Dec 2012 14:30:21 -0600
Subject: [Engine-devel] RFD: API: Identifying vdsm objects in the
	next-gen API
In-Reply-To: <1990527255.11981218.1354229949253.JavaMail.root@redhat.com>
References: <20121129222243.GB4798@aglitke>
	<1990527255.11981218.1354229949253.JavaMail.root@redhat.com>
Message-ID: <20121203203021.GD4798@aglitke>

On Thu, Nov 29, 2012 at 05:59:09PM -0500, Saggi Mizrahi wrote:
> 
> 
> ----- Original Message -----
> > From: "Adam Litke" <agl at us.ibm.com> To: "Saggi Mizrahi"
> > <smizrahi at redhat.com> Cc: engine-devel at linode01.ovirt.org, "Dan Kenigsberg"
> > <danken at redhat.com>, "Federico Simoncelli" <fsimonce at redhat.com>, "Ayal
> > Baron" <abaron at redhat.com>, vdsm-devel at lists.fedorahosted.org Sent:
> > Thursday, November 29, 2012 5:22:43 PM Subject: Re: RFD: API: Identifying
> > vdsm objects in the next-gen API
> > 
> > On Thu, Nov 29, 2012 at 04:52:14PM -0500, Saggi Mizrahi wrote:
> > > They are not future proof as the paradigm is completely different.
> > > Storage domain IDs are not static any more (and are not guaranteed to be
> > > unique or the same across the cluster.  Image IDs represent the ID of the
> > > projected data and not the actual unique path.  Just as an example, to run
> > > a VM you give a list of domains that might contain the needed images in
> > > the chain and the image ID of the tip.  The paradigm is changed to and
> > > most calls get non synchronous number of images and domains.  Further
> > > more, the APIs themselves are completely different. So future proofing is
> > > not really an issue.
> > 
> > I don't understand this at all.  Perhaps we could all use some education on
> > the architecture of the planned architectural changes.  If I can pass an
> > arbitrary list of domainIDs that _might_ contain the data, why wouldn't I
> > just pass all of them every time?  In that case, why are they even required
> > since vdsm would have to search anyway?
> It's for optimization mostly, the engine usually has a good idea of where
> stuff are, having it give hints to VDSM can speed up the search process.
> also, then engines knows how transient some storage pieces are. If you have a
> domain that is only there for backup or "owned" by another manager sharing the
> host, you don't want you VMs using the disks that are on that storage
> effectively preventing it from being removed (though we do have plans to have
> qemu switch base snapshots at runtime for just that).

This is not a clean design.  If the search is slow, then maybe we need to
improve caching internally.  Making a client cache a bunch of internal IDs to
pass around sounds like a complete layering violation to me.

> > 
> > > As to making the current API a bit simpler. As I said, making them opaque
> > > is problematic as currently the engine is responsible for creating the
> > > IDs.
> > 
> > As I mentioned in my last post, engine still can specify the ID's when the
> > object is first created.  From that point forward the ID never changes so it
> > can be baked into the identifier.
> Where will this identifier be persisted?
> > 
> > > Further more, some calls require you to play with these (making a template
> > > instead of a snapshot).  Also, the full chain and topology needs to be
> > > completely visible to the engine.
> > 
> > Please provide a specific example of how you play with the IDs.  I can guess
> > where you are going, but I don't want to divert the thread.
> The relationship between volumes and images is deceptive at the moment.  IMG
> is the chain and volume is a member, IMGUUID is only used to for verification
> and to detect when we hit a template going up the chain.  When you do
> operation on images assumptions are being guaranteed about the resulting IDs.
> When you copy an image, you assume to know all the new IDs as they remain the
> same.  With your method I can't tell what the new "opaque" result is going to
> be.  Preview mode (another abomination being deprecated) relies on the
> disconnect between imgUUID and volUUID.  Live migration currently moves a lot
> of the responsibility to the engine.

No client should need to know about all of these internal details.  I understand
that's the way it is today, and that's one of the main reasons that the API is a
complete pain to use.

> > 
> > > These things, as you said, are problematic. But this is the way things are
> > > today.
> > 
> > We are changing them.
> Any intermediary step is needlessly problematic for existing clients.  Work is
> already in progress for fixing the API properly, making some calls a bit nicer
> isn't an excuse to start making more compatibility code in the engine.

The engine won't need compatibility code.  This only would impact the jsonrpc
bindings which aren't used by engine yet.  When engine switches over, then yes
it would need to adapt.

> > 
> > > As for task IDs.  Currently task IDs are only used for storage and they
> > > get persisted to disk. This is WRONG and is not the case with the new
> > > storage API.  Because we moved to an asynchronous message based protocol
> > > (json-rpc over TCP\AMQP) there is no need to generate a task ID. it is
> > > built in to json-rpc.  json-rpc specifies that the IDs have to be unique
> > > for a client as long as the request is still active.  This is good enough
> > > as internally we can have a verb for a client to query it's own running
> > > tasks and a verb to query other host tasks by mangling in the client
> > > before the ID.  Because the protocol is
> > 
> > So this would rely on the client keeping the connection open and as soon as
> > it disconnects it would lose the ability to query tasks from before the
> > connection went down?  I don't know if it's a good idea to conflate message
> > ID's with task ID's.  While the protocol can operate asynchronously, some
> > calls have synchronous semantics and others have asynchronous semantics.  I
> > would expect sync calls to return their data immediately and async calls to
> > return immediately with either: an error code, or an 'operation started'
> > message and associated ID for querying the status of the operation.
> Upon reflection I agree that having the request ID unique per client is
> problematic and we need to make sure they are unique per host at every point
> in time.
> > 
> > > asynchronous all calls are asynchronous by nature well.  Tasks will no
> > > longer be persisted or expected to be persisted. It's the callers
> > > responsibility to query the state and see if the operation succeeded or
> > > failed if the caller or VDSM died in the middle of the call. The current
> > > "cleanTask()" system can't be used when more then one client is using VDSM
> > > and will not be used for anything other then legacy storage.
> > 
> > I agree about not persisting tasks in the future.  Although I think finished
> > tasks should remain in memory for some time so they can be queried by a
> > client who must reconnect.
> I am completely against keeping the task for a nominal amount of time, it just
> makes another flow.  You need to have code that makes up in case you missed
> that window any way then just have one recovery code path, when VDSM looses
> you task or you lose VDSM recover immediately.  Also, because task IDs can be
> reused once they expire assuming that the task you encountered is the same
> task you originally sent is problematic.
> 
> If you expect intermittent connections use the AMQP backend (which will
> support broker-less p2p communication as well)

HOw will you tell the difference between a completed task and an invalid (no
such task) task?  Do all completed tasks just issue a task completed event?

> > 
> > > AFAIK Apart from storage all objects IDs are constructed with a single ID,
> > > name or alias. VMs, storageConnections, network interfaces. So it's not a
> > > real issue.  I agree that in the future we should keep the idiom of pass
> > > configuration once, name it, and keep using the name to reference the
> > > object.
> > 
> > Yes, storage is the major problem here.
> And, as I said, changing the API is problematic for migration of current
> users.
> > 

-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center



From smizrahi at redhat.com  Mon Dec  3 20:57:42 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 3 Dec 2012 15:57:42 -0500 (EST)
Subject: [Engine-devel] RFD: API: Identifying vdsm objects in the
	next-gen API
In-Reply-To: <20121203203021.GD4798@aglitke>
Message-ID: <1559868483.13318316.1354568262224.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Adam Litke" <agl at us.ibm.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: engine-devel at linode01.ovirt.org, "Dan Kenigsberg" <danken at redhat.com>, "Federico Simoncelli"
> <fsimonce at redhat.com>, "Ayal Baron" <abaron at redhat.com>, vdsm-devel at lists.fedorahosted.org
> Sent: Monday, December 3, 2012 3:30:21 PM
> Subject: Re: RFD: API: Identifying vdsm objects in the next-gen API
> 
> On Thu, Nov 29, 2012 at 05:59:09PM -0500, Saggi Mizrahi wrote:
> > 
> > 
> > ----- Original Message -----
> > > From: "Adam Litke" <agl at us.ibm.com> To: "Saggi Mizrahi"
> > > <smizrahi at redhat.com> Cc: engine-devel at linode01.ovirt.org, "Dan
> > > Kenigsberg"
> > > <danken at redhat.com>, "Federico Simoncelli" <fsimonce at redhat.com>,
> > > "Ayal
> > > Baron" <abaron at redhat.com>, vdsm-devel at lists.fedorahosted.org
> > > Sent:
> > > Thursday, November 29, 2012 5:22:43 PM Subject: Re: RFD: API:
> > > Identifying
> > > vdsm objects in the next-gen API
> > > 
> > > On Thu, Nov 29, 2012 at 04:52:14PM -0500, Saggi Mizrahi wrote:
> > > > They are not future proof as the paradigm is completely
> > > > different.
> > > > Storage domain IDs are not static any more (and are not
> > > > guaranteed to be
> > > > unique or the same across the cluster.  Image IDs represent the
> > > > ID of the
> > > > projected data and not the actual unique path.  Just as an
> > > > example, to run
> > > > a VM you give a list of domains that might contain the needed
> > > > images in
> > > > the chain and the image ID of the tip.  The paradigm is changed
> > > > to and
> > > > most calls get non synchronous number of images and domains.
> > > >  Further
> > > > more, the APIs themselves are completely different. So future
> > > > proofing is
> > > > not really an issue.
> > > 
> > > I don't understand this at all.  Perhaps we could all use some
> > > education on
> > > the architecture of the planned architectural changes.  If I can
> > > pass an
> > > arbitrary list of domainIDs that _might_ contain the data, why
> > > wouldn't I
> > > just pass all of them every time?  In that case, why are they
> > > even required
> > > since vdsm would have to search anyway?
> > It's for optimization mostly, the engine usually has a good idea of
> > where
> > stuff are, having it give hints to VDSM can speed up the search
> > process.
> > also, then engines knows how transient some storage pieces are. If
> > you have a
> > domain that is only there for backup or "owned" by another manager
> > sharing the
> > host, you don't want you VMs using the disks that are on that
> > storage
> > effectively preventing it from being removed (though we do have
> > plans to have
> > qemu switch base snapshots at runtime for just that).
> 
> This is not a clean design.  If the search is slow, then maybe we
> need to
> improve caching internally.  Making a client cache a bunch of
> internal IDs to
> pass around sounds like a complete layering violation to me.
You can't cache this, if the same template exists on an 2 different NFS domains only the engine has enough information to know which you should use.
We only have the engine give us thing information when starting a VM or merging\copying an image that resides on multiple domains.
It is also completely optional. I didn't like it either.
> 
> > > 
> > > > As to making the current API a bit simpler. As I said, making
> > > > them opaque
> > > > is problematic as currently the engine is responsible for
> > > > creating the
> > > > IDs.
> > > 
> > > As I mentioned in my last post, engine still can specify the ID's
> > > when the
> > > object is first created.  From that point forward the ID never
> > > changes so it
> > > can be baked into the identifier.
> > Where will this identifier be persisted?
> > > 
> > > > Further more, some calls require you to play with these (making
> > > > a template
> > > > instead of a snapshot).  Also, the full chain and topology
> > > > needs to be
> > > > completely visible to the engine.
> > > 
> > > Please provide a specific example of how you play with the IDs.
> > >  I can guess
> > > where you are going, but I don't want to divert the thread.
> > The relationship between volumes and images is deceptive at the
> > moment.  IMG
> > is the chain and volume is a member, IMGUUID is only used to for
> > verification
> > and to detect when we hit a template going up the chain.  When you
> > do
> > operation on images assumptions are being guaranteed about the
> > resulting IDs.
> > When you copy an image, you assume to know all the new IDs as they
> > remain the
> > same.  With your method I can't tell what the new "opaque" result
> > is going to
> > be.  Preview mode (another abomination being deprecated) relies on
> > the
> > disconnect between imgUUID and volUUID.  Live migration currently
> > moves a lot
> > of the responsibility to the engine.
> 
> No client should need to know about all of these internal details.  I
> understand
> that's the way it is today, and that's one of the main reasons that
> the API is a
> complete pain to use.
You are correct but this is how this API was designed you can't get away from that.
> 
> > > 
> > > > These things, as you said, are problematic. But this is the way
> > > > things are
> > > > today.
> > > 
> > > We are changing them.
> > Any intermediary step is needlessly problematic for existing
> > clients.  Work is
> > already in progress for fixing the API properly, making some calls
> > a bit nicer
> > isn't an excuse to start making more compatibility code in the
> > engine.
> 
> The engine won't need compatibility code.  This only would impact the
> jsonrpc
> bindings which aren't used by engine yet.  When engine switches over,
> then yes
> it would need to adapt.
This means that you put it as a condition to adapting the json-rpc base API which will postpone engine adoption significantly.
> 
> > > 
> > > > As for task IDs.  Currently task IDs are only used for storage
> > > > and they
> > > > get persisted to disk. This is WRONG and is not the case with
> > > > the new
> > > > storage API.  Because we moved to an asynchronous message based
> > > > protocol
> > > > (json-rpc over TCP\AMQP) there is no need to generate a task
> > > > ID. it is
> > > > built in to json-rpc.  json-rpc specifies that the IDs have to
> > > > be unique
> > > > for a client as long as the request is still active.  This is
> > > > good enough
> > > > as internally we can have a verb for a client to query it's own
> > > > running
> > > > tasks and a verb to query other host tasks by mangling in the
> > > > client
> > > > before the ID.  Because the protocol is
> > > 
> > > So this would rely on the client keeping the connection open and
> > > as soon as
> > > it disconnects it would lose the ability to query tasks from
> > > before the
> > > connection went down?  I don't know if it's a good idea to
> > > conflate message
> > > ID's with task ID's.  While the protocol can operate
> > > asynchronously, some
> > > calls have synchronous semantics and others have asynchronous
> > > semantics.  I
> > > would expect sync calls to return their data immediately and
> > > async calls to
> > > return immediately with either: an error code, or an 'operation
> > > started'
> > > message and associated ID for querying the status of the
> > > operation.
> > Upon reflection I agree that having the request ID unique per
> > client is
> > problematic and we need to make sure they are unique per host at
> > every point
> > in time.
> > > 
> > > > asynchronous all calls are asynchronous by nature well.  Tasks
> > > > will no
> > > > longer be persisted or expected to be persisted. It's the
> > > > callers
> > > > responsibility to query the state and see if the operation
> > > > succeeded or
> > > > failed if the caller or VDSM died in the middle of the call.
> > > > The current
> > > > "cleanTask()" system can't be used when more then one client is
> > > > using VDSM
> > > > and will not be used for anything other then legacy storage.
> > > 
> > > I agree about not persisting tasks in the future.  Although I
> > > think finished
> > > tasks should remain in memory for some time so they can be
> > > queried by a
> > > client who must reconnect.
> > I am completely against keeping the task for a nominal amount of
> > time, it just
> > makes another flow.  You need to have code that makes up in case
> > you missed
> > that window any way then just have one recovery code path, when
> > VDSM looses
> > you task or you lose VDSM recover immediately.  Also, because task
> > IDs can be
> > reused once they expire assuming that the task you encountered is
> > the same
> > task you originally sent is problematic.
> > 
> > If you expect intermittent connections use the AMQP backend (which
> > will
> > support broker-less p2p communication as well)
> 
> HOw will you tell the difference between a completed task and an
> invalid (no
> such task) task?  Do all completed tasks just issue a task completed
> event?
When task complete they return a value (and might even send a task completed even).
If you lost that task you will just have to handle that by inspecting the current state.
As I said, you will have to do that anyway if you put a "sleep" on finished task and you missed the window or you are tracking a task that is owned by another VDSM user.
If we have to handle crash situations might as well be crash only. http://en.wikipedia.org/wiki/Crash-only_software
> 
> > > 
> > > > AFAIK Apart from storage all objects IDs are constructed with a
> > > > single ID,
> > > > name or alias. VMs, storageConnections, network interfaces. So
> > > > it's not a
> > > > real issue.  I agree that in the future we should keep the
> > > > idiom of pass
> > > > configuration once, name it, and keep using the name to
> > > > reference the
> > > > object.
> > > 
> > > Yes, storage is the major problem here.
> > And, as I said, changing the API is problematic for migration of
> > current
> > users.
> > > 
> 
> --
> Adam Litke <agl at us.ibm.com>
> IBM Linux Technology Center
> 
> 


From agl at us.ibm.com  Mon Dec  3 22:55:29 2012
From: agl at us.ibm.com (Adam Litke)
Date: Mon, 3 Dec 2012 16:55:29 -0600
Subject: [Engine-devel] RFD: API: Identifying vdsm objects in the
	next-gen API
In-Reply-To: <1559868483.13318316.1354568262224.JavaMail.root@redhat.com>
References: <20121203203021.GD4798@aglitke>
	<1559868483.13318316.1354568262224.JavaMail.root@redhat.com>
Message-ID: <20121203225529.GG4798@aglitke>

On Mon, Dec 03, 2012 at 03:57:42PM -0500, Saggi Mizrahi wrote:
> 
> 
> ----- Original Message -----
> > From: "Adam Litke" <agl at us.ibm.com> To: "Saggi Mizrahi"
> > <smizrahi at redhat.com> Cc: engine-devel at linode01.ovirt.org, "Dan Kenigsberg"
> > <danken at redhat.com>, "Federico Simoncelli" <fsimonce at redhat.com>, "Ayal
> > Baron" <abaron at redhat.com>, vdsm-devel at lists.fedorahosted.org Sent: Monday,
> > December 3, 2012 3:30:21 PM Subject: Re: RFD: API: Identifying vdsm objects
> > in the next-gen API
> > 
> > On Thu, Nov 29, 2012 at 05:59:09PM -0500, Saggi Mizrahi wrote:
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Adam Litke" <agl at us.ibm.com> To: "Saggi Mizrahi"
> > > > <smizrahi at redhat.com> Cc: engine-devel at linode01.ovirt.org, "Dan
> > > > Kenigsberg" <danken at redhat.com>, "Federico Simoncelli"
> > > > <fsimonce at redhat.com>, "Ayal Baron" <abaron at redhat.com>,
> > > > vdsm-devel at lists.fedorahosted.org Sent: Thursday, November 29, 2012
> > > > 5:22:43 PM Subject: Re: RFD: API: Identifying vdsm objects in the
> > > > next-gen API
> > > > 
> > > > On Thu, Nov 29, 2012 at 04:52:14PM -0500, Saggi Mizrahi wrote:
> > > > > They are not future proof as the paradigm is completely different.
> > > > > Storage domain IDs are not static any more (and are not guaranteed to
> > > > > be unique or the same across the cluster.  Image IDs represent the ID
> > > > > of the projected data and not the actual unique path.  Just as an
> > > > > example, to run a VM you give a list of domains that might contain the
> > > > > needed images in the chain and the image ID of the tip.  The paradigm
> > > > > is changed to and most calls get non synchronous number of images and
> > > > > domains.  Further more, the APIs themselves are completely different.
> > > > > So future proofing is not really an issue.
> > > > 
> > > > I don't understand this at all.  Perhaps we could all use some education
> > > > on the architecture of the planned architectural changes.  If I can pass
> > > > an arbitrary list of domainIDs that _might_ contain the data, why
> > > > wouldn't I just pass all of them every time?  In that case, why are they
> > > > even required since vdsm would have to search anyway?
> > > It's for optimization mostly, the engine usually has a good idea of where
> > > stuff are, having it give hints to VDSM can speed up the search process.
> > > also, then engines knows how transient some storage pieces are. If you
> > > have a domain that is only there for backup or "owned" by another manager
> > > sharing the host, you don't want you VMs using the disks that are on that
> > > storage effectively preventing it from being removed (though we do have
> > > plans to have qemu switch base snapshots at runtime for just that).
> > 
> > This is not a clean design.  If the search is slow, then maybe we need to
> > improve caching internally.  Making a client cache a bunch of internal IDs
> > to pass around sounds like a complete layering violation to me.
> You can't cache this, if the same template exists on an 2 different NFS
> domains only the engine has enough information to know which you should use.
> We only have the engine give us thing information when starting a VM or
> merging\copying an image that resides on multiple domains.  It is also
> completely optional. I didn't like it either.

Is it even valid for the same template (with identical uuids) to exist in two
places?  I thought uuids aren't supposed to collide.  I can envision some
scenario where a cached storagedomain/storagepool relationship becomes invalid
because another user detached the storagedomain.  In that case, the API just
returns the normal error about "sd XXX is not attached to sp XXX".  So I don't
see any problem here.

> > 
> > > > 
> > > > > As to making the current API a bit simpler. As I said, making them
> > > > > opaque is problematic as currently the engine is responsible for
> > > > > creating the IDs.
> > > > 
> > > > As I mentioned in my last post, engine still can specify the ID's when
> > > > the object is first created.  From that point forward the ID never
> > > > changes so it can be baked into the identifier.
> > > Where will this identifier be persisted?
> > > > 
> > > > > Further more, some calls require you to play with these (making a
> > > > > template instead of a snapshot).  Also, the full chain and topology
> > > > > needs to be completely visible to the engine.
> > > > 
> > > > Please provide a specific example of how you play with the IDs.  I can
> > > > guess where you are going, but I don't want to divert the thread.
> > > The relationship between volumes and images is deceptive at the moment.
> > > IMG is the chain and volume is a member, IMGUUID is only used to for
> > > verification and to detect when we hit a template going up the chain.
> > > When you do operation on images assumptions are being guaranteed about the
> > > resulting IDs.  When you copy an image, you assume to know all the new IDs
> > > as they remain the same.  With your method I can't tell what the new
> > > "opaque" result is going to be.  Preview mode (another abomination being
> > > deprecated) relies on the disconnect between imgUUID and volUUID.  Live
> > > migration currently moves a lot of the responsibility to the engine.
> > 
> > No client should need to know about all of these internal details.  I
> > understand that's the way it is today, and that's one of the main reasons
> > that the API is a complete pain to use.
> You are correct but this is how this API was designed you can't get away from
> that.

We are trying to fix the problems on the second try.  This will inevitably cause
a little discomfort in the short term but will be worth it in the long run.

> > 
> > > > 
> > > > > These things, as you said, are problematic. But this is the way things
> > > > > are today.
> > > > 
> > > > We are changing them.
> > > Any intermediary step is needlessly problematic for existing clients.
> > > Work is already in progress for fixing the API properly, making some calls
> > > a bit nicer isn't an excuse to start making more compatibility code in the
> > > engine.
> > 
> > The engine won't need compatibility code.  This only would impact the
> > jsonrpc bindings which aren't used by engine yet.  When engine switches
> > over, then yes it would need to adapt.
> This means that you put it as a condition to adapting the json-rpc base API
> which will postpone engine adoption significantly.

No one has said the transition to json-rpc would be effortless.  We're not
switching protocols merely of disdain for xmlrpc.  We are planning some
structural changes to the API to improve usability for everyone.

> > 
> > > > 
> > > > > As for task IDs.  Currently task IDs are only used for storage and
> > > > > they get persisted to disk. This is WRONG and is not the case with the
> > > > > new storage API.  Because we moved to an asynchronous message based
> > > > > protocol (json-rpc over TCP\AMQP) there is no need to generate a task
> > > > > ID. it is built in to json-rpc.  json-rpc specifies that the IDs have
> > > > > to be unique for a client as long as the request is still active.
> > > > > This is good enough as internally we can have a verb for a client to
> > > > > query it's own running tasks and a verb to query other host tasks by
> > > > > mangling in the client before the ID.  Because the protocol is
> > > > 
> > > > So this would rely on the client keeping the connection open and as soon
> > > > as it disconnects it would lose the ability to query tasks from before
> > > > the connection went down?  I don't know if it's a good idea to conflate
> > > > message ID's with task ID's.  While the protocol can operate
> > > > asynchronously, some calls have synchronous semantics and others have
> > > > asynchronous semantics.  I would expect sync calls to return their data
> > > > immediately and async calls to return immediately with either: an error
> > > > code, or an 'operation started' message and associated ID for querying
> > > > the status of the operation.
> > > Upon reflection I agree that having the request ID unique per client is
> > > problematic and we need to make sure they are unique per host at every
> > > point in time.
> > > > 
> > > > > asynchronous all calls are asynchronous by nature well.  Tasks will no
> > > > > longer be persisted or expected to be persisted. It's the callers
> > > > > responsibility to query the state and see if the operation succeeded
> > > > > or failed if the caller or VDSM died in the middle of the call.  The
> > > > > current "cleanTask()" system can't be used when more then one client
> > > > > is using VDSM and will not be used for anything other then legacy
> > > > > storage.
> > > > 
> > > > I agree about not persisting tasks in the future.  Although I think
> > > > finished tasks should remain in memory for some time so they can be
> > > > queried by a client who must reconnect.
> > > I am completely against keeping the task for a nominal amount of time, it
> > > just makes another flow.  You need to have code that makes up in case you
> > > missed that window any way then just have one recovery code path, when
> > > VDSM looses you task or you lose VDSM recover immediately.  Also, because
> > > task IDs can be reused once they expire assuming that the task you
> > > encountered is the same task you originally sent is problematic.
> > > 
> > > If you expect intermittent connections use the AMQP backend (which will
> > > support broker-less p2p communication as well)
> > 
> > HOw will you tell the difference between a completed task and an invalid (no
> > such task) task?  Do all completed tasks just issue a task completed event?
> When task complete they return a value (and might even send a task completed
> even).  If you lost that task you will just have to handle that by inspecting
> the current state.  As I said, you will have to do that anyway if you put a
> "sleep" on finished task and you missed the window or you are tracking a task
> that is owned by another VDSM user.  If we have to handle crash situations
> might as well be crash only. http://en.wikipedia.org/wiki/Crash-only_software
> > 
> > > > 
> > > > > AFAIK Apart from storage all objects IDs are constructed with a single
> > > > > ID, name or alias. VMs, storageConnections, network interfaces. So
> > > > > it's not a real issue.  I agree that in the future we should keep the
> > > > > idiom of pass configuration once, name it, and keep using the name to
> > > > > reference the object.
> > > > 
> > > > Yes, storage is the major problem here.
> > > And, as I said, changing the API is problematic for migration of current
> > > users.
> > > > 
> > 
> > -- Adam Litke <agl at us.ibm.com> IBM Linux Technology Center
> > 
> > 
> 

-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center



From iheim at redhat.com  Tue Dec  4 00:09:56 2012
From: iheim at redhat.com (Itamar Heim)
Date: Tue, 04 Dec 2012 02:09:56 +0200
Subject: [Engine-devel] Design review summary for 3.2 - adding support
 for External Events
In-Reply-To: <1652353217.16442236.1354223516335.JavaMail.root@redhat.com>
References: <1652353217.16442236.1354223516335.JavaMail.root@redhat.com>
Message-ID: <50BD3F54.4000508@redhat.com>

On 11/29/2012 11:11 PM, Eli Mesika wrote:
> Hi
>
> please note that this section
> http://wiki.ovirt.org/wiki/Features/Design/DetailedExternalEvents#Permissions
> was changed and approved by PM (Miki)
>
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
>



> Invoke ..api/events/add API giving at least AuditLogType, Severity, Origin & CustomEventId

why do you need the AuditLogType - you already know it is 
ExternalEvent/Alert? you only need to know the severity iiuc.

> When the Event/Alert is on a specific object, the object instance id should be set.

what if it is on multiple objects, like events are today?

...

> Permissions on Entity Instances
>
> There will be no permission check on entity instances on which External Events are injected

this means this permission is always at system level, breaking the 
current permission scheme where permission can be given on a portion of 
the system.

> The reason is that in order to invoke an External Events on an entity instance, the invoker should know the entity instance UUID and therefore we had already checked that the invoker has the right permissions on the entity instance when he gets the information.

UUID are not cryptographically secure. knowing a UUID is not a way to 
decide to give a permission.
(also, this is an admin action, and admins always see all objects in the 
system today)

> Also, double checking that in the AddExternalEvent command is not simple, since each Entity may have several ActionGroups (Create, Edit etc.) associated with it, so it is not clear which to check

it is very clear - you check for each entity the user has the new 
ActionGroup that allows to inject events (same one you added to the 
superuser and to the newly created special role).

> So, in order to keep things simple, we will assume that if the caller to Add External Event has the Entity UUID in hand, all we have to check is that he has permission to inject External Events

as i stated above, this means you are creating an ActionGroup which is 
global, regardless of the entity the role with it is given on.

...


> User Experience
>
> Global External Events will be displayed on the Global Events TAB
> Entity instance External Events will be displayed on the Events TAB when selecting the Entity instance

1. not clear here they will also be displayed in the global tab?
2. not clear here that multiple entities can be specified for an event.

...

> Installation/Upgrade
>
> Add additional fields to audit_log table upon upgrade
> Add the permission(ActionGroup) to manipulate External Events to other admin roles already defined upon upgrade.

but in permission it is stated only superuser and a new role will be 
given this actiongroup, and here more admin roles are mentioned (if so, 
which is the correct one).

Thanks,
    Itamar


From iheim at redhat.com  Mon Dec  3 23:40:10 2012
From: iheim at redhat.com (Itamar Heim)
Date: Tue, 04 Dec 2012 01:40:10 +0200
Subject: [Engine-devel] Revisiting Java7
In-Reply-To: <1482018768.9611429.1354543401912.JavaMail.root@redhat.com>
References: <1482018768.9611429.1354543401912.JavaMail.root@redhat.com>
Message-ID: <50BD385A.1030001@redhat.com>

On 12/03/2012 04:03 PM, Allon Mureinik wrote:
> Hi guys,
>
> Earlier today, Java6 compatibility was broken (http://gerrit.ovirt.org/#/c/9430/).
> This was picked up on pretty quickly, and easily fixed (http://gerrit.ovirt.org/#/c/9666/).
>
> However, I think this is a good opportunity to revisit our policy towards Java 7.
> Currently, we have an "odd" setup, where we recommend running a compiling /with/ Java 7 [1] but comply to Java 6 language level [2] and JDK [3].
> Inevitably, mistakes like the that happened today will happen.
>
> I know we're holding back due to GWT issues, but looking forward to oVirt 4.0, is Java 7 on our roadmap?

what was the GWT issue?

>
> -Allon
>
>
> [1] http://wiki.ovirt.org/Building_oVirt_engine#Installing_OpenJDK
> [2] maven-compiler-plugin section in ${OVIRT_GIT}/pom.xml
> [3] http://jenkins.ovirt.org/job/ovirt_engine_animal_sniffer_check/
>
> P.S.
> If you want to check that you aren't breaking Java/JDK 6 compatibility locally without installing java 6, you can run "mvn animal-sniffer:check". Note that animal sniffer analyzes binaries, so this has to be done /after/ the project was built. Of course, you can do this in a single line "mvn install animal-sniffer:check"
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
>




From amureini at redhat.com  Tue Dec  4 08:07:00 2012
From: amureini at redhat.com (Allon Mureinik)
Date: Tue, 4 Dec 2012 03:07:00 -0500 (EST)
Subject: [Engine-devel] Proposal to add Shireesh Anjal as maintainer
	to	engine-gluster
In-Reply-To: <50B89873.8030404@redhat.com>
Message-ID: <1840938534.10081182.1354608420445.JavaMail.root@redhat.com>

+1

----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: engine-devel at ovirt.org
> Sent: Friday, November 30, 2012 1:28:51 PM
> Subject: [Engine-devel] Proposal to add Shireesh Anjal as maintainer to	engine-gluster
> 
> Shireesh has been working on the engine gluster support for the past
> year, covering patches and leading reviews of the various engine
> gluster
> patches.
> 
> I'd like to propose a new sub-component of the engine for the gluster
> module, adding shireesh as its maintainer.
> 
> Thanks,
>     Itamar
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From jhernand at redhat.com  Tue Dec  4 08:59:53 2012
From: jhernand at redhat.com (Juan Hernandez)
Date: Tue, 04 Dec 2012 09:59:53 +0100
Subject: [Engine-devel] Change of engine version number to 3.2.0
Message-ID: <50BDBB89.7000501@redhat.com>

Hello all,

Please note that the change to move engine version number to 3.2.0
(http://gerrit.ovirt.org/8993) has been merged. Please be careful not to
undo it accidentally if you need to push changes to the pom.xml files.

Regards,
Juan Hernandez
-- 
Direcci?n Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
3?D, 28016 Madrid, Spain
Inscrita en el Reg. Mercantil de Madrid ? C.I.F. B82657941 - Red Hat S.L.


From amureini at redhat.com  Tue Dec  4 09:20:49 2012
From: amureini at redhat.com (Allon Mureinik)
Date: Tue, 4 Dec 2012 04:20:49 -0500 (EST)
Subject: [Engine-devel] Revisiting Java7
In-Reply-To: <50BD385A.1030001@redhat.com>
Message-ID: <1666988898.10106991.1354612849908.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Allon Mureinik" <amureini at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Tuesday, December 4, 2012 1:40:10 AM
> Subject: Re: [Engine-devel] Revisiting Java7
> 
> On 12/03/2012 04:03 PM, Allon Mureinik wrote:
> > Hi guys,
> >
> > Earlier today, Java6 compatibility was broken
> > (http://gerrit.ovirt.org/#/c/9430/).
> > This was picked up on pretty quickly, and easily fixed
> > (http://gerrit.ovirt.org/#/c/9666/).
> >
> > However, I think this is a good opportunity to revisit our policy
> > towards Java 7.
> > Currently, we have an "odd" setup, where we recommend running a
> > compiling /with/ Java 7 [1] but comply to Java 6 language level
> > [2] and JDK [3].
> > Inevitably, mistakes like the that happened today will happen.
> >
> > I know we're holding back due to GWT issues, but looking forward to
> > oVirt 4.0, is Java 7 on our roadmap?
> 
> what was the GWT issue?
GWT 2.3 that we use doesn't support java 7 syntax.
The latest version, 2.5, doesn't either.
I'm not sure, however, this is a good reason to enforce the java 6 limitation on the entire project (including backed, rest, etc.)
> 
> >
> > -Allon
> >
> >
> > [1] http://wiki.ovirt.org/Building_oVirt_engine#Installing_OpenJDK
> > [2] maven-compiler-plugin section in ${OVIRT_GIT}/pom.xml
> > [3] http://jenkins.ovirt.org/job/ovirt_engine_animal_sniffer_check/
> >
> > P.S.
> > If you want to check that you aren't breaking Java/JDK 6
> > compatibility locally without installing java 6, you can run "mvn
> > animal-sniffer:check". Note that animal sniffer analyzes binaries,
> > so this has to be done /after/ the project was built. Of course,
> > you can do this in a single line "mvn install
> > animal-sniffer:check"
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> >
> 
> 
> 


From amureini at redhat.com  Tue Dec  4 09:22:15 2012
From: amureini at redhat.com (Allon Mureinik)
Date: Tue, 4 Dec 2012 04:22:15 -0500 (EST)
Subject: [Engine-devel] Revisiting Java7
In-Reply-To: <1180046676.5062327.1354545402880.JavaMail.root@redhat.com>
Message-ID: <1909563038.10107249.1354612935305.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Laszlo Hornyak" <lhornyak at redhat.com>
> To: "Allon Mureinik" <amureini at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Monday, December 3, 2012 4:36:42 PM
> Subject: Re: [Engine-devel] Revisiting Java7
> 
> Hi,
> 
> 
> ----- Original Message -----
> > From: "Allon Mureinik" <amureini at redhat.com>
> > To: "engine-devel" <engine-devel at ovirt.org>
> > Sent: Monday, December 3, 2012 3:03:21 PM
> > Subject: [Engine-devel] Revisiting Java7
> > 
> > Hi guys,
> > 
> > Earlier today, Java6 compatibility was broken
> > (http://gerrit.ovirt.org/#/c/9430/).
> > This was picked up on pretty quickly, and easily fixed
> > (http://gerrit.ovirt.org/#/c/9666/).
> > 
> > However, I think this is a good opportunity to revisit our policy
> > towards Java 7.
> > Currently, we have an "odd" setup, where we recommend running a
> > compiling /with/ Java 7 [1] but comply to Java 6 language level [2]
> > and JDK [3].
> > Inevitably, mistakes like the that happened today will happen.
> > 
> > I know we're holding back due to GWT issues, but looking forward to
> > oVirt 4.0, is Java 7 on our roadmap?
> 
> Last time I checked the latest checkstyle, it did not work with java
> 7 syntax unfortunately...
2.9.1 seems to support it fine.
I had some issues getting Treewalker to work in it (some weird error with google collections API incompatibility), but I'm sure it's doable.

> Theoretically checkstyle already supports java 7, probably they only
> forgot about the maven plugin.
> 
> I think this is blocking the way from using java 7 in the code that
> we do not share with GWT.
> 
> > 
> > -Allon
> > 
> > 
> > [1] http://wiki.ovirt.org/Building_oVirt_engine#Installing_OpenJDK
> > [2] maven-compiler-plugin section in ${OVIRT_GIT}/pom.xml
> > [3] http://jenkins.ovirt.org/job/ovirt_engine_animal_sniffer_check/
> > 
> > P.S.
> > If you want to check that you aren't breaking Java/JDK 6
> > compatibility locally without installing java 6, you can run "mvn
> > animal-sniffer:check". Note that animal sniffer analyzes binaries,
> > so this has to be done /after/ the project was built. Of course,
> > you
> > can do this in a single line "mvn install animal-sniffer:check"
> 
> Nice :)
> 
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > 
> 


From danken at redhat.com  Tue Dec  4 09:38:13 2012
From: danken at redhat.com (Dan Kenigsberg)
Date: Tue, 4 Dec 2012 11:38:13 +0200
Subject: [Engine-devel] Task cancelation feature
In-Reply-To: <1520134129.13217619.1354554907074.JavaMail.root@redhat.com>
References: <2070626308.3115040.1354526388119.JavaMail.root@redhat.com>
	<1520134129.13217619.1354554907074.JavaMail.root@redhat.com>
Message-ID: <20121204093813.GB18178@redhat.com>

On Mon, Dec 03, 2012 at 12:15:07PM -0500, Saggi Mizrahi wrote:
> VDSM tasks are changing to something completely different.
> It's still under discussion but the general direction is that:
> - TaskIDs will be decided by the caller.
> - VDSM can start tasks on it's own
> - There will be no distinction between async tasks and sync tasks. Everything is always async.
> - There will be no cleanTask() when tasks are done they return result to the caller and disappear immediately.

I'm not sure I understand the motivation for the latter change. I kinda
like the unix process semantics, were the return code of a process is
kept with its id after the process ends, until the process parent calls
wait(2). Otherwise, how can the caller tell why its task has failed?

For example, I'd like to see vmCreate using async tasks like that.
vmCreate returns immediately, and a vdsm task is tracking the vm
creation. If something bad happens, the information about the failure
can be polled by the Engine that created the vm (or a new Engine
instance, after an Engine crash).

Similarily, we may need to make setupNetwork asynchronous, since we
depend on dhclient, which may take a lot of time to finish.

Have these future use cases been debated?

Dan.
> 
> Also, some stuff you consider tasks will no longer be tasks any more.
> For instance, copying and image will finish successfully once VDSM registers the operation for with the storage subsystem and creates the image handle.
> After that the status of the copy is bound to the status of the new image and is tracked that way.
> This means that the thing you track when you do copyImage() is actually the creation of the image handle and the metadata to make it usable.
> After that is done any host can query the state of the new image by using the image ID and not the task Id which was deprecated.
> This will be true for all storage operations.
> 
> 
> ----- Original Message -----
> > From: "Michael Kublin" <mkublin at redhat.com>
> > To: "engine-devel" <engine-devel at ovirt.org>
> > Sent: Monday, December 3, 2012 4:19:48 AM
> > Subject: [Engine-devel] Task cancelation feature
> > 
> > Hi, I created a wiki page with design of task cancellation feature.
> > The url is : http://www.ovirt.org/Features/TaskManagerCancelTask
> > I can not call these design, I have not any requirements , except a
> > name of the feature,
> > so my wiki doesn't contains anything except open questions.
> > Also, I think that it is impossible to make a good feature based on
> > very problematic infrastructure,
> > I think before we should fix all our infrastructure problems, and
> > after that to add any cancellation task
> > feature will be a meter of couple hours of work
> > 
> > Regards Michael
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > 
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel


From alonbl at redhat.com  Tue Dec  4 11:11:52 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Tue, 4 Dec 2012 06:11:52 -0500 (EST)
Subject: [Engine-devel] engine-config utility
In-Reply-To: <2110012948.3152186.1354618916760.JavaMail.root@redhat.com>
Message-ID: <850941977.3152966.1354619512987.JavaMail.root@redhat.com>

Hello,

I had interesting discussion with Eli regarding the engine-config utility and recent commit.

Is the following claim correct?

"""Every change of a variable by engine-config cannot break engine"""

This issue came up because we used to export the CABaseDirectory parameter to engine-config. Changing it without attending to copy pki files may lead to failure. A change was already committed to remove the ability to update this parameter.

Most applications I know, do allow to change variable that in certain cases may break the application. The assumption that I am usually familiar with is that the target audience of these kind of tools are system admins or integrators, which "should know what they are doing", so it is OK to expose parameters that may lead to application breakage, as the ability to customize the application is important.

I am carious what other think in this regard.

Regards,
Alon Bar-Lev.



From iheim at redhat.com  Tue Dec  4 11:13:49 2012
From: iheim at redhat.com (Itamar Heim)
Date: Tue, 04 Dec 2012 13:13:49 +0200
Subject: [Engine-devel] Revisiting Java7
In-Reply-To: <1666988898.10106991.1354612849908.JavaMail.root@redhat.com>
References: <1666988898.10106991.1354612849908.JavaMail.root@redhat.com>
Message-ID: <50BDDAED.6040007@redhat.com>

On 12/04/2012 11:20 AM, Allon Mureinik wrote:
>
>
> ----- Original Message -----
>> From: "Itamar Heim" <iheim at redhat.com>
>> To: "Allon Mureinik" <amureini at redhat.com>
>> Cc: "engine-devel" <engine-devel at ovirt.org>
>> Sent: Tuesday, December 4, 2012 1:40:10 AM
>> Subject: Re: [Engine-devel] Revisiting Java7
>>
>> On 12/03/2012 04:03 PM, Allon Mureinik wrote:
>>> Hi guys,
>>>
>>> Earlier today, Java6 compatibility was broken
>>> (http://gerrit.ovirt.org/#/c/9430/).
>>> This was picked up on pretty quickly, and easily fixed
>>> (http://gerrit.ovirt.org/#/c/9666/).
>>>
>>> However, I think this is a good opportunity to revisit our policy
>>> towards Java 7.
>>> Currently, we have an "odd" setup, where we recommend running a
>>> compiling /with/ Java 7 [1] but comply to Java 6 language level
>>> [2] and JDK [3].
>>> Inevitably, mistakes like the that happened today will happen.
>>>
>>> I know we're holding back due to GWT issues, but looking forward to
>>> oVirt 4.0, is Java 7 on our roadmap?
>>
>> what was the GWT issue?
> GWT 2.3 that we use doesn't support java 7 syntax.
> The latest version, 2.5, doesn't either.
> I'm not sure, however, this is a good reason to enforce the java 6 limitation on the entire project (including backed, rest, etc.)

i agree. you can limit it to the parts used by GWT for now.


From alonbl at redhat.com  Tue Dec  4 13:53:49 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Tue, 4 Dec 2012 08:53:49 -0500 (EST)
Subject: [Engine-devel] Revisiting Java7
In-Reply-To: <50BDDAED.6040007@redhat.com>
Message-ID: <1401320037.3175423.1354629229583.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Allon Mureinik" <amureini at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Tuesday, December 4, 2012 1:13:49 PM
> Subject: Re: [Engine-devel] Revisiting Java7
> 
> On 12/04/2012 11:20 AM, Allon Mureinik wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Itamar Heim" <iheim at redhat.com>
> >> To: "Allon Mureinik" <amureini at redhat.com>
> >> Cc: "engine-devel" <engine-devel at ovirt.org>
> >> Sent: Tuesday, December 4, 2012 1:40:10 AM
> >> Subject: Re: [Engine-devel] Revisiting Java7
> >>
> >> On 12/03/2012 04:03 PM, Allon Mureinik wrote:
> >>> Hi guys,
> >>>
> >>> Earlier today, Java6 compatibility was broken
> >>> (http://gerrit.ovirt.org/#/c/9430/).
> >>> This was picked up on pretty quickly, and easily fixed
> >>> (http://gerrit.ovirt.org/#/c/9666/).
> >>>
> >>> However, I think this is a good opportunity to revisit our policy
> >>> towards Java 7.
> >>> Currently, we have an "odd" setup, where we recommend running a
> >>> compiling /with/ Java 7 [1] but comply to Java 6 language level
> >>> [2] and JDK [3].
> >>> Inevitably, mistakes like the that happened today will happen.
> >>>
> >>> I know we're holding back due to GWT issues, but looking forward
> >>> to
> >>> oVirt 4.0, is Java 7 on our roadmap?
> >>
> >> what was the GWT issue?
> > GWT 2.3 that we use doesn't support java 7 syntax.
> > The latest version, 2.5, doesn't either.
> > I'm not sure, however, this is a good reason to enforce the java 6
> > limitation on the entire project (including backed, rest, etc.)
> 
> i agree. you can limit it to the parts used by GWT for now.

The major advantage to move to java7 is the try-with-resources statement.

If we can do this for 3.2 it will be great. From what I understand from Allon, the only noisy change is to move the common and compat out of the backend directory into its own top-level directory, so we can apply a different maven policy on these two easily.

This way the common/compat and frontend will be built using 1.6 and backend will be built using 1.7.

Alon


From sanjal at redhat.com  Tue Dec  4 14:23:53 2012
From: sanjal at redhat.com (Shireesh Anjal)
Date: Tue, 4 Dec 2012 09:23:53 -0500 (EST)
Subject: [Engine-devel] engine-config utility
In-Reply-To: <850941977.3152966.1354619512987.JavaMail.root@redhat.com>
Message-ID: <1763200990.13696265.1354631033125.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Alon Bar-Lev" <alonbl at redhat.com>
> To: "engine-devel" <engine-devel at ovirt.org>
> Sent: Tuesday, December 4, 2012 4:41:52 PM
> Subject: [Engine-devel] engine-config utility
> 
> Hello,
> 
> I had interesting discussion with Eli regarding the engine-config
> utility and recent commit.
> 
> Is the following claim correct?
> 
> """Every change of a variable by engine-config cannot break engine"""
> 
> This issue came up because we used to export the CABaseDirectory
> parameter to engine-config. Changing it without attending to copy
> pki files may lead to failure. A change was already committed to
> remove the ability to update this parameter.
> 
> Most applications I know, do allow to change variable that in certain
> cases may break the application. The assumption that I am usually
> familiar with is that the target audience of these kind of tools are
> system admins or integrators, which "should know what they are
> doing", so it is OK to expose parameters that may lead to
> application breakage, as the ability to customize the application is
> important.

I agree with Alon. I don't think it is feasible to prevent the user from every possible update of configuration that can break the system.

> 
> I am carious what other think in this regard.
> 
> Regards,
> Alon Bar-Lev.
> 
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From sanjal at redhat.com  Tue Dec  4 14:48:50 2012
From: sanjal at redhat.com (Shireesh Anjal)
Date: Tue, 4 Dec 2012 09:48:50 -0500 (EST)
Subject: [Engine-devel] Task cancelation feature
In-Reply-To: <20121204093813.GB18178@redhat.com>
Message-ID: <1449153029.13708703.1354632530685.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Dan Kenigsberg" <danken at redhat.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Tuesday, December 4, 2012 3:08:13 PM
> Subject: Re: [Engine-devel] Task cancelation feature
> 
> On Mon, Dec 03, 2012 at 12:15:07PM -0500, Saggi Mizrahi wrote:
> > VDSM tasks are changing to something completely different.
> > It's still under discussion but the general direction is that:
> > - TaskIDs will be decided by the caller.
> > - VDSM can start tasks on it's own
> > - There will be no distinction between async tasks and sync tasks.
> > Everything is always async.
> > - There will be no cleanTask() when tasks are done they return
> > result to the caller and disappear immediately.
> 
> I'm not sure I understand the motivation for the latter change. I
> kinda
> like the unix process semantics, were the return code of a process is
> kept with its id after the process ends, until the process parent
> calls
> wait(2). Otherwise, how can the caller tell why its task has failed?
> 
> For example, I'd like to see vmCreate using async tasks like that.
> vmCreate returns immediately, and a vdsm task is tracking the vm
> creation. If something bad happens, the information about the failure
> can be polled by the Engine that created the vm (or a new Engine
> instance, after an Engine crash).
> 
> Similarily, we may need to make setupNetwork asynchronous, since we
> depend on dhclient, which may take a lot of time to finish.
> 
> Have these future use cases been debated?

There are also a few requirements from gluster perspective that may need enhancements in vdsm as well as engine.
- Tasks are created and managed by glusterfs and not by vdsm.
- Gluster tasks are not bound to a particular host, but are cluster-wide, and their status can be checked from any of the hosts of the cluster.
- Concept of SPM does not come into picture in case of gluster clusters/hosts
- Apart from starting, aborting and checking status, some of the gluster tasks support additional actions like pause, resume and commit.

Based on some of the telephonic conversations with maintainers of engine and vdsm, we were planning to enhance the existing task management as follows:
- Enhance the getAllTasks verb in vdsm to accept one or more tags for filtering tasks. (http://gerrit.ovirt.org/7579)
- Currently all tasks created in vdsm through requests from engine have the 'spm' tag as they are SPM tasks.
- Introduce a new field, say task_target in engine, which indicates what kind of a task it is. Possible values:
   - SPM (SPM tasks)
   - CLUSTER (cluster-wide tasks e.g. gluster tasks)
   - HOST (tasks specific to a particular host e.g. format a disk on a host)
- Enhance the async task manager in engine, to fetch details of all types of tasks, by sending appropriate tags to the new getAllTasks verb. (At present it fetches only SPM tasks)
- Once all the tasks are fetched, rest of the processing (updating status) remains same as before

I would like to know whether we should stop working on above approach in case the new design is coming up immediately. If yes, we should make sure that the new design is capable of handling the gluster tasks as well.

In case it is too far in the future,

1) We will start working with above approach. Any comments/suggestions/concerns are welcome.
2) Instead of focusing on just 'cancellation', we should try and come up with a more generic approach which can help in easily supporting more actions like pause, resume, commit, etc. (I haven't yet gone through the feature page, and please pardon me if this is already taken care)


> 
> Dan.
> > 
> > Also, some stuff you consider tasks will no longer be tasks any
> > more.
> > For instance, copying and image will finish successfully once VDSM
> > registers the operation for with the storage subsystem and creates
> > the image handle.
> > After that the status of the copy is bound to the status of the new
> > image and is tracked that way.
> > This means that the thing you track when you do copyImage() is
> > actually the creation of the image handle and the metadata to make
> > it usable.
> > After that is done any host can query the state of the new image by
> > using the image ID and not the task Id which was deprecated.
> > This will be true for all storage operations.
> > 
> > 
> > ----- Original Message -----
> > > From: "Michael Kublin" <mkublin at redhat.com>
> > > To: "engine-devel" <engine-devel at ovirt.org>
> > > Sent: Monday, December 3, 2012 4:19:48 AM
> > > Subject: [Engine-devel] Task cancelation feature
> > > 
> > > Hi, I created a wiki page with design of task cancellation
> > > feature.
> > > The url is : http://www.ovirt.org/Features/TaskManagerCancelTask
> > > I can not call these design, I have not any requirements , except
> > > a
> > > name of the feature,
> > > so my wiki doesn't contains anything except open questions.
> > > Also, I think that it is impossible to make a good feature based
> > > on
> > > very problematic infrastructure,
> > > I think before we should fix all our infrastructure problems, and
> > > after that to add any cancellation task
> > > feature will be a meter of couple hours of work
> > > 
> > > Regards Michael
> > > _______________________________________________
> > > Engine-devel mailing list
> > > Engine-devel at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > 
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From smizrahi at redhat.com  Tue Dec  4 14:55:56 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Tue, 4 Dec 2012 09:55:56 -0500 (EST)
Subject: [Engine-devel] Task cancelation feature
In-Reply-To: <20121204093813.GB18178@redhat.com>
Message-ID: <1457277702.13712093.1354632956214.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Dan Kenigsberg" <danken at redhat.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "Michael Kublin" <mkublin at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Tuesday, December 4, 2012 4:38:13 AM
> Subject: Re: [Engine-devel] Task cancelation feature
> 
> On Mon, Dec 03, 2012 at 12:15:07PM -0500, Saggi Mizrahi wrote:
> > VDSM tasks are changing to something completely different.
> > It's still under discussion but the general direction is that:
> > - TaskIDs will be decided by the caller.
> > - VDSM can start tasks on it's own
> > - There will be no distinction between async tasks and sync tasks.
> > Everything is always async.
> > - There will be no cleanTask() when tasks are done they return
> > result to the caller and disappear immediately.
> 
> I'm not sure I understand the motivation for the latter change. I
> kinda
> like the unix process semantics, were the return code of a process is
> kept with its id after the process ends, until the process parent
> calls
> wait(2). Otherwise, how can the caller tell why its task has failed?
> 
> For example, I'd like to see vmCreate using async tasks like that.
> vmCreate returns immediately, and a vdsm task is tracking the vm
> creation. If something bad happens, the information about the failure
> can be polled by the Engine that created the vm (or a new Engine
> instance, after an Engine crash).
You can have clean task only when there is a clear owner for a task.
This is not the case because it has been decided VDSM can be used by more then one client.
I know that people kind of like how convenient it is but it's problematic from the same reason that the unix method is problematic.
- If you die you can no longer track the task.
- If the kid dies and restarts itself you no longer can track the task.
- If the task is started by another process you can no longer track it.
- If you happen to die and not restart all your tasks stay there until someone clears them and it's impossible to know for sure when that is safe.

We already have issues because this semantic, doesn't hold up for a lot of the error cases. It doesn't hold up at all when there are more then one task instigators.
Because in TCP and and AMQP you are guaranteed to get the result message as long as you are connected to the bus you will always know the result unless you or VDSM crashed.
When you restart, you have to write code to sync up in case something happen while you where down anyway.
When VDSM restart\errors you need to write code to check how the task ended anyway because even "failed" tasks can finish successfully.
Also, in the new storage, VDSM restarts usually don't fail big operations and the can continue even on another host.
(We send the commit write and the storage fails just after the request was sent but we didn't get the write ack. The task is committed but it failed on timeout.)
> 
> Similarily, we may need to make setupNetwork asynchronous, since we
> depend on dhclient, which may take a lot of time to finish.
As I said, everything *is* asynchronous. It's impossible to have synchronous calls because you can't have synchronous calls on AMQP\TCP.
Because the engine is responsible for generating the ID there is no need for the silly 2 step approach we have now.
> 
> Have these future use cases been debated?
Yes, If you convince everyone to move away from message based communication and multiplexing requests on the same TCP connection (or go back to only HTTP) you can get synchronous tasks back.
If you can find a way to make "clean task" make sense and work then you can have that back too.
I think that after thinking about it you'll realize that considering all the possible error flows, sync-up on failure is the most robust and flexible way to go.
Further more, I doubt you can find a system where you never need to sync-up on the state of VDSM so might as well make that the only flow.
> 
> Dan.
> > 
> > Also, some stuff you consider tasks will no longer be tasks any
> > more.
> > For instance, copying and image will finish successfully once VDSM
> > registers the operation for with the storage subsystem and creates
> > the image handle.
> > After that the status of the copy is bound to the status of the new
> > image and is tracked that way.
> > This means that the thing you track when you do copyImage() is
> > actually the creation of the image handle and the metadata to make
> > it usable.
> > After that is done any host can query the state of the new image by
> > using the image ID and not the task Id which was deprecated.
> > This will be true for all storage operations.
> > 
> > 
> > ----- Original Message -----
> > > From: "Michael Kublin" <mkublin at redhat.com>
> > > To: "engine-devel" <engine-devel at ovirt.org>
> > > Sent: Monday, December 3, 2012 4:19:48 AM
> > > Subject: [Engine-devel] Task cancelation feature
> > > 
> > > Hi, I created a wiki page with design of task cancellation
> > > feature.
> > > The url is : http://www.ovirt.org/Features/TaskManagerCancelTask
> > > I can not call these design, I have not any requirements , except
> > > a
> > > name of the feature,
> > > so my wiki doesn't contains anything except open questions.
> > > Also, I think that it is impossible to make a good feature based
> > > on
> > > very problematic infrastructure,
> > > I think before we should fix all our infrastructure problems, and
> > > after that to add any cancellation task
> > > feature will be a meter of couple hours of work
> > > 
> > > Regards Michael
> > > _______________________________________________
> > > Engine-devel mailing list
> > > Engine-devel at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > 
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From smizrahi at redhat.com  Tue Dec  4 15:35:01 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Tue, 4 Dec 2012 10:35:01 -0500 (EST)
Subject: [Engine-devel] VDSM tasks, the future
In-Reply-To: <1681541014.13716926.1354633521990.JavaMail.root@redhat.com>
Message-ID: <71823151.13736106.1354635301538.JavaMail.root@redhat.com>

Because I started hinting about how VDSM tasks are going to look going forward I thought it's better I'll just write everything in an email so we can talk about it in context.
This is not set in stone and I'm still debating things myself but it's very close to being done.

- Everything is asynchronous.
  The nature of message based communication is that you can't have synchronous operations.
  This is not really debatable because it's just how TCP\AMQP\<messaging> works.

- Task IDs will be decided by the caller.
  This is how json-rpc works and also makes sense because no the engine can track the task without needing to have a stage where we give it the task ID back.
  IDs are reusable as long as no one else is using them at the time so they can be used for synchronizing operations between clients (making sure a command is
  only executed once on a specific host without locking).

- Tasks are transient
  If VDSM restarts it forgets all the task information.
  There are 2 ways to have persistent tasks:
  1. The task creates an object that you can continue work on in VDSM.
     The new storage does that by the fact that copyImage() returns one the target volume has been created but before the data has been fully copied.
     From that moment on the stat of the copy can be queried from any host using getImageStatus() and the specific copy operation can be queried with getTaskStatus() on the host performing it.
     After VDSM crashes, depending on policy, either VDSM will create a new task to continue the copy or someone else will send a command to continue the operation and that will be a new task.
  2. VDSM tasks just start other operations track-able not through the task interface. For example Gluster.
     gluster.startVolumeRebalance() will return once it has been registered with Gluster.
     glster.getOperationStatuses() will return the state of the operation from any host.
     Each call is a task in itself.
  
- No task tags.
  They are silly and the caller can mangle whatever in the task ID if he really wants to tag tasks.

- No explicit recovery stage.
  VDSM will be crash-only, there should be efforts to make everything crash-safe.
  If that is problematic, in case of networking, VDSM will recover on start without having a task for it.

- No clean Task:
  Tasks can be started by any number of hosts this means that there is no way to own all tasks.
  There could be cases where VDSM starts tasks on it's own and thus they have no owner at all.
  The caller needs to continually track the state of VDSM. We will have brodcasted events to mitigate polling.

- No revert
  Impossible to implement safely.

- No SPM\HSM tasks
  SPM\SDM is no longer necessary for all domain types (only for type).
  What used to be SPM tasks, or tasks that persist and can be restarted on other hosts is talked about in previous bullet points.


From iheim at redhat.com  Tue Dec  4 16:43:24 2012
From: iheim at redhat.com (Itamar Heim)
Date: Tue, 04 Dec 2012 18:43:24 +0200
Subject: [Engine-devel] Revisiting Java7
In-Reply-To: <1401320037.3175423.1354629229583.JavaMail.root@redhat.com>
References: <1401320037.3175423.1354629229583.JavaMail.root@redhat.com>
Message-ID: <50BE282C.5080308@redhat.com>

On 12/04/2012 03:53 PM, Alon Bar-Lev wrote:
>
>
> ----- Original Message -----
>> From: "Itamar Heim" <iheim at redhat.com>
>> To: "Allon Mureinik" <amureini at redhat.com>
>> Cc: "engine-devel" <engine-devel at ovirt.org>
>> Sent: Tuesday, December 4, 2012 1:13:49 PM
>> Subject: Re: [Engine-devel] Revisiting Java7
>>
>> On 12/04/2012 11:20 AM, Allon Mureinik wrote:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Itamar Heim" <iheim at redhat.com>
>>>> To: "Allon Mureinik" <amureini at redhat.com>
>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>> Sent: Tuesday, December 4, 2012 1:40:10 AM
>>>> Subject: Re: [Engine-devel] Revisiting Java7
>>>>
>>>> On 12/03/2012 04:03 PM, Allon Mureinik wrote:
>>>>> Hi guys,
>>>>>
>>>>> Earlier today, Java6 compatibility was broken
>>>>> (http://gerrit.ovirt.org/#/c/9430/).
>>>>> This was picked up on pretty quickly, and easily fixed
>>>>> (http://gerrit.ovirt.org/#/c/9666/).
>>>>>
>>>>> However, I think this is a good opportunity to revisit our policy
>>>>> towards Java 7.
>>>>> Currently, we have an "odd" setup, where we recommend running a
>>>>> compiling /with/ Java 7 [1] but comply to Java 6 language level
>>>>> [2] and JDK [3].
>>>>> Inevitably, mistakes like the that happened today will happen.
>>>>>
>>>>> I know we're holding back due to GWT issues, but looking forward
>>>>> to
>>>>> oVirt 4.0, is Java 7 on our roadmap?
>>>>
>>>> what was the GWT issue?
>>> GWT 2.3 that we use doesn't support java 7 syntax.
>>> The latest version, 2.5, doesn't either.
>>> I'm not sure, however, this is a good reason to enforce the java 6
>>> limitation on the entire project (including backed, rest, etc.)
>>
>> i agree. you can limit it to the parts used by GWT for now.
>
> The major advantage to move to java7 is the try-with-resources statement.
>
> If we can do this for 3.2 it will be great. From what I understand from Allon, the only noisy change is to move the common and compat out of the backend directory into its own top-level directory, so we can apply a different maven policy on these two easily.
>
> This way the common/compat and frontend will be built using 1.6 and backend will be built using 1.7.

and searchbackend.
asaf, is there no easier way to enable java 7 compilation/enforcement 
except for these 3?


From emesika at redhat.com  Tue Dec  4 17:12:28 2012
From: emesika at redhat.com (Eli Mesika)
Date: Tue, 4 Dec 2012 12:12:28 -0500 (EST)
Subject: [Engine-devel] RHEVM 3.1 DB Schema diagram
In-Reply-To: <2119544101.18362996.1354640863420.JavaMail.root@redhat.com>
Message-ID: <485093285.18366122.1354641148339.JavaMail.root@redhat.com>

Hi

There were some requests on the users list to get our DB schema diagram.
After looking a while for a tool for generating that , I have found the DBSchema tool.

Please find attached the RHEVM 3.1 DB Schema diagram.

I intend to get it into the oVirt site but currently it's too large and does not fit well.

Regards

Eli Mesika
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rhevm-3-1.png
Type: image/png
Size: 530643 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121204/3492967f/attachment.png>

From ashakarc at redhat.com  Tue Dec  4 18:25:06 2012
From: ashakarc at redhat.com (Asaf Shakarchi)
Date: Tue, 4 Dec 2012 13:25:06 -0500 (EST)
Subject: [Engine-devel] Revisiting Java7
In-Reply-To: <50BE282C.5080308@redhat.com>
Message-ID: <1134847122.40563373.1354645506659.JavaMail.root@redhat.com>

----- Original Message -----
> On 12/04/2012 03:53 PM, Alon Bar-Lev wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Itamar Heim" <iheim at redhat.com>
> >> To: "Allon Mureinik" <amureini at redhat.com>
> >> Cc: "engine-devel" <engine-devel at ovirt.org>
> >> Sent: Tuesday, December 4, 2012 1:13:49 PM
> >> Subject: Re: [Engine-devel] Revisiting Java7
> >>
> >> On 12/04/2012 11:20 AM, Allon Mureinik wrote:
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Itamar Heim" <iheim at redhat.com>
> >>>> To: "Allon Mureinik" <amureini at redhat.com>
> >>>> Cc: "engine-devel" <engine-devel at ovirt.org>
> >>>> Sent: Tuesday, December 4, 2012 1:40:10 AM
> >>>> Subject: Re: [Engine-devel] Revisiting Java7
> >>>>
> >>>> On 12/03/2012 04:03 PM, Allon Mureinik wrote:
> >>>>> Hi guys,
> >>>>>
> >>>>> Earlier today, Java6 compatibility was broken
> >>>>> (http://gerrit.ovirt.org/#/c/9430/).
> >>>>> This was picked up on pretty quickly, and easily fixed
> >>>>> (http://gerrit.ovirt.org/#/c/9666/).
> >>>>>
> >>>>> However, I think this is a good opportunity to revisit our
> >>>>> policy
> >>>>> towards Java 7.
> >>>>> Currently, we have an "odd" setup, where we recommend running a
> >>>>> compiling /with/ Java 7 [1] but comply to Java 6 language level
> >>>>> [2] and JDK [3].
> >>>>> Inevitably, mistakes like the that happened today will happen.
> >>>>>
> >>>>> I know we're holding back due to GWT issues, but looking
> >>>>> forward
> >>>>> to
> >>>>> oVirt 4.0, is Java 7 on our roadmap?
> >>>>
> >>>> what was the GWT issue?
> >>> GWT 2.3 that we use doesn't support java 7 syntax.
> >>> The latest version, 2.5, doesn't either.
> >>> I'm not sure, however, this is a good reason to enforce the java
> >>> 6
> >>> limitation on the entire project (including backed, rest, etc.)
> >>
> >> i agree. you can limit it to the parts used by GWT for now.
> >
> > The major advantage to move to java7 is the try-with-resources
> > statement.
> >
> > If we can do this for 3.2 it will be great. From what I understand
> > from Allon, the only noisy change is to move the common and compat
> > out of the backend directory into its own top-level directory, so
> > we can apply a different maven policy on these two easily.
> >
> > This way the common/compat and frontend will be built using 1.6 and
> > backend will be built using 1.7.
> 
> and searchbackend.
> asaf, is there no easier way to enable java 7 compilation/enforcement
> except for these 3?

In the long run consider having a project layer such as :
Frontend / Backend / Shared [common/compat/searchbackend] , this will allow us to use plugins inheritance cleanly [regarding versions],

Of course inheritance is nicer but it requires a lot of movements (maybe in v4?)


For now its easy - we can just use the maven-compiler to define source/target of [v1.7] for the root project, while defining explicitly different versions per frontend/shared module [v1.6]
Checkstyles should be upgraded as well.
> 


From iheim at redhat.com  Tue Dec  4 20:50:09 2012
From: iheim at redhat.com (Itamar Heim)
Date: Tue, 04 Dec 2012 22:50:09 +0200
Subject: [Engine-devel] engine-config utility
In-Reply-To: <1763200990.13696265.1354631033125.JavaMail.root@redhat.com>
References: <1763200990.13696265.1354631033125.JavaMail.root@redhat.com>
Message-ID: <50BE6201.7060705@redhat.com>

On 12/04/2012 04:23 PM, Shireesh Anjal wrote:
>
>
> ----- Original Message -----
>> From: "Alon Bar-Lev" <alonbl at redhat.com>
>> To: "engine-devel" <engine-devel at ovirt.org>
>> Sent: Tuesday, December 4, 2012 4:41:52 PM
>> Subject: [Engine-devel] engine-config utility
>>
>> Hello,
>>
>> I had interesting discussion with Eli regarding the engine-config
>> utility and recent commit.
>>
>> Is the following claim correct?
>>
>> """Every change of a variable by engine-config cannot break engine"""
>>
>> This issue came up because we used to export the CABaseDirectory
>> parameter to engine-config. Changing it without attending to copy
>> pki files may lead to failure. A change was already committed to
>> remove the ability to update this parameter.
>>
>> Most applications I know, do allow to change variable that in certain
>> cases may break the application. The assumption that I am usually
>> familiar with is that the target audience of these kind of tools are
>> system admins or integrators, which "should know what they are
>> doing", so it is OK to expose parameters that may lead to
>> application breakage, as the ability to customize the application is
>> important.
>
> I agree with Alon. I don't think it is feasible to prevent the user from every possible update of configuration that can break the system.

I agree in general, and i don't think we should extend the config 
utility to test such things.
but some basic sanity should be done (config key is legit, config value 
is validated for type safety/regex/enum, etc.

>
>>
>> I am carious what other think in this regard.
>>
>> Regards,
>> Alon Bar-Lev.
>>
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>>
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
>




From agl at us.ibm.com  Tue Dec  4 20:50:28 2012
From: agl at us.ibm.com (Adam Litke)
Date: Tue, 4 Dec 2012 14:50:28 -0600
Subject: [Engine-devel] VDSM tasks, the future
In-Reply-To: <71823151.13736106.1354635301538.JavaMail.root@redhat.com>
References: <1681541014.13716926.1354633521990.JavaMail.root@redhat.com>
	<71823151.13736106.1354635301538.JavaMail.root@redhat.com>
Message-ID: <20121204205028.GM4798@aglitke>

On Tue, Dec 04, 2012 at 10:35:01AM -0500, Saggi Mizrahi wrote:
> Because I started hinting about how VDSM tasks are going to look going forward
> I thought it's better I'll just write everything in an email so we can talk
> about it in context.  This is not set in stone and I'm still debating things
> myself but it's very close to being done.

Don't debate them yourself, debate them here!  Even better, propose your idea in
schema form to show how a command might work exactly.

> - Everything is asynchronous.  The nature of message based communication is
> that you can't have synchronous operations.  This is not really debatable
> because it's just how TCP\AMQP\<messaging> works.

Can you show how a traditionally synchronous command might work?  Let's take
Host.getVmList as an example.

> - Task IDs will be decided by the caller.  This is how json-rpc works and also
> makes sense because no the engine can track the task without needing to have a
> stage where we give it the task ID back.  IDs are reusable as long as no one
> else is using them at the time so they can be used for synchronizing
> operations between clients (making sure a command is only executed once on a
> specific host without locking).
> 
> - Tasks are transient If VDSM restarts it forgets all the task information.
> There are 2 ways to have persistent tasks: 1. The task creates an object that
> you can continue work on in VDSM.  The new storage does that by the fact that
> copyImage() returns one the target volume has been created but before the data
> has been fully copied.  From that moment on the stat of the copy can be
> queried from any host using getImageStatus() and the specific copy operation
> can be queried with getTaskStatus() on the host performing it.  After VDSM
> crashes, depending on policy, either VDSM will create a new task to continue
> the copy or someone else will send a command to continue the operation and
> that will be a new task.  2. VDSM tasks just start other operations track-able
> not through the task interface. For example Gluster.
> gluster.startVolumeRebalance() will return once it has been registered with
> Gluster.  glster.getOperationStatuses() will return the state of the operation
> from any host.  Each call is a task in itself.

I worry about this approach because every command has a different semantic for
checking progress.  For migration, we have to check VM status on the src and
dest hosts.  For image copy we need to use a special status call on the dest
image.  It would be nice if there was a unified method for checking on an
operation.  Maybe that can be completion events.

Client:               vdsm:
-------               -----

Image.copy(...)  -->
                 <--  Operation Started
Wait for event   ...
                 <--  Event: Operation <id> done <code>

For an early error:

Client:               vdsm:
-------               -----

Image.copy(...)  -->
                 <--  Error: <code>


> - No task tags.  They are silly and the caller can mangle whatever in the task
> ID if he really wants to tag tasks.

Yes.  Agreed.

> - No explicit recovery stage.  VDSM will be crash-only, there should be
> efforts to make everything crash-safe.  If that is problematic, in case of
> networking, VDSM will recover on start without having a task for it.

How does this work in practice for something like creating a new image from a
template?

> - No clean Task: Tasks can be started by any number of hosts this means that
> there is no way to own all tasks.  There could be cases where VDSM starts
> tasks on it's own and thus they have no owner at all.  The caller needs to
> continually track the state of VDSM. We will have brodcasted events to
> mitigate polling.

If a disconnected client might have missed a completion event, it will need to
check state.  This means each async operation that changes state must document a
proceedure for checking progress of a potentially ongoing operation.  For
Image.copy, that process would be to lookup the new image and check its state.

> - No revert Impossible to implement safely.

How do the engine folks feel about this?  I am ok with it :)

> - No SPM\HSM tasks SPM\SDM is no longer necessary for all domain types (only
> for type).  What used to be SPM tasks, or tasks that persist and can be
> restarted on other hosts is talked about in previous bullet points.
> 
A nice simplification.


-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center



From alonbl at redhat.com  Tue Dec  4 20:55:48 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Tue, 4 Dec 2012 15:55:48 -0500 (EST)
Subject: [Engine-devel] engine-config utility
In-Reply-To: <50BE6201.7060705@redhat.com>
Message-ID: <330186676.3293384.1354654548598.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Shireesh Anjal" <sanjal at redhat.com>
> Cc: "Alon Bar-Lev" <alonbl at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Tuesday, December 4, 2012 10:50:09 PM
> Subject: Re: [Engine-devel] engine-config utility
> 
> On 12/04/2012 04:23 PM, Shireesh Anjal wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Alon Bar-Lev" <alonbl at redhat.com>
> >> To: "engine-devel" <engine-devel at ovirt.org>
> >> Sent: Tuesday, December 4, 2012 4:41:52 PM
> >> Subject: [Engine-devel] engine-config utility
> >>
> >> Hello,
> >>
> >> I had interesting discussion with Eli regarding the engine-config
> >> utility and recent commit.
> >>
> >> Is the following claim correct?
> >>
> >> """Every change of a variable by engine-config cannot break
> >> engine"""
> >>
> >> This issue came up because we used to export the CABaseDirectory
> >> parameter to engine-config. Changing it without attending to copy
> >> pki files may lead to failure. A change was already committed to
> >> remove the ability to update this parameter.
> >>
> >> Most applications I know, do allow to change variable that in
> >> certain
> >> cases may break the application. The assumption that I am usually
> >> familiar with is that the target audience of these kind of tools
> >> are
> >> system admins or integrators, which "should know what they are
> >> doing", so it is OK to expose parameters that may lead to
> >> application breakage, as the ability to customize the application
> >> is
> >> important.
> >
> > I agree with Alon. I don't think it is feasible to prevent the user
> > from every possible update of configuration that can break the
> > system.
> 
> I agree in general, and i don't think we should extend the config
> utility to test such things.
> but some basic sanity should be done (config key is legit, config
> value
> is validated for type safety/regex/enum, etc.

Right.
if Value1+Value2 = crash is not something that should be checked.
if Value1=Path while Path does not exist or contains wrong files is not something we should check.

> 
> >
> >>
> >> I am carious what other think in this regard.
> >>
> >> Regards,
> >> Alon Bar-Lev.
> >>
> >> _______________________________________________
> >> Engine-devel mailing list
> >> Engine-devel at ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/engine-devel
> >>
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> >
> 
> 
> 


From smizrahi at redhat.com  Tue Dec  4 21:52:40 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Tue, 4 Dec 2012 16:52:40 -0500 (EST)
Subject: [Engine-devel] RFC: New Storage API
In-Reply-To: <1351630053.13865580.1354653355435.JavaMail.root@redhat.com>
Message-ID: <1661336497.13921817.1354657960654.JavaMail.root@redhat.com>

I've been throwing a lot of bits out about the new storage API and I think it's time to talk a bit.
I will purposefully try and keep implementation details away and concentrate about how the API looks and how you use it.

First major change is in terminology, there is no long a storage domain but a storage repository.
This change is done because so many things are already called domain in the system and this will make things less confusing for new-commers with a libvirt background.

One other changes is that repositories no longer have a UUID.
The UUID was only used in the pool members manifest and is no longer needed.


connectStorageRepository(repoId, repoFormat, connectionParameters={}):
repoId - is a transient name that will be used to refer to the connected domain, it is not persisted and doesn't have to be the same across the cluster.
repoFormat - Similar to what used to be type (eg. localfs-1.0, nfs-3.4, clvm-1.2).
connectionParameters - This is format specific and will used to tell VDSM how to connect to the repo.

disconnectStorageRepository(self, repoId):


In the new API there are only images, some images are mutable and some are not.
mutable images are also called VirtualDisks
immutable images are also called Snapshots

There are no explicit templates, you can create as many images as you want from any snapshot.

There are 4 major image operations:


createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
                  userData={}, options={}):

targetRepoId - ID of a connected repo where the disk will be created
size - The size of the image you wish to create
baseSnapshotId - the ID of the snapshot you want the base the new virtual disk on
userData - optional data that will be attached to the new VD, could be anything that the user desires.
options - options to modify VDSMs default behavior

returns the id of the new VD

createSnapshot(targetRepoId, baseVirtualDiskId,
               userData={}, options={}):
targetRepoId - The ID of a connected repo where the new sanpshot will be created and the original image exists as well.
size - The size of the image you wish to create
baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want to snapshot
userData - optional data that will be attached to the new Snapshot, could be anything that the user desires.
options - options to modify VDSMs default behavior

returns the id of the new Snapshot

copyImage(targetRepoId, imageId, baseImageId=None, userData={}, options={})
targetRepoId - The ID of a connected repo where the new image will be created
imageId - The image you wish to copy
baseImageId - if specified, the new image will contain only the diff between image and Id.
              If None the new image will contain all the bits of image Id. This can be used to copy partial parts of images for export.
userData - optional data that will be attached to the new image, could be anything that the user desires.
options - options to modify VDSMs default behavior

return the Id of the new image. In case of copying an immutable image the ID will be identical to the original image as they contain the same data. However the user should not assume that and always use the value returned from the method.

removeImage(repositoryId, imageId, options={}):
repositoryId - The ID of a connected repo where the image to delete resides
imageId - The id of the image you wish to delete.


----
getImageStatus(repositoryId, imageId)
repositoryId - The ID of a connected repo where the image to check resides
imageId - The id of the image you wish to check.

All operations return once the operations has been committed to disk NOT when the operation actually completes.
This is done so that:
- operation come to a stable state as quickly as possible.
- In case where there is an SDM, only small portion of the operation actually needs to be performed on the SDM host.
- No matter how many times the operation fails and on how many hosts, you can always resume the operation and choose when to do it.
- You can stop an operation at any time and remove the resulting object making a distinction between "stop because the host is overloaded" to "I don't want that image"

This means that after calling any operation that creates a new image the user must then call getImageStatus() to check what is the status of the image.
The status of the image can be either optimized, degraded, or broken.
"Optimized" means that the image is available and you can run VMs of it.
"Degraded" means that the image is available and will run VMs but it might be a better way VDSM can represent the underlying data. 
"Broken" means that the image can't be used at the moment, probably because not all the data has been set up on the volume.

Apart from that VDSM will also return the last persisted status information which will conatin
hostID - the last host to try and optimize of fix the image
stage - X/Y (eg. 1/10) the last persisted stage of the fix.
percent_complete - -1 or 0-100, the last persisted completion percentage of the aforementioned stage. -1 means that no progress is available for that operation.
last_error - This will only be filled if the operation failed because of something other then IO or a VDSM crash for obvious reasons.
             It will usually be set if the task was manually stopped

The user can either be satisfied with that information or as the host specified in host ID if it is still working on that image by checking it's running tasks.

checkStorageRepository(self, repositoryId, options={}):
A method to go over a storage repository and scan for any existing problems. This includes degraded\broken images and deleted images that have no yet been physically deleted\merged.
It returns a list of Fix objects.
Fix objects come in 4 types:
clean - cleans data, run them to get more space.
optimize - run them to optimize a degraded image
merge - Merges two images together. Doing this sometimes
        makes more images ready optimizing or cleaning.
        The reason it is different from optimize is that
        unmerged images are considered optimized.
mend - mends a broken image

The user can read these types and prioritize fixes. Fixes also contain opaque FIX data and they should be sent as received to
fixStorageRepository(self, repositoryId, fix, options={}):

That will start a fix operation.


All major operations automatically start the appropriate "Fix" to bring the created object to an optimize\degraded state (the one that is quicker) unless one of the options is
AutoFix=False. This is only useful for repos that might not be able to create volumes on all hosts (SDM) but would like to have the actual IO distributed in the cluster.

Other common options is the strategy option:
It has currently 2 possible values
space and performance - In case VDSM has 2 ways of completing the same operation it will tell it to value one over the other. For example, whether to copy all the data or just create a qcow based of a snapshot.
The default is space.

You might have also noticed that it is never explicitly specified where to look for existing images. This is done purposefully, VDSM will always look in all connected repositories for existing objects.
For very large setups this might be problematic. To mitigate the problem you have these options:
participatingRepositories=[repoId, ...] which tell VDSM to narrow the search to just these repositories
and
imageHints={imgId: repoId} which will force VDSM to look for those image ID just in those repositories and fail if it doesn't find them there.


From smizrahi at redhat.com  Tue Dec  4 22:20:10 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Tue, 4 Dec 2012 17:20:10 -0500 (EST)
Subject: [Engine-devel] VDSM tasks, the future
In-Reply-To: <20121204205028.GM4798@aglitke>
Message-ID: <170195820.13932156.1354659610794.JavaMail.root@redhat.com>

As the only subsystem to use asynchronous tasks until now is the storage subsystem I suggest going over how
I suggest tackling task creation, task stop, task remove and task recovery.
Other subsystem can create similar mechanisms depending on their needs.

There is no way of avoiding it, different types of tasks need different ways of tracking\recovering from them.
network should always auto-recover because it can't get a "please fix" command if the network is down.
Storage on the other hand should never start operations on it's own because it might take up valuable resources from the host.
Tasks that need to be tracked on a single host, 2 hosts, or the entire cluster need to have their own APIs.
VM configuration never persist across reboots, networking sometimes persists and storage always persists.
This means that recovery procedures (from the managers point of view) need to be vastly different.
Add policy, resource allocation, and error flows you see that VDSM doesn't have nearly as much information to deal with the tasks.

----- Original Message -----
> From: "Adam Litke" <agl at us.ibm.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "VDSM Project Development" <vdsm-devel at lists.fedorahosted.org>, "engine-devel" <engine-devel at ovirt.org>, "Ayal
> Baron" <abaron at redhat.com>, "Barak Azulay" <bazulay at redhat.com>, "Shireesh Anjal" <sanjal at redhat.com>
> Sent: Tuesday, December 4, 2012 3:50:28 PM
> Subject: Re: VDSM tasks, the future
> 
> On Tue, Dec 04, 2012 at 10:35:01AM -0500, Saggi Mizrahi wrote:
> > Because I started hinting about how VDSM tasks are going to look
> > going forward
> > I thought it's better I'll just write everything in an email so we
> > can talk
> > about it in context.  This is not set in stone and I'm still
> > debating things
> > myself but it's very close to being done.
> 
> Don't debate them yourself, debate them here!  Even better, propose
> your idea in
> schema form to show how a command might work exactly.
I don't like throwing ideas in the air
> 
> > - Everything is asynchronous.  The nature of message based
> > communication is
> > that you can't have synchronous operations.  This is not really
> > debatable
> > because it's just how TCP\AMQP\<messaging> works.
> 
> Can you show how a traditionally synchronous command might work?
>  Let's take
> Host.getVmList as an example.
The same as it works today, it's all a matter of how you wrap the transport layer.
You will send a json-rpc request and wait for a response with the same id.

As for the bindings, there are a lot of way we can tackle that.
Always wait for the response and simulate synchronous behavior.
Make every method return an object to track the task.
task = host.getVmList()
if not task.wait(1):
    task.cancel()
else:
    res = task.result()

Have it both ways (it's auto generated anyway) and have
list = host.getVmList()
task = host.getVmList_async()

Have a high level and low level interfaces.
host = host()
host.connect("tcp://host:3233")
req = host.sendRequest("123213", "getVmList", [])
if not req.wait(1):
   ....

shost = SynchHost(host)
shost.getVmList() # Actually wraps a request object
ahost = AsyncHost(host)
task = getVmList() # Actually wraps a request object
> 
> > - Task IDs will be decided by the caller.  This is how json-rpc
> > works and also
> > makes sense because no the engine can track the task without
> > needing to have a
> > stage where we give it the task ID back.  IDs are reusable as long
> > as no one
> > else is using them at the time so they can be used for
> > synchronizing
> > operations between clients (making sure a command is only executed
> > once on a
> > specific host without locking).
> > 
> > - Tasks are transient If VDSM restarts it forgets all the task
> > information.
> > There are 2 ways to have persistent tasks: 1. The task creates an
> > object that
> > you can continue work on in VDSM.  The new storage does that by the
> > fact that
> > copyImage() returns one the target volume has been created but
> > before the data
> > has been fully copied.  From that moment on the stat of the copy
> > can be
> > queried from any host using getImageStatus() and the specific copy
> > operation
> > can be queried with getTaskStatus() on the host performing it.
> >  After VDSM
> > crashes, depending on policy, either VDSM will create a new task to
> > continue
> > the copy or someone else will send a command to continue the
> > operation and
> > that will be a new task.  2. VDSM tasks just start other operations
> > track-able
> > not through the task interface. For example Gluster.
> > gluster.startVolumeRebalance() will return once it has been
> > registered with
> > Gluster.  glster.getOperationStatuses() will return the state of
> > the operation
> > from any host.  Each call is a task in itself.
> 
> I worry about this approach because every command has a different
> semantic for
> checking progress.  For migration, we have to check VM status on the
> src and
> dest hosts.  For image copy we need to use a special status call on
> the dest
> image.  It would be nice if there was a unified method for checking
> on an
> operation.  Maybe that can be completion events.
> 
> Client:               vdsm:
> -------               -----
> 
> Image.copy(...)  -->
>                  <--  Operation Started
> Wait for event   ...
>                  <--  Event: Operation <id> done <code>
> 
> For an early error:
> 
> Client:               vdsm:
> -------               -----
> 
> Image.copy(...)  -->
>                  <--  Error: <code>
> 
> 
The thing is that a lot of things need a different way of tracking their progress.
Storage have completely different semantics from network or VM operations.
> > - No task tags.  They are silly and the caller can mangle whatever
> > in the task
> > ID if he really wants to tag tasks.
> 
> Yes.  Agreed.
> 
> > - No explicit recovery stage.  VDSM will be crash-only, there
> > should be
> > efforts to make everything crash-safe.  If that is problematic, in
> > case of
> > networking, VDSM will recover on start without having a task for
> > it.
> 
> How does this work in practice for something like creating a new
> image from a
> template?
> 
> > - No clean Task: Tasks can be started by any number of hosts this
> > means that
> > there is no way to own all tasks.  There could be cases where VDSM
> > starts
> > tasks on it's own and thus they have no owner at all.  The caller
> > needs to
> > continually track the state of VDSM. We will have brodcasted events
> > to
> > mitigate polling.
> 
> If a disconnected client might have missed a completion event, it
> will need to
> check state.  This means each async operation that changes state must
> document a
> proceedure for checking progress of a potentially ongoing operation.
>  For
> Image.copy, that process would be to lookup the new image and check
> its state.
> 
> > - No revert Impossible to implement safely.
> 
> How do the engine folks feel about this?  I am ok with it :)
I don't care, unless they find a way to change they way logic works they can't have it.
The whole concept of recovery (as it is defined now) doesn't work in an HA cluster.
> 
> > - No SPM\HSM tasks SPM\SDM is no longer necessary for all domain
> > types (only
> > for type).  What used to be SPM tasks, or tasks that persist and
> > can be
> > restarted on other hosts is talked about in previous bullet points.
> > 
> A nice simplification.
> 
> 
> --
> Adam Litke <agl at us.ibm.com>
> IBM Linux Technology Center
> 
> 


From agl at us.ibm.com  Tue Dec  4 23:08:25 2012
From: agl at us.ibm.com (Adam Litke)
Date: Tue, 4 Dec 2012 17:08:25 -0600
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <1661336497.13921817.1354657960654.JavaMail.root@redhat.com>
References: <1351630053.13865580.1354653355435.JavaMail.root@redhat.com>
	<1661336497.13921817.1354657960654.JavaMail.root@redhat.com>
Message-ID: <20121204230825.GP4798@aglitke>

Thanks for sharing this.  It's nice to have something a little more concrete to
think about.  Just a few comments and questions inline to get some discussion
flowing.

On Tue, Dec 04, 2012 at 04:52:40PM -0500, Saggi Mizrahi wrote:
> I've been throwing a lot of bits out about the new storage API and I think it's time to talk a bit.
> I will purposefully try and keep implementation details away and concentrate about how the API looks and how you use it.
> 
> First major change is in terminology, there is no long a storage domain but a storage repository.
> This change is done because so many things are already called domain in the system and this will make things less confusing for new-commers with a libvirt background.
> 
> One other changes is that repositories no longer have a UUID.
> The UUID was only used in the pool members manifest and is no longer needed.
> 
> 
> connectStorageRepository(repoId, repoFormat, connectionParameters={}):

We should probably add an options/flags parameter for extension of all new
APIs.

> repoId - is a transient name that will be used to refer to the connected domain, it is not persisted and doesn't have to be the same across the cluster.
> repoFormat - Similar to what used to be type (eg. localfs-1.0, nfs-3.4, clvm-1.2).
> connectionParameters - This is format specific and will used to tell VDSM how to connect to the repo.
> 
> disconnectStorageRepository(self, repoId):

I assume 'self' is a mistake here.  Just want to clarify given all of the recent
talk about instances vs. namespaces.

> In the new API there are only images, some images are mutable and some are not.
> mutable images are also called VirtualDisks
> immutable images are also called Snapshots

By mutable you mean writable right?  Or does the word mutable imply more than
that?

> There are no explicit templates, you can create as many images as you want from any snapshot.
> 
> There are 4 major image operations:
> 
> 
> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
>                   userData={}, options={}):

Is userdata a 'StringMap'?

I will reopen the argument about an options dict vs a flags parameter.  I oppose
the dict for expansion because I think it causes APIs to devolve into a mess
where lots of arbitrary and not well thought out overrides are packed into the
dict over time.  A flags argument (in json and python it can be an enum array)
limits us to really switching flags on and off instead of passing arbitrary
data.

> targetRepoId - ID of a connected repo where the disk will be created
> size - The size of the image you wish to create
> baseSnapshotId - the ID of the snapshot you want the base the new virtual disk on
> userData - optional data that will be attached to the new VD, could be anything that the user desires.
> options - options to modify VDSMs default behavior
> 
> returns the id of the new VD
> 
> createSnapshot(targetRepoId, baseVirtualDiskId,
>                userData={}, options={}):
> targetRepoId - The ID of a connected repo where the new sanpshot will be created and the original image exists as well.
> size - The size of the image you wish to create

Why is this needed?  Doesn't the size of a snapshot have to be equal to its
base image?

> baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want to snapshot

Can you snapshot a snapshot?  In that case, this parameter should be called
baseImage.

> userData - optional data that will be attached to the new Snapshot, could be anything that the user desires.
> options - options to modify VDSMs default behavior
> 
> returns the id of the new Snapshot
> 
> copyImage(targetRepoId, imageId, baseImageId=None, userData={}, options={})
> targetRepoId - The ID of a connected repo where the new image will be created
> imageId - The image you wish to copy

Do we locate the sourceRepoId automatically based on the imageId?

> baseImageId - if specified, the new image will contain only the diff between image and Id.
>               If None the new image will contain all the bits of image Id. This can be used to copy partial parts of images for export.
> userData - optional data that will be attached to the new image, could be anything that the user desires.
> options - options to modify VDSMs default behavior
> 
> return the Id of the new image. In case of copying an immutable image the ID will be identical to the original image as they contain the same data. However the user should not assume that and always use the value returned from the method.
> 
> removeImage(repositoryId, imageId, options={}):
> repositoryId - The ID of a connected repo where the image to delete resides
> imageId - The id of the image you wish to delete.
> 
> ----
> getImageStatus(repositoryId, imageId)
> repositoryId - The ID of a connected repo where the image to check resides
> imageId - The id of the image you wish to check.

What is in this return value?  Is it a single enum indicating whether the image
is locked (being copied, etc.) or a list of detailed information (like
Volume.getInfo)?  (I see some more info below...)

> All operations return once the operations has been committed to disk NOT when the operation actually completes.
> This is done so that:
> - operation come to a stable state as quickly as possible.
> - In case where there is an SDM, only small portion of the operation actually needs to be performed on the SDM host.
> - No matter how many times the operation fails and on how many hosts, you can always resume the operation and choose when to do it.
> - You can stop an operation at any time and remove the resulting object making a distinction between "stop because the host is overloaded" to "I don't want that image"
> 
> This means that after calling any operation that creates a new image the user must then call getImageStatus() to check what is the status of the image.
> The status of the image can be either optimized, degraded, or broken.
> "Optimized" means that the image is available and you can run VMs of it.
> "Degraded" means that the image is available and will run VMs but it might be a better way VDSM can represent the underlying data. 
> "Broken" means that the image can't be used at the moment, probably because not all the data has been set up on the volume.
> 
> Apart from that VDSM will also return the last persisted status information which will conatin
> hostID - the last host to try and optimize of fix the image
> stage - X/Y (eg. 1/10) the last persisted stage of the fix.

Do you have some examples of what the stages would be?  I think these should be
defined in enums so that the user can check on what the individual stages mean.
What happens when the low level implementation of an operation changes?  The
meaning of the stages will change completely.

> percent_complete - -1 or 0-100, the last persisted completion percentage of the aforementioned stage. -1 means that no progress is available for that operation.

> last_error - This will only be filled if the operation failed because of something other then IO or a VDSM crash for obvious reasons.
>              It will usually be set if the task was manually stopped
> 
> The user can either be satisfied with that information or as the host specified in host ID if it is still working on that image by checking it's running tasks.
> 
> checkStorageRepository(self, repositoryId, options={}):
> A method to go over a storage repository and scan for any existing problems. This includes degraded\broken images and deleted images that have no yet been physically deleted\merged.
> It returns a list of Fix objects.
> Fix objects come in 4 types:
> clean - cleans data, run them to get more space.
> optimize - run them to optimize a degraded image

What is an example of a degraded image?

> merge - Merges two images together. Doing this sometimes
>         makes more images ready optimizing or cleaning.
>         The reason it is different from optimize is that
>         unmerged images are considered optimized.
> mend - mends a broken image

What does this mean?

> The user can read these types and prioritize fixes. Fixes also contain opaque FIX data and they should be sent as received to
> fixStorageRepository(self, repositoryId, fix, options={}):
> 
> That will start a fix operation.

Could we have an automatic fix mode where vdsm just does the right thing (for
most things)?

> All major operations automatically start the appropriate "Fix" to bring the created object to an optimize\degraded state (the one that is quicker) unless one of the options is
> AutoFix=False. This is only useful for repos that might not be able to create volumes on all hosts (SDM) but would like to have the actual IO distributed in the cluster.
> 
> Other common options is the strategy option:
> It has currently 2 possible values
> space and performance - In case VDSM has 2 ways of completing the same operation it will tell it to value one over the other. For example, whether to copy all the data or just create a qcow based of a snapshot.
> The default is space.

I like this a lot.

> You might have also noticed that it is never explicitly specified where to look for existing images. This is done purposefully, VDSM will always look in all connected repositories for existing objects.
> For very large setups this might be problematic. To mitigate the problem you have these options:
> participatingRepositories=[repoId, ...] which tell VDSM to narrow the search to just these repositories
> and
> imageHints={imgId: repoId} which will force VDSM to look for those image ID just in those repositories and fail if it doesn't find them there.

I would like to have a better way of specifying these optional parameters
without burying them in an options structure.  I will think a little more about
this.  Strategy can just be a two optional flags in a 'flags' argument.  For the
participatingRepositories and imageHints options, I think we need to use real
parameters.

-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center



From smizrahi at redhat.com  Wed Dec  5 00:00:33 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Tue, 4 Dec 2012 19:00:33 -0500 (EST)
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <20121204230825.GP4798@aglitke>
Message-ID: <1723725719.13957891.1354665633001.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Adam Litke" <agl at us.ibm.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "VDSM Project Development" <vdsm-devel at lists.fedorahosted.org>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Tuesday, December 4, 2012 6:08:25 PM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> Thanks for sharing this.  It's nice to have something a little more
> concrete to
> think about.  Just a few comments and questions inline to get some
> discussion
> flowing.
> 
> On Tue, Dec 04, 2012 at 04:52:40PM -0500, Saggi Mizrahi wrote:
> > I've been throwing a lot of bits out about the new storage API and
> > I think it's time to talk a bit.
> > I will purposefully try and keep implementation details away and
> > concentrate about how the API looks and how you use it.
> > 
> > First major change is in terminology, there is no long a storage
> > domain but a storage repository.
> > This change is done because so many things are already called
> > domain in the system and this will make things less confusing for
> > new-commers with a libvirt background.
> > 
> > One other changes is that repositories no longer have a UUID.
> > The UUID was only used in the pool members manifest and is no
> > longer needed.
> > 
> > 
> > connectStorageRepository(repoId, repoFormat,
> > connectionParameters={}):
> 
> We should probably add an options/flags parameter for extension of
> all new
> APIs.
Usually I agree but connectionParameters is already generic enough :)
> 
> > repoId - is a transient name that will be used to refer to the
> > connected domain, it is not persisted and doesn't have to be the
> > same across the cluster.
> > repoFormat - Similar to what used to be type (eg. localfs-1.0,
> > nfs-3.4, clvm-1.2).
> > connectionParameters - This is format specific and will used to
> > tell VDSM how to connect to the repo.
> > 
> > disconnectStorageRepository(self, repoId):
> 
> I assume 'self' is a mistake here.  Just want to clarify given all of
> the recent
> talk about instances vs. namespaces.
Yea, it's just pasted from my code
> 
> > In the new API there are only images, some images are mutable and
> > some are not.
> > mutable images are also called VirtualDisks
> > immutable images are also called Snapshots
> 
> By mutable you mean writable right?  Or does the word mutable imply
> more than
> that?
It's a semantic distinction due to implementation details, in general terms, yes.
> 
> > There are no explicit templates, you can create as many images as
> > you want from any snapshot.
> > 
> > There are 4 major image operations:
> > 
> > 
> > createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> >                   userData={}, options={}):
> 
> Is userdata a 'StringMap'?
currently it's a json object. We could limit it to a string map and trust the client to parse types.
We can have it be a string\blob and trust the user to serialize the data.
It's pass-through object either way.
> 
> I will reopen the argument about an options dict vs a flags
> parameter.  I oppose
> the dict for expansion because I think it causes APIs to devolve into
> a mess
> where lots of arbitrary and not well thought out overrides are packed
> into the
> dict over time.  A flags argument (in json and python it can be an
> enum array)
> limits us to really switching flags on and off instead of passing
> arbitrary
> data.
We already have strategy that we know we want to have several options.
Other stuff that have been suggested is to be able to override the img format (qcow2\qed)

The way I envision it is having an class
opts = CommandOptions()
that you add
opts.addStringOption("key", "value")
opts.addIntOption("key", 3)
opt.addBoolOption("key", True)

I know you could just as well have
strategy_space_flag and strategy_performance_flag
and fail the operation if they both exist.
Since it is a matter of personal taste I think it should be decided by a vote.
> 
> > targetRepoId - ID of a connected repo where the disk will be
> > created
> > size - The size of the image you wish to create
> > baseSnapshotId - the ID of the snapshot you want the base the new
> > virtual disk on
> > userData - optional data that will be attached to the new VD, could
> > be anything that the user desires.
> > options - options to modify VDSMs default behavior
> > 
> > returns the id of the new VD
> > 
> > createSnapshot(targetRepoId, baseVirtualDiskId,
> >                userData={}, options={}):
> > targetRepoId - The ID of a connected repo where the new sanpshot
> > will be created and the original image exists as well.
> > size - The size of the image you wish to create
> 
> Why is this needed?  Doesn't the size of a snapshot have to be equal
> to its
> base image?
oops, another copy\paste error, you can see this arg doesn't exist in the method signature.
My proof reading do need more work.
> 
> > baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want
> > to snapshot
> 
> Can you snapshot a snapshot?  In that case, this parameter should be
> called
> baseImage.
You can't snapshot a sanpshot, it makes no sense as it can't change and you will get the same object.
> 
> > userData - optional data that will be attached to the new Snapshot,
> > could be anything that the user desires.
> > options - options to modify VDSMs default behavior
> > 
> > returns the id of the new Snapshot
> > 
> > copyImage(targetRepoId, imageId, baseImageId=None, userData={},
> > options={})
> > targetRepoId - The ID of a connected repo where the new image will
> > be created
> > imageId - The image you wish to copy
> 
> Do we locate the sourceRepoId automatically based on the imageId?
I have written about that below when I talk about common options
> 
> > baseImageId - if specified, the new image will contain only the
> > diff between image and Id.
> >               If None the new image will contain all the bits of
> >               image Id. This can be used to copy partial parts of
> >               images for export.
> > userData - optional data that will be attached to the new image,
> > could be anything that the user desires.
> > options - options to modify VDSMs default behavior
> > 
> > return the Id of the new image. In case of copying an immutable
> > image the ID will be identical to the original image as they
> > contain the same data. However the user should not assume that and
> > always use the value returned from the method.
> > 
> > removeImage(repositoryId, imageId, options={}):
> > repositoryId - The ID of a connected repo where the image to delete
> > resides
> > imageId - The id of the image you wish to delete.
> > 
> > ----
> > getImageStatus(repositoryId, imageId)
> > repositoryId - The ID of a connected repo where the image to check
> > resides
> > imageId - The id of the image you wish to check.
> 
> What is in this return value?  Is it a single enum indicating whether
> the image
> is locked (being copied, etc.) or a list of detailed information
> (like
> Volume.getInfo)?  (I see some more info below...)
The description below explains everything.
> 
> > All operations return once the operations has been committed to
> > disk NOT when the operation actually completes.
> > This is done so that:
> > - operation come to a stable state as quickly as possible.
> > - In case where there is an SDM, only small portion of the
> > operation actually needs to be performed on the SDM host.
> > - No matter how many times the operation fails and on how many
> > hosts, you can always resume the operation and choose when to do
> > it.
> > - You can stop an operation at any time and remove the resulting
> > object making a distinction between "stop because the host is
> > overloaded" to "I don't want that image"
> > 
> > This means that after calling any operation that creates a new
> > image the user must then call getImageStatus() to check what is
> > the status of the image.
> > The status of the image can be either optimized, degraded, or
> > broken.
> > "Optimized" means that the image is available and you can run VMs
> > of it.
> > "Degraded" means that the image is available and will run VMs but
> > it might be a better way VDSM can represent the underlying data.
> > "Broken" means that the image can't be used at the moment, probably
> > because not all the data has been set up on the volume.
> > 
> > Apart from that VDSM will also return the last persisted status
> > information which will conatin
> > hostID - the last host to try and optimize of fix the image
> > stage - X/Y (eg. 1/10) the last persisted stage of the fix.
> 
> Do you have some examples of what the stages would be?  I think these
> should be
> defined in enums so that the user can check on what the individual
> stages mean.
> What happens when the low level implementation of an operation
> changes?  The
> meaning of the stages will change completely.
No stages can change and are implementation specific, they will be used in the UI to show progress.
I added stages because sometimes it's impossible to give percentage for multiple operations.
(If you need to do 2 copy operations is each of them 50%?)
I agree that we need some way of transferring text. Maybe having another field called current operations.
It will be something from a preset enum list:
"Copying Image" 1 
"Creating Volume" 2
"Changing Metadata" 3
That way we don't commit on number of stages or what they contain but still expose what it is we are doing to display in the UI
> 
> > percent_complete - -1 or 0-100, the last persisted completion
> > percentage of the aforementioned stage. -1 means that no progress
> > is available for that operation.
> 
> > last_error - This will only be filled if the operation failed
> > because of something other then IO or a VDSM crash for obvious
> > reasons.
> >              It will usually be set if the task was manually
> >              stopped
> > 
> > The user can either be satisfied with that information or as the
> > host specified in host ID if it is still working on that image by
> > checking it's running tasks.
> > 
> > checkStorageRepository(self, repositoryId, options={}):
> > A method to go over a storage repository and scan for any existing
> > problems. This includes degraded\broken images and deleted images
> > that have no yet been physically deleted\merged.
> > It returns a list of Fix objects.
> > Fix objects come in 4 types:
> > clean - cleans data, run them to get more space.
> > optimize - run them to optimize a degraded image
> 
> What is an example of a degraded image?
qcow snapshot when the user asked for performance driven image.
We create the QCow snapshot first because it will make the VM usable without doing a lot of IO.
Optimizing it will be to actually flatten the data and reach the desired state.
> 
> > merge - Merges two images together. Doing this sometimes
> >         makes more images ready optimizing or cleaning.
> >         The reason it is different from optimize is that
> >         unmerged images are considered optimized.
> > mend - mends a broken image
> 
> What does this mean?
The image is half copied. Snapshot that we know got corrupted and we have backups of. Whatever operation that moves an image status from broken->(optimized\degraded)
> 
> > The user can read these types and prioritize fixes. Fixes also
> > contain opaque FIX data and they should be sent as received to
> > fixStorageRepository(self, repositoryId, fix, options={}):
> > 
> > That will start a fix operation.
> 
> Could we have an automatic fix mode where vdsm just does the right
> thing (for
> most things)?
I wanted to have that, but deciding what VDSM should do automatically and when is really difficult and policy driven.
The best way to go IMHO is to have policy aware daemons monitor VDSM on the same host and auto-fix according to policy.
This shouldn't be part of VDSM as policy changes from manager to manager.
> 
> > All major operations automatically start the appropriate "Fix" to
> > bring the created object to an optimize\degraded state (the one
> > that is quicker) unless one of the options is
> > AutoFix=False. This is only useful for repos that might not be able
> > to create volumes on all hosts (SDM) but would like to have the
> > actual IO distributed in the cluster.
> > 
> > Other common options is the strategy option:
> > It has currently 2 possible values
> > space and performance - In case VDSM has 2 ways of completing the
> > same operation it will tell it to value one over the other. For
> > example, whether to copy all the data or just create a qcow based
> > of a snapshot.
> > The default is space.
> 
> I like this a lot.
:)
> 
> > You might have also noticed that it is never explicitly specified
> > where to look for existing images. This is done purposefully, VDSM
> > will always look in all connected repositories for existing
> > objects.
> > For very large setups this might be problematic. To mitigate the
> > problem you have these options:
> > participatingRepositories=[repoId, ...] which tell VDSM to narrow
> > the search to just these repositories
> > and
> > imageHints={imgId: repoId} which will force VDSM to look for those
> > image ID just in those repositories and fail if it doesn't find
> > them there.
> 
> I would like to have a better way of specifying these optional
> parameters
> without burying them in an options structure.  I will think a little
> more about
> this.  Strategy can just be a two optional flags in a 'flags'
> argument.  For the
> participatingRepositories and imageHints options, I think we need to
> use real
> parameters.
That was what I did in my original design, but then I chose to just move everything to options.
The reasoning is that without any options VDSM will try and do what it think is best for the user.
Without any options VDSM will use the combination that is most likely to work and in the shortest amount of resources (time\IO) with as little complications as possible.
This means space driven operations as they require less IO and space and "automatic source image discovery" as it has the best chance of finding the resources you need.
Whatever you put in the options essentially overrides the default behavior for cases where the user is willing to take a risk for some theoretical gain.
Using this distinction makes it clear what regular users should use and what expert users should use.
Further more, some options might be repository format specific (striping, pre-zeroing) making them part of the interface is problematic.

Also, the fact that we know about these options now doesn't mean they should get special treatment, they are still saying "override the default behavior".
> 
> --
> Adam Litke <agl at us.ibm.com>
> IBM Linux Technology Center
> 
> 


From amureini at redhat.com  Wed Dec  5 07:21:34 2012
From: amureini at redhat.com (Allon Mureinik)
Date: Wed, 5 Dec 2012 02:21:34 -0500 (EST)
Subject: [Engine-devel] [Users] RHEVM 3.1 DB Schema diagram
In-Reply-To: <485093285.18366122.1354641148339.JavaMail.root@redhat.com>
Message-ID: <624278119.10664647.1354692094895.JavaMail.root@redhat.com>

Nice!

How are the names of the blocks generated?
Some names strike me a bit there (e.g., images is in the storage_domains_static block).

----- Original Message -----
> From: "Eli Mesika" <emesika at redhat.com>
> To: "users" <users at ovirt.org>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Tuesday, December 4, 2012 7:12:28 PM
> Subject: [Users] RHEVM 3.1 DB Schema diagram
> 
> Hi
> 
> There were some requests on the users list to get our DB schema
> diagram.
> After looking a while for a tool for generating that , I have found
> the DBSchema tool.
> 
> Please find attached the RHEVM 3.1 DB Schema diagram.
> 
> I intend to get it into the oVirt site but currently it's too large
> and does not fit well.
> 
> Regards
> 
> Eli Mesika
> 
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
> 


From ofrenkel at redhat.com  Wed Dec  5 08:29:34 2012
From: ofrenkel at redhat.com (Omer Frenkel)
Date: Wed, 5 Dec 2012 03:29:34 -0500 (EST)
Subject: [Engine-devel] Proposal to add Shireesh Anjal as maintainer
	to	engine-gluster
In-Reply-To: <50B89873.8030404@redhat.com>
Message-ID: <4723502.41038682.1354696174837.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: engine-devel at ovirt.org
> Sent: Friday, November 30, 2012 1:28:51 PM
> Subject: [Engine-devel] Proposal to add Shireesh Anjal as maintainer to	engine-gluster
> 
> Shireesh has been working on the engine gluster support for the past
> year, covering patches and leading reviews of the various engine
> gluster
> patches.
> 
> I'd like to propose a new sub-component of the engine for the gluster
> module, adding shireesh as its maintainer.
> 

+1

> Thanks,
>     Itamar
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 



From yzaslavs at redhat.com  Wed Dec  5 08:34:01 2012
From: yzaslavs at redhat.com (Yair Zaslavsky)
Date: Wed, 5 Dec 2012 03:34:01 -0500 (EST)
Subject: [Engine-devel] Proposal to add Shireesh Anjal as
	maintainer	to	engine-gluster
In-Reply-To: <4723502.41038682.1354696174837.JavaMail.root@redhat.com>
Message-ID: <1984198008.43138201.1354696441836.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Omer Frenkel" <ofrenkel at redhat.com>
> To: "Itamar Heim" <iheim at redhat.com>
> Cc: engine-devel at ovirt.org
> Sent: Wednesday, December 5, 2012 10:29:34 AM
> Subject: Re: [Engine-devel] Proposal to add Shireesh Anjal as maintainer	to	engine-gluster
> 
> 
> 
> ----- Original Message -----
> > From: "Itamar Heim" <iheim at redhat.com>
> > To: engine-devel at ovirt.org
> > Sent: Friday, November 30, 2012 1:28:51 PM
> > Subject: [Engine-devel] Proposal to add Shireesh Anjal as
> > maintainer to	engine-gluster
> > 
> > Shireesh has been working on the engine gluster support for the
> > past
> > year, covering patches and leading reviews of the various engine
> > gluster
> > patches.
> > 
> > I'd like to propose a new sub-component of the engine for the
> > gluster
> > module, adding shireesh as its maintainer.
> > 
> 
> +1
+1
> 
> > Thanks,
> >     Itamar
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > 
> 
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From lhornyak at redhat.com  Wed Dec  5 10:32:51 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Wed, 5 Dec 2012 05:32:51 -0500 (EST)
Subject: [Engine-devel] host cpu feature
Message-ID: <600774010.5914603.1354703571901.JavaMail.root@redhat.com>

Hi,

CPU-Host support allows the virtual machines to see and utilize the host's CPU flags, this enables better performance in VM's, at the price of worse portablity.

http://www.ovirt.org/Features/Cpu-host_Support

Your feedback is welcome!

Thank you,
Laszlo


From emesika at redhat.com  Wed Dec  5 10:51:20 2012
From: emesika at redhat.com (Eli Mesika)
Date: Wed, 5 Dec 2012 05:51:20 -0500 (EST)
Subject: [Engine-devel] [Users] RHEVM 3.1 DB Schema diagram
In-Reply-To: <624278119.10664647.1354692094895.JavaMail.root@redhat.com>
Message-ID: <1679431457.18711815.1354704680489.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Allon Mureinik" <amureini at redhat.com>
> To: "Eli Mesika" <emesika at redhat.com>
> Cc: "users" <users at ovirt.org>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 9:21:34 AM
> Subject: Re: [Users] RHEVM 3.1 DB Schema diagram
> 
> Nice!
> 
> How are the names of the blocks generated?
> Some names strike me a bit there (e.g., images is in the
> storage_domains_static block).

Auto-generated ... (nice work IMHO)

> 
> ----- Original Message -----
> > From: "Eli Mesika" <emesika at redhat.com>
> > To: "users" <users at ovirt.org>, "engine-devel"
> > <engine-devel at ovirt.org>
> > Sent: Tuesday, December 4, 2012 7:12:28 PM
> > Subject: [Users] RHEVM 3.1 DB Schema diagram
> > 
> > Hi
> > 
> > There were some requests on the users list to get our DB schema
> > diagram.
> > After looking a while for a tool for generating that , I have found
> > the DBSchema tool.
> > 
> > Please find attached the RHEVM 3.1 DB Schema diagram.
> > 
> > I intend to get it into the oVirt site but currently it's too large
> > and does not fit well.
> > 
> > Regards
> > 
> > Eli Mesika
> > 
> > _______________________________________________
> > Users mailing list
> > Users at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> > 
> 


From ykaul at redhat.com  Wed Dec  5 11:23:47 2012
From: ykaul at redhat.com (Yaniv Kaul)
Date: Wed, 05 Dec 2012 13:23:47 +0200
Subject: [Engine-devel] host cpu feature
In-Reply-To: <600774010.5914603.1354703571901.JavaMail.root@redhat.com>
References: <600774010.5914603.1354703571901.JavaMail.root@redhat.com>
Message-ID: <50BF2EC3.6080900@redhat.com>

On 12/05/2012 12:32 PM, Laszlo Hornyak wrote:
> Hi,
>
> CPU-Host support allows the virtual machines to see and utilize the host's CPU flags, this enables better performance in VM's, at the price of worse portablity.
>
> http://www.ovirt.org/Features/Cpu-host_Support
>
> Your feedback is welcome!
>
> Thank you,
> Laszlo
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel

- I assume that when you allow migration, you'd use host-model? This is 
not clear from the design. It seems like we VDSM developers can choose 
to use either this or passthrough, while in practice we should support both.

- I'm still convinced and commented on both relevat oVirt and libvirt 
BZs that we need to add x2apic support to the CPU, regardless of what 
the host CPU exposes.
AFAIK, the KVM developers agree with me.
Y.



From lhornyak at redhat.com  Wed Dec  5 11:46:09 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Wed, 5 Dec 2012 06:46:09 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <50BF2EC3.6080900@redhat.com>
Message-ID: <205435578.5927365.1354707969229.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Yaniv Kaul" <ykaul at redhat.com>
> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 12:23:47 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> On 12/05/2012 12:32 PM, Laszlo Hornyak wrote:
> > Hi,
> >
> > CPU-Host support allows the virtual machines to see and utilize the
> > host's CPU flags, this enables better performance in VM's, at the
> > price of worse portablity.
> >
> > http://www.ovirt.org/Features/Cpu-host_Support
> >
> > Your feedback is welcome!
> >
> > Thank you,
> > Laszlo
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> 
> - I assume that when you allow migration, you'd use host-model? This
> is
> not clear from the design. It seems like we VDSM developers can
> choose
> to use either this or passthrough, while in practice we should
> support both.

If AllowMigrateCPUHost is set to true (in case you have the same cpu model everywhere in your DC) migration of such hsots will be enabled. Otherwise it will not be enabled.

> 
> - I'm still convinced and commented on both relevat oVirt and libvirt
> BZs that we need to add x2apic support to the CPU, regardless of what
> the host CPU exposes.
> AFAIK, the KVM developers agree with me.

Not quite sure how is this related... could you send some URL's for the bugreports?

> Y.
> 
> 


From ykaul at redhat.com  Wed Dec  5 12:31:51 2012
From: ykaul at redhat.com (Yaniv Kaul)
Date: Wed, 05 Dec 2012 14:31:51 +0200
Subject: [Engine-devel] host cpu feature
In-Reply-To: <205435578.5927365.1354707969229.JavaMail.root@redhat.com>
References: <205435578.5927365.1354707969229.JavaMail.root@redhat.com>
Message-ID: <50BF3EB7.7000804@redhat.com>

On 12/05/2012 01:46 PM, Laszlo Hornyak wrote:
>
> ----- Original Message -----
>> From: "Yaniv Kaul" <ykaul at redhat.com>
>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>> Cc: "engine-devel" <engine-devel at ovirt.org>
>> Sent: Wednesday, December 5, 2012 12:23:47 PM
>> Subject: Re: [Engine-devel] host cpu feature
>>
>> On 12/05/2012 12:32 PM, Laszlo Hornyak wrote:
>>> Hi,
>>>
>>> CPU-Host support allows the virtual machines to see and utilize the
>>> host's CPU flags, this enables better performance in VM's, at the
>>> price of worse portablity.
>>>
>>> http://www.ovirt.org/Features/Cpu-host_Support
>>>
>>> Your feedback is welcome!
>>>
>>> Thank you,
>>> Laszlo
>>> _______________________________________________
>>> Engine-devel mailing list
>>> Engine-devel at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>> - I assume that when you allow migration, you'd use host-model? This
>> is
>> not clear from the design. It seems like we VDSM developers can
>> choose
>> to use either this or passthrough, while in practice we should
>> support both.
> If AllowMigrateCPUHost is set to true (in case you have the same cpu model everywhere in your DC) migration of such hsots will be enabled. Otherwise it will not be enabled.

It's not going to help just enabling it - you need to use 'host-model' 
and not 'host-passthrough', so there'll be a reasonable chance to 
succeed the migration, AFAIK.

>
>> - I'm still convinced and commented on both relevat oVirt and libvirt
>> BZs that we need to add x2apic support to the CPU, regardless of what
>> the host CPU exposes.
>> AFAIK, the KVM developers agree with me.
> Not quite sure how is this related... could you send some URL's for the bugreports?

It's related because x2apic improves performance, and so does '-cpu 
host', but that doesn't enable x2apic implicitly.
https://bugzilla.redhat.com/show_bug.cgi?id=838469#c7
https://bugzilla.redhat.com/show_bug.cgi?id=700272#c28

Y.

>
>> Y.
>>
>>



From danken at redhat.com  Wed Dec  5 12:55:19 2012
From: danken at redhat.com (Dan Kenigsberg)
Date: Wed, 5 Dec 2012 14:55:19 +0200
Subject: [Engine-devel] host cpu feature
In-Reply-To: <205435578.5927365.1354707969229.JavaMail.root@redhat.com>
References: <50BF2EC3.6080900@redhat.com>
	<205435578.5927365.1354707969229.JavaMail.root@redhat.com>
Message-ID: <20121205125519.GD21647@redhat.com>

On Wed, Dec 05, 2012 at 06:46:09AM -0500, Laszlo Hornyak wrote:
> 
> 
> ----- Original Message -----
> > From: "Yaniv Kaul" <ykaul at redhat.com>
> > To: "Laszlo Hornyak" <lhornyak at redhat.com>
> > Cc: "engine-devel" <engine-devel at ovirt.org>
> > Sent: Wednesday, December 5, 2012 12:23:47 PM
> > Subject: Re: [Engine-devel] host cpu feature
> > 
> > On 12/05/2012 12:32 PM, Laszlo Hornyak wrote:
> > > Hi,
> > >
> > > CPU-Host support allows the virtual machines to see and utilize the
> > > host's CPU flags, this enables better performance in VM's, at the
> > > price of worse portablity.
> > >
> > > http://www.ovirt.org/Features/Cpu-host_Support
> > >
> > > Your feedback is welcome!
> > >
> > > Thank you,
> > > Laszlo
> > > _______________________________________________
> > > Engine-devel mailing list
> > > Engine-devel at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > 
> > - I assume that when you allow migration, you'd use host-model? This
> > is
> > not clear from the design. It seems like we VDSM developers can
> > choose
> > to use either this or passthrough, while in practice we should
> > support both.

I join Kaul's question: it is an ovirt-level question whether
hostPassthrough or hostModel or both should be supported. It should not
be a unilateral vdsm decision.

> 
> If AllowMigrateCPUHost is set to true (in case you have the same cpu model everywhere in your DC) migration of such hsots will be enabled. Otherwise it will not be enabled.

What is the breadth of AllowMigrateCPUHost? Engine wide? Per DC? Per cluster?
I favor the latter; a user may have a cluster of exact-same hosts, where
hostcpu migration is allowed, and other cluster where it is forbiden.

The nice thing about hostModel (unlike hostPassthrough) is that once we
created the VM we can migrate it to stronger hosts, and back to the
original host. I suppose that it complicates the scheduler.

> 
> > 
> > - I'm still convinced and commented on both relevat oVirt and libvirt
> > BZs that we need to add x2apic support to the CPU, regardless of what
> > the host CPU exposes.
> > AFAIK, the KVM developers agree with me.
> 
> Not quite sure how is this related... could you send some URL's for the bugreports?


From lhornyak at redhat.com  Wed Dec  5 13:39:52 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Wed, 5 Dec 2012 08:39:52 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <20121205125519.GD21647@redhat.com>
Message-ID: <612723448.5982144.1354714792522.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Dan Kenigsberg" <danken at redhat.com>
> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> Cc: "Yaniv Kaul" <ykaul at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 1:55:19 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> On Wed, Dec 05, 2012 at 06:46:09AM -0500, Laszlo Hornyak wrote:
> > 
> > 
> > ----- Original Message -----
> > > From: "Yaniv Kaul" <ykaul at redhat.com>
> > > To: "Laszlo Hornyak" <lhornyak at redhat.com>
> > > Cc: "engine-devel" <engine-devel at ovirt.org>
> > > Sent: Wednesday, December 5, 2012 12:23:47 PM
> > > Subject: Re: [Engine-devel] host cpu feature
> > > 
> > > On 12/05/2012 12:32 PM, Laszlo Hornyak wrote:
> > > > Hi,
> > > >
> > > > CPU-Host support allows the virtual machines to see and utilize
> > > > the
> > > > host's CPU flags, this enables better performance in VM's, at
> > > > the
> > > > price of worse portablity.
> > > >
> > > > http://www.ovirt.org/Features/Cpu-host_Support
> > > >
> > > > Your feedback is welcome!
> > > >
> > > > Thank you,
> > > > Laszlo
> > > > _______________________________________________
> > > > Engine-devel mailing list
> > > > Engine-devel at ovirt.org
> > > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > 
> > > - I assume that when you allow migration, you'd use host-model?
> > > This
> > > is
> > > not clear from the design. It seems like we VDSM developers can
> > > choose
> > > to use either this or passthrough, while in practice we should
> > > support both.
> 
> I join Kaul's question: it is an ovirt-level question whether
> hostPassthrough or hostModel or both should be supported. It should
> not
> be a unilateral vdsm decision.

Ah, possibly misunderstanding, I did not mean that VDSM should decide whether to use host-passthrough or host-model. The engine should decide.
I meant _you_ should decide which version of vdsm api modification do you want :)

> 
> > 
> > If AllowMigrateCPUHost is set to true (in case you have the same
> > cpu model everywhere in your DC) migration of such hsots will be
> > enabled. Otherwise it will not be enabled.
> 
> What is the breadth of AllowMigrateCPUHost? Engine wide? Per DC? Per
> cluster?

I thought of eninge-wide. The of course you can have different models in two different DC, but they should be unique in one.
We can add this to DC or cluster level, imho it would be just another checkbox on the UI that most users would not use.

> I favor the latter; a user may have a cluster of exact-same hosts,
> where
> hostcpu migration is allowed, and other cluster where it is forbiden.
> 
> The nice thing about hostModel (unlike hostPassthrough) is that once
> we
> created the VM we can migrate it to stronger hosts, and back to the
> original host. I suppose that it complicates the scheduler.

Yes with host-model you get the features that libvirt handles. In such cases the engine could decide, if you want this functionality. Well the scheduler architecture is just being reinvented.

For the host-passthrough, I think the AllowMigrateCPUHost configuration option would be a simple decision for the administrator: set it to true if all hosts are uniform. If it is not set to true, then we will not allow migration of such VMs.

> 
> > 
> > > 
> > > - I'm still convinced and commented on both relevat oVirt and
> > > libvirt
> > > BZs that we need to add x2apic support to the CPU, regardless of
> > > what
> > > the host CPU exposes.
> > > AFAIK, the KVM developers agree with me.
> > 
> > Not quite sure how is this related... could you send some URL's for
> > the bugreports?
> 


From ykaul at redhat.com  Wed Dec  5 13:45:46 2012
From: ykaul at redhat.com (Yaniv Kaul)
Date: Wed, 05 Dec 2012 15:45:46 +0200
Subject: [Engine-devel] host cpu feature
In-Reply-To: <612723448.5982144.1354714792522.JavaMail.root@redhat.com>
References: <612723448.5982144.1354714792522.JavaMail.root@redhat.com>
Message-ID: <50BF500A.2060406@redhat.com>

On 12/05/2012 03:39 PM, Laszlo Hornyak wrote:
>
> ----- Original Message -----
>> From: "Dan Kenigsberg" <danken at redhat.com>
>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>> Cc: "Yaniv Kaul" <ykaul at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
>> Sent: Wednesday, December 5, 2012 1:55:19 PM
>> Subject: Re: [Engine-devel] host cpu feature
>>
>> On Wed, Dec 05, 2012 at 06:46:09AM -0500, Laszlo Hornyak wrote:
>>>
>>> ----- Original Message -----
>>>> From: "Yaniv Kaul" <ykaul at redhat.com>
>>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>> Sent: Wednesday, December 5, 2012 12:23:47 PM
>>>> Subject: Re: [Engine-devel] host cpu feature
>>>>
>>>> On 12/05/2012 12:32 PM, Laszlo Hornyak wrote:
>>>>> Hi,
>>>>>
>>>>> CPU-Host support allows the virtual machines to see and utilize
>>>>> the
>>>>> host's CPU flags, this enables better performance in VM's, at
>>>>> the
>>>>> price of worse portablity.
>>>>>
>>>>> http://www.ovirt.org/Features/Cpu-host_Support
>>>>>
>>>>> Your feedback is welcome!
>>>>>
>>>>> Thank you,
>>>>> Laszlo
>>>>> _______________________________________________
>>>>> Engine-devel mailing list
>>>>> Engine-devel at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>>>> - I assume that when you allow migration, you'd use host-model?
>>>> This
>>>> is
>>>> not clear from the design. It seems like we VDSM developers can
>>>> choose
>>>> to use either this or passthrough, while in practice we should
>>>> support both.
>> I join Kaul's question: it is an ovirt-level question whether
>> hostPassthrough or hostModel or both should be supported. It should
>> not
>> be a unilateral vdsm decision.
> Ah, possibly misunderstanding, I did not mean that VDSM should decide whether to use host-passthrough or host-model. The engine should decide.
> I meant _you_ should decide which version of vdsm api modification do you want :)
>
>>> If AllowMigrateCPUHost is set to true (in case you have the same
>>> cpu model everywhere in your DC) migration of such hsots will be
>>> enabled. Otherwise it will not be enabled.
>> What is the breadth of AllowMigrateCPUHost? Engine wide? Per DC? Per
>> cluster?
> I thought of eninge-wide. The of course you can have different models in two different DC, but they should be unique in one.
> We can add this to DC or cluster level, imho it would be just another checkbox on the UI that most users would not use.
>
>> I favor the latter; a user may have a cluster of exact-same hosts,
>> where
>> hostcpu migration is allowed, and other cluster where it is forbiden.
>>
>> The nice thing about hostModel (unlike hostPassthrough) is that once
>> we
>> created the VM we can migrate it to stronger hosts, and back to the
>> original host. I suppose that it complicates the scheduler.
> Yes with host-model you get the features that libvirt handles. In such cases the engine could decide, if you want this functionality. Well the scheduler architecture is just being reinvented.
>
> For the host-passthrough, I think the AllowMigrateCPUHost configuration option would be a simple decision for the administrator: set it to true if all hosts are uniform. If it is not set to true, then we will not allow migration of such VMs.

That's not what I understood from libvirt's documentation. I understood 
that if you want host+migration, you need to use host-model. Otherwise - 
host-passthrough.
Y.

>
>>>> - I'm still convinced and commented on both relevat oVirt and
>>>> libvirt
>>>> BZs that we need to add x2apic support to the CPU, regardless of
>>>> what
>>>> the host CPU exposes.
>>>> AFAIK, the KVM developers agree with me.
>>> Not quite sure how is this related... could you send some URL's for
>>> the bugreports?



From djasa at redhat.com  Wed Dec  5 13:46:05 2012
From: djasa at redhat.com (David =?UTF-8?Q?Ja=C5=A1a?=)
Date: Wed, 05 Dec 2012 14:46:05 +0100
Subject: [Engine-devel] Sophie's choice: corrupt user data (on USB disk) or
 cancel migration?
Message-ID: <1354715165.3329.152.camel@dhcp-29-7.brq.redhat.com>

Hi,

I've encountered an interesting problem: I did a live migration through
libvirt (virsh migrate --live <domain> <dst_libvirt_uri>
[<dst_qemu_uri>]) with usb transfer ongoing from qemu guest through
usbredir over spice to usb device plugged to the client. The spice
client failed to connect to the destination qemu/spice but the migration
was conducted anyway, so the data transfer was abruptly terminated.

Background
----------
When performing migration with client connected, two monitor commands
are issued to the qemu:
(qemu) client_migrate_info spice <client connection details>
main_channel_client_handle_migrate_connected: client <ID> connected: 1
seamless: 1
(qemu) migrate -d tcp:<dst_qemu_address>:<port>

The first command instructs spice client to reconnect to the new machine
and the reply says that if connection is successful and if it is
seamless (required for migration with USB redirection active).

Problem
-------
The behaviour should follow "least harm" principle:

1) In "severe" scenarios when the choice is between VM migration causing
potential data corruption on redirected USB and stopping of the VM,
current behavior is perfectly fine

2) In "regular" scenarios (automatic load balancing, manual migration
requests by administrator), refuse to migrate (unless some "I really
know what I'm doing" button is pressed) is better thing to do

Possible solutions?
-------------------
Just one possible way to fix this occured to me:
1. libvirt gets new switch to migrate command to force migration even
when client can not connect
2. if this switch is not set and client won't connect after
client_migrate_info qemu command is issued, cance migration
3. management application (ovirt-engine through vdsm) or administrator
sets the switch to force migration in cases like 1) above

I'll file the necessary bugs but it should be IMO first agreed what the
correct approach is in such situations.

David

-- 

David Ja?a, RHCE

SPICE QE based in Brno
GPG Key:     22C33E24 
Fingerprint: 513A 060B D1B4 2A72 7F0D 0278 B125 CD00 22C3 3E24





From lhornyak at redhat.com  Wed Dec  5 13:55:53 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Wed, 5 Dec 2012 08:55:53 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <50BF500A.2060406@redhat.com>
Message-ID: <310221607.5986866.1354715753473.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Yaniv Kaul" <ykaul at redhat.com>
> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> Cc: "Dan Kenigsberg" <danken at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 2:45:46 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> On 12/05/2012 03:39 PM, Laszlo Hornyak wrote:
> >
> > ----- Original Message -----
> >> From: "Dan Kenigsberg" <danken at redhat.com>
> >> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> >> Cc: "Yaniv Kaul" <ykaul at redhat.com>, "engine-devel"
> >> <engine-devel at ovirt.org>
> >> Sent: Wednesday, December 5, 2012 1:55:19 PM
> >> Subject: Re: [Engine-devel] host cpu feature
> >>
> >> On Wed, Dec 05, 2012 at 06:46:09AM -0500, Laszlo Hornyak wrote:
> >>>
> >>> ----- Original Message -----
> >>>> From: "Yaniv Kaul" <ykaul at redhat.com>
> >>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> >>>> Cc: "engine-devel" <engine-devel at ovirt.org>
> >>>> Sent: Wednesday, December 5, 2012 12:23:47 PM
> >>>> Subject: Re: [Engine-devel] host cpu feature
> >>>>
> >>>> On 12/05/2012 12:32 PM, Laszlo Hornyak wrote:
> >>>>> Hi,
> >>>>>
> >>>>> CPU-Host support allows the virtual machines to see and utilize
> >>>>> the
> >>>>> host's CPU flags, this enables better performance in VM's, at
> >>>>> the
> >>>>> price of worse portablity.
> >>>>>
> >>>>> http://www.ovirt.org/Features/Cpu-host_Support
> >>>>>
> >>>>> Your feedback is welcome!
> >>>>>
> >>>>> Thank you,
> >>>>> Laszlo
> >>>>> _______________________________________________
> >>>>> Engine-devel mailing list
> >>>>> Engine-devel at ovirt.org
> >>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> >>>> - I assume that when you allow migration, you'd use host-model?
> >>>> This
> >>>> is
> >>>> not clear from the design. It seems like we VDSM developers can
> >>>> choose
> >>>> to use either this or passthrough, while in practice we should
> >>>> support both.
> >> I join Kaul's question: it is an ovirt-level question whether
> >> hostPassthrough or hostModel or both should be supported. It
> >> should
> >> not
> >> be a unilateral vdsm decision.
> > Ah, possibly misunderstanding, I did not mean that VDSM should
> > decide whether to use host-passthrough or host-model. The engine
> > should decide.
> > I meant _you_ should decide which version of vdsm api modification
> > do you want :)
> >
> >>> If AllowMigrateCPUHost is set to true (in case you have the same
> >>> cpu model everywhere in your DC) migration of such hsots will be
> >>> enabled. Otherwise it will not be enabled.
> >> What is the breadth of AllowMigrateCPUHost? Engine wide? Per DC?
> >> Per
> >> cluster?
> > I thought of eninge-wide. The of course you can have different
> > models in two different DC, but they should be unique in one.
> > We can add this to DC or cluster level, imho it would be just
> > another checkbox on the UI that most users would not use.
> >
> >> I favor the latter; a user may have a cluster of exact-same hosts,
> >> where
> >> hostcpu migration is allowed, and other cluster where it is
> >> forbiden.
> >>
> >> The nice thing about hostModel (unlike hostPassthrough) is that
> >> once
> >> we
> >> created the VM we can migrate it to stronger hosts, and back to
> >> the
> >> original host. I suppose that it complicates the scheduler.
> > Yes with host-model you get the features that libvirt handles. In
> > such cases the engine could decide, if you want this
> > functionality. Well the scheduler architecture is just being
> > reinvented.
> >
> > For the host-passthrough, I think the AllowMigrateCPUHost
> > configuration option would be a simple decision for the
> > administrator: set it to true if all hosts are uniform. If it is
> > not set to true, then we will not allow migration of such VMs.
> 
> That's not what I understood from libvirt's documentation. I

You may be right, could you send an URL to that point of the documentation or copy-paste?

> understood
> that if you want host+migration, you need to use host-model.
> Otherwise -
> host-passthrough.
> Y.
> 
> >
> >>>> - I'm still convinced and commented on both relevat oVirt and
> >>>> libvirt
> >>>> BZs that we need to add x2apic support to the CPU, regardless of
> >>>> what
> >>>> the host CPU exposes.
> >>>> AFAIK, the KVM developers agree with me.
> >>> Not quite sure how is this related... could you send some URL's
> >>> for
> >>> the bugreports?
> 
> 


From danken at redhat.com  Wed Dec  5 14:16:05 2012
From: danken at redhat.com (Dan Kenigsberg)
Date: Wed, 5 Dec 2012 16:16:05 +0200
Subject: [Engine-devel] host cpu feature
In-Reply-To: <612723448.5982144.1354714792522.JavaMail.root@redhat.com>
References: <20121205125519.GD21647@redhat.com>
	<612723448.5982144.1354714792522.JavaMail.root@redhat.com>
Message-ID: <20121205141605.GH21647@redhat.com>

On Wed, Dec 05, 2012 at 08:39:52AM -0500, Laszlo Hornyak wrote:
> 
> 
> ----- Original Message -----
> > From: "Dan Kenigsberg" <danken at redhat.com>
> > To: "Laszlo Hornyak" <lhornyak at redhat.com>
> > Cc: "Yaniv Kaul" <ykaul at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> > Sent: Wednesday, December 5, 2012 1:55:19 PM
> > Subject: Re: [Engine-devel] host cpu feature
> > 
> > On Wed, Dec 05, 2012 at 06:46:09AM -0500, Laszlo Hornyak wrote:
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Yaniv Kaul" <ykaul at redhat.com>
> > > > To: "Laszlo Hornyak" <lhornyak at redhat.com>
> > > > Cc: "engine-devel" <engine-devel at ovirt.org>
> > > > Sent: Wednesday, December 5, 2012 12:23:47 PM
> > > > Subject: Re: [Engine-devel] host cpu feature
> > > > 
> > > > On 12/05/2012 12:32 PM, Laszlo Hornyak wrote:
> > > > > Hi,
> > > > >
> > > > > CPU-Host support allows the virtual machines to see and utilize
> > > > > the
> > > > > host's CPU flags, this enables better performance in VM's, at
> > > > > the
> > > > > price of worse portablity.
> > > > >
> > > > > http://www.ovirt.org/Features/Cpu-host_Support
> > > > >
> > > > > Your feedback is welcome!
> > > > >
> > > > > Thank you,
> > > > > Laszlo
> > > > > _______________________________________________
> > > > > Engine-devel mailing list
> > > > > Engine-devel at ovirt.org
> > > > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > 
> > > > - I assume that when you allow migration, you'd use host-model?
> > > > This
> > > > is
> > > > not clear from the design. It seems like we VDSM developers can
> > > > choose
> > > > to use either this or passthrough, while in practice we should
> > > > support both.
> > 
> > I join Kaul's question: it is an ovirt-level question whether
> > hostPassthrough or hostModel or both should be supported. It should
> > not
> > be a unilateral vdsm decision.
> 
> Ah, possibly misunderstanding, I did not mean that VDSM should decide whether to use host-passthrough or host-model. The engine should decide.
> I meant _you_ should decide which version of vdsm api modification do you want :)
> 
> > 
> > > 
> > > If AllowMigrateCPUHost is set to true (in case you have the same
> > > cpu model everywhere in your DC) migration of such hsots will be
> > > enabled. Otherwise it will not be enabled.
> > 
> > What is the breadth of AllowMigrateCPUHost? Engine wide? Per DC? Per
> > cluster?
> 
> I thought of eninge-wide. The of course you can have different models in two different DC, but they should be unique in one.
> We can add this to DC or cluster level, imho it would be just another checkbox on the UI that most users would not use.

Most users are not going to use hostcpu. This feature is intended to
people who have performance-critical VMs, and a set of hosts that can
run them. These very same people may have less critical desktops that
are to be allowed to run on any host.


From danken at redhat.com  Wed Dec  5 14:22:18 2012
From: danken at redhat.com (Dan Kenigsberg)
Date: Wed, 5 Dec 2012 16:22:18 +0200
Subject: [Engine-devel] host cpu feature
In-Reply-To: <50BF548C.5090909@redhat.com>
References: <310221607.5986866.1354715753473.JavaMail.root@redhat.com>
	<50BF548C.5090909@redhat.com>
Message-ID: <20121205142218.GI21647@redhat.com>

On Wed, Dec 05, 2012 at 04:05:00PM +0200, Yaniv Kaul wrote:
> On 12/05/2012 03:55 PM, Laszlo Hornyak wrote:
> >
> >>>>
> >>>>The nice thing about hostModel (unlike hostPassthrough) is that
> >>>>once
> >>>>we
> >>>>created the VM we can migrate it to stronger hosts, and back to
> >>>>the
> >>>>original host. I suppose that it complicates the scheduler.
> >>>Yes with host-model you get the features that libvirt handles. In
> >>>such cases the engine could decide, if you want this
> >>>functionality. Well the scheduler architecture is just being
> >>>reinvented.
> >>>
> >>>For the host-passthrough, I think the AllowMigrateCPUHost
> >>>configuration option would be a simple decision for the
> >>>administrator: set it to true if all hosts are uniform.

If it is THAT simple, Engine could take this decision without human
intervension.

> >>>If it is
> >>>not set to true, then we will not allow migration of such VMs.
> >>That's not what I understood from libvirt's documentation. I
> >You may be right, could you send an URL to that point of the documentation or copy-paste?
> 
> The link I followed from your feature page:
> http://libvirt.org/formatdomain.html#elementsCPU :
> 
> host-model
> The host-model mode is essentially a shortcut to copying host CPU
> definition from capabilities XML into domain XML. Since the CPU
> definition is copied just before starting a domain, exactly the same
> XML can be used on different hosts while still providing the best
> guest CPU each host supports. Neither match attribute nor any
> feature elements can be used in this mode. Specifying CPU model is
> not supported either, but model's fallback attribute may still be
> used. Libvirt does not model every aspect of each CPU so the guest
> CPU will not match the host CPU exactly. On the other hand, the ABI
> provided to the guest is reproducible. During migration, complete
> CPU model definition is transferred to the destination host so the
> migrated guest will see exactly the same CPU model even if the
> destination host contains more capable CPUs for the running instance
> of the guest; but shutting down and restarting the guest may present
> different hardware to the guest according to the capabilities of the
> new host.
> host-passthrough
> With this mode, the CPU visible to the guest should be exactly the
> same as the host CPU even in the aspects that libvirt does not
> understand. Though the downside of this mode is that the guest
> environment cannot be reproduced on different hardware. Thus, if you
> hit any bugs, you are on your own.

That's exactly where AllowMigrateCPUHost fits well: when a user ticks
this for a cluster he says "yeah, I like to be on my own."


From lhornyak at redhat.com  Wed Dec  5 14:22:24 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Wed, 5 Dec 2012 09:22:24 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <50BF548C.5090909@redhat.com>
Message-ID: <1937999159.6007673.1354717344906.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Yaniv Kaul" <ykaul at redhat.com>
> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> Cc: "Dan Kenigsberg" <danken at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 3:05:00 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> 
> On 12/05/2012 03:55 PM, Laszlo Hornyak wrote:
> 
> 
> ----- Original Message -----
> 
> From: "Yaniv Kaul" <ykaul at redhat.com> To: "Laszlo Hornyak"
> <lhornyak at redhat.com> Cc: "Dan Kenigsberg" <danken at redhat.com> ,
> "engine-devel" <engine-devel at ovirt.org> Sent: Wednesday, December 5,
> 2012 2:45:46 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> On 12/05/2012 03:39 PM, Laszlo Hornyak wrote:
> 
> ----- Original Message -----
> 
> From: "Dan Kenigsberg" <danken at redhat.com> To: "Laszlo Hornyak"
> <lhornyak at redhat.com> Cc: "Yaniv Kaul" <ykaul at redhat.com> ,
> "engine-devel" <engine-devel at ovirt.org> Sent: Wednesday, December 5,
> 2012 1:55:19 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> On Wed, Dec 05, 2012 at 06:46:09AM -0500, Laszlo Hornyak wrote:
> 
> ----- Original Message -----
> 
> From: "Yaniv Kaul" <ykaul at redhat.com> To: "Laszlo Hornyak"
> <lhornyak at redhat.com> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 12:23:47 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> On 12/05/2012 12:32 PM, Laszlo Hornyak wrote:
> 
> Hi,
> 
> CPU-Host support allows the virtual machines to see and utilize
> the
> host's CPU flags, this enables better performance in VM's, at
> the
> price of worse portablity.
> http://www.ovirt.org/Features/Cpu-host_Support Your feedback is
> welcome!
> 
> Thank you,
> Laszlo
> _______________________________________________
> Engine-devel mailing list Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel - I assume that
> when you allow migration, you'd use host-model?
> This
> is
> not clear from the design. It seems like we VDSM developers can
> choose
> to use either this or passthrough, while in practice we should
> support both. I join Kaul's question: it is an ovirt-level question
> whether
> hostPassthrough or hostModel or both should be supported. It
> should
> not
> be a unilateral vdsm decision. Ah, possibly misunderstanding, I did
> not mean that VDSM should
> decide whether to use host-passthrough or host-model. The engine
> should decide.
> I meant _you_ should decide which version of vdsm api modification
> do you want :)
> 
> 
> 
> If AllowMigrateCPUHost is set to true (in case you have the same
> cpu model everywhere in your DC) migration of such hsots will be
> enabled. Otherwise it will not be enabled. What is the breadth of
> AllowMigrateCPUHost? Engine wide? Per DC?
> Per
> cluster? I thought of eninge-wide. The of course you can have
> different
> models in two different DC, but they should be unique in one.
> We can add this to DC or cluster level, imho it would be just
> another checkbox on the UI that most users would not use.
> 
> I favor the latter; a user may have a cluster of exact-same hosts,
> where
> hostcpu migration is allowed, and other cluster where it is
> forbiden.
> 
> The nice thing about hostModel (unlike hostPassthrough) is that
> once
> we
> created the VM we can migrate it to stronger hosts, and back to
> the
> original host. I suppose that it complicates the scheduler. Yes with
> host-model you get the features that libvirt handles. In
> such cases the engine could decide, if you want this
> functionality. Well the scheduler architecture is just being
> reinvented.
> 
> For the host-passthrough, I think the AllowMigrateCPUHost
> configuration option would be a simple decision for the
> administrator: set it to true if all hosts are uniform. If it is
> not set to true, then we will not allow migration of such VMs. That's
> not what I understood from libvirt's documentation. I You may be
> right, could you send an URL to that point of the documentation or
> copy-paste?
> The link I followed from your feature page:
> http://libvirt.org/formatdomain.html#elementsCPU :
> 
> host-model
> The host-model mode is essentially a shortcut to copying host CPU
> definition from capabilities XML into domain XML. Since the CPU
> definition is copied just before starting a domain, exactly the same
> XML can be used on different hosts while still providing the best
> guest CPU each host supports. Neither match attribute nor any
> feature elements can be used in this mode. Specifying CPU model is
> not supported either, but model's fallback attribute may still be
> used. Libvirt does not model every aspect of each CPU so the guest
> CPU will not match the host CPU exactly. On the other hand, the ABI
> provided to the guest is reproducible. During migration, complete
> CPU model definition is transferred to the destination host so the
> migrated guest will see exactly the same CPU model even if the
> destination host contains more capable CPUs for the running instance
> of the guest; but shutting down and restarting the guest may present
> different hardware to the guest according to the capabilities of the
> new host.
> host-passthrough
> With this mode, the CPU visible to the guest should be exactly the
> same as the host CPU even in the aspects that libvirt does not
> understand. Though the downside of this mode is that the guest
> environment cannot be reproduced on different hardware. Thus, if you

"Though the downside of this mode is that the guest environment cannot be reproduced on different hardware"

Ok, what I understand from this is that if the cpu-model is the same on the other host, then it is OK, other CPU models will likely fail.
I need to talk to libvirt guys about this.

But anyway, I am OK with making host-passthrough VM's pinned to host. In that way I think it will be easier for the users as well.

> hit any bugs, you are on your own. Neither model nor feature
> elements are allowed in this mode.
> 
> 
> Y.
> 
> 
> 
> 
> 
> understood
> that if you want host+migration, you need to use host-model.
> Otherwise -
> host-passthrough.
> Y.
> 
> 
> 
> 
> 
> 
> 
> - I'm still convinced and commented on both relevat oVirt and
> libvirt
> BZs that we need to add x2apic support to the CPU, regardless of
> what
> the host CPU exposes.
> AFAIK, the KVM developers agree with me. Not quite sure how is this
> related... could you send some URL's
> for
> the bugreports?
> 


From smizrahi at redhat.com  Wed Dec  5 14:45:48 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Wed, 5 Dec 2012 09:45:48 -0500 (EST)
Subject: [Engine-devel] [vdsm] VDSM tasks, the future
In-Reply-To: <50BF4E13.7090700@redhat.com>
Message-ID: <1468813011.14266792.1354718748927.JavaMail.root@redhat.com>

I'm sorry but your email client messed up the formatting and I can't figure out what are you comments.
Could you please use text only emails.

----- Original Message -----
> From: "ybronhei" <ybronhei at redhat.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "Adam Litke" <agl at us.ibm.com>, "engine-devel" <engine-devel at ovirt.org>, "VDSM Project Development"
> <vdsm-devel at lists.fedorahosted.org>
> Sent: Wednesday, December 5, 2012 8:37:23 AM
> Subject: Re: [vdsm] VDSM tasks, the future
> 
> 
> On 12/05/2012 12:20 AM, Saggi Mizrahi wrote:
> 
> 
> As the only subsystem to use asynchronous tasks until now is the
> storage subsystem I suggest going over how
> I suggest tackling task creation, task stop, task remove and task
> recovery.
> Other subsystem can create similar mechanisms depending on their
> needs.
> 
> There is no way of avoiding it, different types of tasks need
> different ways of tracking\recovering from them.
> network should always auto-recover because it can't get a "please
> fix" command if the network is down.
> Storage on the other hand should never start operations on it's own
> because it might take up valuable resources from the host.
> Tasks that need to be tracked on a single host, 2 hosts, or the
> entire cluster need to have their own APIs.
> VM configuration never persist across reboots, networking sometimes
> persists and storage always persists.
> This means that recovery procedures (from the managers point of view)
> need to be vastly different.
> Add policy, resource allocation, and error flows you see that VDSM
> doesn't have nearly as much information to deal with the tasks.
> 
> ----- Original Message -----
> 
> From: "Adam Litke" <agl at us.ibm.com> To: "Saggi Mizrahi"
> <smizrahi at redhat.com> Cc: "VDSM Project Development"
> <vdsm-devel at lists.fedorahosted.org> , "engine-devel"
> <engine-devel at ovirt.org> , "Ayal
> Baron" <abaron at redhat.com> , "Barak Azulay" <bazulay at redhat.com> ,
> "Shireesh Anjal" <sanjal at redhat.com> Sent: Tuesday, December 4, 2012
> 3:50:28 PM
> Subject: Re: VDSM tasks, the future
> 
> On Tue, Dec 04, 2012 at 10:35:01AM -0500, Saggi Mizrahi wrote:
> 
> Because I started hinting about how VDSM tasks are going to look
> going forward
> I thought it's better I'll just write everything in an email so we
> can talk
> about it in context.  This is not set in stone and I'm still
> debating things
> myself but it's very close to being done. Don't debate them yourself,
> debate them here!  Even better, propose
> your idea in
> schema form to show how a command might work exactly. I don't like
> throwing ideas in the air It can be much easier to understand the
> flow of a task in vdsm and outside vdsm by a small schema, mainly
> for the each task's states.
> To define the flow of a task you can separate between type of tasks
> (network, storage, vms, or else), we should have task's states that
> clarify if the task can be recovered or not, can be canceled or not
> and inc..
> 
> Canceling\Aborting\Reverting states should be more clarified and not
> every state can lead to all types of states.
> I tries to figure how task flow works today in vdsm, and this is what
> I've got - http://wiki.ovirt.org/Vdsm_tasks
> 
> 
> 
> 
> 
> 
> - Everything is asynchronous.  The nature of message based
> communication is
> that you can't have synchronous operations.  This is not really
> debatable
> because it's just how TCP\AMQP\<messaging> works. Can you show how a
> traditionally synchronous command might work?
>  Let's take
> Host.getVmList as an example. The same as it works today, it's all a
> matter of how you wrap the transport layer.
> You will send a json-rpc request and wait for a response with the
> same id.
> 
> As for the bindings, there are a lot of way we can tackle that.
> Always wait for the response and simulate synchronous behavior.
> Make every method return an object to track the task.
> task = host.getVmList()
> if not task.wait(1):
>     task.cancel()
> else:
>     res = task.result() It looks like traditional timeout.. why not
>     to split blocking actions and non-blocking actions, non-blocking
>     action will supply callback function to return to if the task
>     fails or success. for example:
> 
> createAsyncTask(host.getVmList, params, timeout=30,
> callbackGetVmList)
> 
> Instead of using the dispatcher? Do you want to keep the dispatcher
> concept?
> 
> 
> 
> Have it both ways (it's auto generated anyway) and have
> list = host.getVmList()
> task = host.getVmList_async()
> 
> Have a high level and low level interfaces.
> host = host()
> host.connect("tcp://host:3233")
> req = host.sendRequest("123213", "getVmList", [])
> if not req.wait(1):
>    ....
> 
> shost = SynchHost(host)
> shost.getVmList() # Actually wraps a request object
> ahost = AsyncHost(host)
> task = getVmList() # Actually wraps a request object
> 
> 
> 
> - Task IDs will be decided by the caller.  This is how json-rpc
> works and also
> makes sense because no the engine can track the task without
> needing to have a
> stage where we give it the task ID back.  IDs are reusable as long
> as no one
> else is using them at the time so they can be used for
> synchronizing
> operations between clients (making sure a command is only executed
> once on a
> specific host without locking).
> 
> - Tasks are transient If VDSM restarts it forgets all the task
> information.
> There are 2 ways to have persistent tasks: 1. The task creates an
> object that
> you can continue work on in VDSM.  The new storage does that by the
> fact that
> copyImage() returns one the target volume has been created but
> before the data
> has been fully copied.  From that moment on the stat of the copy
> can be
> queried from any host using getImageStatus() and the specific copy
> operation
> can be queried with getTaskStatus() on the host performing it.
>  After VDSM
> crashes, depending on policy, either VDSM will create a new task to
> continue
> the copy or someone else will send a command to continue the
> operation and
> that will be a new task.  2. VDSM tasks just start other operations
> track-able
> not through the task interface. For example Gluster.
> gluster.startVolumeRebalance() will return once it has been
> registered with
> Gluster.  glster.getOperationStatuses() will return the state of
> the operation
> from any host.  Each call is a task in itself. I worry about this
> approach because every command has a different
> semantic for
> checking progress.  For migration, we have to check VM status on the
> src and
> dest hosts.  For image copy we need to use a special status call on
> the dest
> image.  It would be nice if there was a unified method for checking
> on an
> operation.  Maybe that can be completion events.
> 
> Client:               vdsm:
> -------               -----
> 
> Image.copy(...)  -->
>                  <--  Operation Started
> Wait for event   ...
>                  <--  Event: Operation <id> done <code>
> 
> For an early error:
> 
> Client:               vdsm:
> -------               -----
> 
> Image.copy(...)  -->
>                  <--  Error: <code> The thing is that a lot of things
>                  need a different way of tracking their progress.
> Storage have completely different semantics from network or VM
> operations. This is the reason why we can use the implementation of
> task as something generic for all processes that we have.
> of course things need different ways of tracking their progress...
> That's why we need to use task's states with meaning, and split
> storageTaskStates and networkTaskStates that inherit of TaskStates
> and add their parts as in the new bootstrap implementation.
> Also we can add hooks for each state as alonbl did in his otopi code
> (not sure if we need that)
> 
> Like for instance: general states can be - starting, started,
> finishing, finished, and each specific implementation adds middle
> states. like waitForResource, processing, recovering and inc..
> for each one you can add levels (pre state, post state) that can add
> more flexibility.
> 
> That way Task Object will be a general way to implement specific
> process, you will have a NetworkTask and StorageTask and the
> infrastructure will be the interface and implementation of the
> generic parts.
> 
> So here how vdsm can work that way:
> client: vsdm:
> -------- ---------
> image.copy() ---> copyImage::starting (same starting code - keeping
> the id, and move forward to next state)
> copyImage::started (waiting to recovery file that task is started)
> copyImage::part1 (whatever you want to do)
> copyImage::part2 (whatever you want to do)
> copyImage::part3 (whatever you want to do) -- for each process the
> programmers will add their states as they want in a sequence flow
> result <------ copyImage::finishing (send back to client a success
> and clean recovery file)
> copyImage::finished (sign task id as succeeded)
> 
> If somewhere in the middle an error occurred, it easier to start over
> and remember where we were.
> The problem with that is that we need to modify the current
> implementation for each process, and I'm not sure if we want to get
> there.. but if we do, it won't be so hard.
> We can split the logic of each process to define a logic of each
> state, and then arranging the states flow for each process and
> clarify what can be recovered or not, what signs corruption or
> errors, and how the returned result can point of the current process
> status (\state)
> 
> 
> 
> 
> 
> 
> 
> - No task tags.  They are silly and the caller can mangle whatever
> in the task
> ID if he really wants to tag tasks. Yes.  Agreed.
> 
> - No explicit recovery stage.  VDSM will be crash-only, there
> should be
> efforts to make everything crash-safe.  If that is problematic, in
> case of
> networking, VDSM will recover on start without having a task for
> it. How does this work in practice for something like creating a new
> image from a
> template?
> 
> - No clean Task: Tasks can be started by any number of hosts this
> means that
> there is no way to own all tasks.  There could be cases where VDSM
> starts
> tasks on it's own and thus they have no owner at all.  The caller
> needs to
> continually track the state of VDSM. We will have brodcasted events
> to
> mitigate polling. If a disconnected client might have missed a
> completion event, it
> will need to
> check state.  This means each async operation that changes state must
> document a
> proceedure for checking progress of a potentially ongoing operation.
>  For
> Image.copy, that process would be to lookup the new image and check
> its state.
> 
> - No revert Impossible to implement safely. How do the engine folks
> feel about this?  I am ok with it :) I don't care, unless they find
> a way to change they way logic works they can't have it.
> The whole concept of recovery (as it is defined now) doesn't work in
> an HA cluster.
> 
> 
> 
> - No SPM\HSM tasks SPM\SDM is no longer necessary for all domain
> types (only
> for type).  What used to be SPM tasks, or tasks that persist and
> can be
> restarted on other hosts is talked about in previous bullet points. A
> nice simplification.
> 
> 
> --
> Adam Litke <agl at us.ibm.com> IBM Linux Technology Center
> _______________________________________________
> vdsm-devel mailing list vdsm-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 
> --
> Yaniv Bronhaim.
> RedHat, Israel
> 09-7692289
> 054-7744187


From ybronhei at redhat.com  Wed Dec  5 13:37:23 2012
From: ybronhei at redhat.com (ybronhei)
Date: Wed, 05 Dec 2012 15:37:23 +0200
Subject: [Engine-devel] [vdsm] VDSM tasks, the future
In-Reply-To: <170195820.13932156.1354659610794.JavaMail.root@redhat.com>
References: <170195820.13932156.1354659610794.JavaMail.root@redhat.com>
Message-ID: <50BF4E13.7090700@redhat.com>

On 12/05/2012 12:20 AM, Saggi Mizrahi wrote:
> As the only subsystem to use asynchronous tasks until now is the storage subsystem I suggest going over how
> I suggest tackling task creation, task stop, task remove and task recovery.
> Other subsystem can create similar mechanisms depending on their needs.
>
> There is no way of avoiding it, different types of tasks need different ways of tracking\recovering from them.
> network should always auto-recover because it can't get a "please fix" command if the network is down.
> Storage on the other hand should never start operations on it's own because it might take up valuable resources from the host.
> Tasks that need to be tracked on a single host, 2 hosts, or the entire cluster need to have their own APIs.
> VM configuration never persist across reboots, networking sometimes persists and storage always persists.
> This means that recovery procedures (from the managers point of view) need to be vastly different.
> Add policy, resource allocation, and error flows you see that VDSM doesn't have nearly as much information to deal with the tasks.
>
> ----- Original Message -----
>> From: "Adam Litke" <agl at us.ibm.com>
>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
>> Cc: "VDSM Project Development" <vdsm-devel at lists.fedorahosted.org>, "engine-devel" <engine-devel at ovirt.org>, "Ayal
>> Baron" <abaron at redhat.com>, "Barak Azulay" <bazulay at redhat.com>, "Shireesh Anjal" <sanjal at redhat.com>
>> Sent: Tuesday, December 4, 2012 3:50:28 PM
>> Subject: Re: VDSM tasks, the future
>>
>> On Tue, Dec 04, 2012 at 10:35:01AM -0500, Saggi Mizrahi wrote:
>>> Because I started hinting about how VDSM tasks are going to look
>>> going forward
>>> I thought it's better I'll just write everything in an email so we
>>> can talk
>>> about it in context.  This is not set in stone and I'm still
>>> debating things
>>> myself but it's very close to being done.
>> Don't debate them yourself, debate them here!  Even better, propose
>> your idea in
>> schema form to show how a command might work exactly.
> I don't like throwing ideas in the air
It can be much easier to understand the flow of a task in vdsm and 
outside vdsm by a small schema, mainly for the each task's states.
To define the flow of a task you can separate between type of tasks 
(network, storage, vms, or else), we should have task's states that 
clarify if the task can be recovered or not, can be canceled or not and 
inc..

Canceling\Aborting\Reverting states should be more clarified and not 
every state can lead to all types of states.
I tries to figure how task flow works today in vdsm, and this is what 
I've got - http://wiki.ovirt.org/Vdsm_tasks
>>> - Everything is asynchronous.  The nature of message based
>>> communication is
>>> that you can't have synchronous operations.  This is not really
>>> debatable
>>> because it's just how TCP\AMQP\<messaging> works.
>> Can you show how a traditionally synchronous command might work?
>>   Let's take
>> Host.getVmList as an example.
> The same as it works today, it's all a matter of how you wrap the transport layer.
> You will send a json-rpc request and wait for a response with the same id.
>
> As for the bindings, there are a lot of way we can tackle that.
> Always wait for the response and simulate synchronous behavior.
> Make every method return an object to track the task.
> task = host.getVmList()
> if not task.wait(1):
>      task.cancel()
> else:
>      res = task.result()
It looks like traditional timeout.. why not to split blocking actions 
and non-blocking actions, non-blocking action will supply callback 
function to return to if the task fails or success. for example:

createAsyncTask(host.getVmList, params, timeout=30, callbackGetVmList)

Instead of using the dispatcher? Do you want to keep the dispatcher concept?

> Have it both ways (it's auto generated anyway) and have
> list = host.getVmList()
> task = host.getVmList_async()
>
> Have a high level and low level interfaces.
> host = host()
> host.connect("tcp://host:3233")
> req = host.sendRequest("123213", "getVmList", [])
> if not req.wait(1):
>     ....
>
> shost = SynchHost(host)
> shost.getVmList() # Actually wraps a request object
> ahost = AsyncHost(host)
> task = getVmList() # Actually wraps a request object
>>> - Task IDs will be decided by the caller.  This is how json-rpc
>>> works and also
>>> makes sense because no the engine can track the task without
>>> needing to have a
>>> stage where we give it the task ID back.  IDs are reusable as long
>>> as no one
>>> else is using them at the time so they can be used for
>>> synchronizing
>>> operations between clients (making sure a command is only executed
>>> once on a
>>> specific host without locking).
>>>
>>> - Tasks are transient If VDSM restarts it forgets all the task
>>> information.
>>> There are 2 ways to have persistent tasks: 1. The task creates an
>>> object that
>>> you can continue work on in VDSM.  The new storage does that by the
>>> fact that
>>> copyImage() returns one the target volume has been created but
>>> before the data
>>> has been fully copied.  From that moment on the stat of the copy
>>> can be
>>> queried from any host using getImageStatus() and the specific copy
>>> operation
>>> can be queried with getTaskStatus() on the host performing it.
>>>   After VDSM
>>> crashes, depending on policy, either VDSM will create a new task to
>>> continue
>>> the copy or someone else will send a command to continue the
>>> operation and
>>> that will be a new task.  2. VDSM tasks just start other operations
>>> track-able
>>> not through the task interface. For example Gluster.
>>> gluster.startVolumeRebalance() will return once it has been
>>> registered with
>>> Gluster.  glster.getOperationStatuses() will return the state of
>>> the operation
>>> from any host.  Each call is a task in itself.
>> I worry about this approach because every command has a different
>> semantic for
>> checking progress.  For migration, we have to check VM status on the
>> src and
>> dest hosts.  For image copy we need to use a special status call on
>> the dest
>> image.  It would be nice if there was a unified method for checking
>> on an
>> operation.  Maybe that can be completion events.
>>
>> Client:               vdsm:
>> -------               -----
>>
>> Image.copy(...)  -->
>>                   <--  Operation Started
>> Wait for event   ...
>>                   <--  Event: Operation <id> done <code>
>>
>> For an early error:
>>
>> Client:               vdsm:
>> -------               -----
>>
>> Image.copy(...)  -->
>>                   <--  Error: <code>
>>
>>
> The thing is that a lot of things need a different way of tracking their progress.
> Storage have completely different semantics from network or VM operations.
This is the reason why we can use the implementation of task as 
something generic for all processes that we have.
of course things need different ways of tracking their progress... 
That's why we need to use task's states with meaning, and split 
storageTaskStates and networkTaskStates that inherit of TaskStates and 
add their parts as in the new bootstrap implementation.
Also we can add hooks for each state as alonbl did in his otopi code 
(not sure if we need that)

Like for instance: general states can be - starting, started, finishing, 
finished, and each specific implementation adds middle states. like 
waitForResource, processing, recovering and inc..
for each one you can add levels (pre state, post state) that can add 
more flexibility.

That way Task Object will be a general way to implement specific 
process, you will have a NetworkTask and StorageTask and the 
infrastructure will be the interface and implementation of the generic 
parts.

So here how vdsm can work that way:
client:            vsdm:
--------            ---------
image.copy() ---> copyImage::starting (same starting code - keeping the 
id, and move forward to next state)
                                copyImage::started  (waiting to recovery 
file that task is started)
                                copyImage::part1 (whatever you want to do)
                                copyImage::part2 (whatever you want to do)
                                copyImage::part3 (whatever you want to 
do)  -- for each process the programmers will add their states as they 
want in a sequence flow
result     <------      copyImage::finishing (send back to client a 
success and clean recovery file)
                                copyImage::finished (sign task id as 
succeeded)

If somewhere in the middle an error occurred, it easier to start over 
and remember where we were.
The problem with that is that we need to modify the current 
implementation for each process, and I'm not sure if we want to get 
there.. but if we do, it won't be so hard.
We can split the logic of each process to define a logic of each state, 
and then arranging the states flow for each process and clarify what can 
be recovered or not, what signs corruption or errors, and how the 
returned result can point of the current process status (\state)

>>> - No task tags.  They are silly and the caller can mangle whatever
>>> in the task
>>> ID if he really wants to tag tasks.
>> Yes.  Agreed.
>>
>>> - No explicit recovery stage.  VDSM will be crash-only, there
>>> should be
>>> efforts to make everything crash-safe.  If that is problematic, in
>>> case of
>>> networking, VDSM will recover on start without having a task for
>>> it.
>> How does this work in practice for something like creating a new
>> image from a
>> template?
>>
>>> - No clean Task: Tasks can be started by any number of hosts this
>>> means that
>>> there is no way to own all tasks.  There could be cases where VDSM
>>> starts
>>> tasks on it's own and thus they have no owner at all.  The caller
>>> needs to
>>> continually track the state of VDSM. We will have brodcasted events
>>> to
>>> mitigate polling.
>> If a disconnected client might have missed a completion event, it
>> will need to
>> check state.  This means each async operation that changes state must
>> document a
>> proceedure for checking progress of a potentially ongoing operation.
>>   For
>> Image.copy, that process would be to lookup the new image and check
>> its state.
>>
>>> - No revert Impossible to implement safely.
>> How do the engine folks feel about this?  I am ok with it :)
> I don't care, unless they find a way to change they way logic works they can't have it.
> The whole concept of recovery (as it is defined now) doesn't work in an HA cluster.
>>> - No SPM\HSM tasks SPM\SDM is no longer necessary for all domain
>>> types (only
>>> for type).  What used to be SPM tasks, or tasks that persist and
>>> can be
>>> restarted on other hosts is talked about in previous bullet points.
>>>
>> A nice simplification.
>>
>>
>> --
>> Adam Litke <agl at us.ibm.com>
>> IBM Linux Technology Center
>>
>>
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


-- 
Yaniv Bronhaim.
RedHat, Israel
09-7692289
054-7744187

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121205/207b68c8/attachment.html>

From lhornyak at redhat.com  Wed Dec  5 14:59:28 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Wed, 5 Dec 2012 09:59:28 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <20121205142218.GI21647@redhat.com>
Message-ID: <1135933549.6037019.1354719568977.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Dan Kenigsberg" <danken at redhat.com>
> To: "Yaniv Kaul" <ykaul at redhat.com>
> Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 3:22:18 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> On Wed, Dec 05, 2012 at 04:05:00PM +0200, Yaniv Kaul wrote:
> > On 12/05/2012 03:55 PM, Laszlo Hornyak wrote:
> > >
> > >>>>
> > >>>>The nice thing about hostModel (unlike hostPassthrough) is that
> > >>>>once
> > >>>>we
> > >>>>created the VM we can migrate it to stronger hosts, and back to
> > >>>>the
> > >>>>original host. I suppose that it complicates the scheduler.
> > >>>Yes with host-model you get the features that libvirt handles.
> > >>>In
> > >>>such cases the engine could decide, if you want this
> > >>>functionality. Well the scheduler architecture is just being
> > >>>reinvented.
> > >>>
> > >>>For the host-passthrough, I think the AllowMigrateCPUHost
> > >>>configuration option would be a simple decision for the
> > >>>administrator: set it to true if all hosts are uniform.
> 
> If it is THAT simple, Engine could take this decision without human
> intervension.

Is there a way engine can figure out if the cpu-models in all the hosts are the same?
I mean even if some host flags are not handled by libvirt and therefore vdsm and engine...
So I would really need that permission from the user.

> 
> > >>>If it is
> > >>>not set to true, then we will not allow migration of such VMs.
> > >>That's not what I understood from libvirt's documentation. I
> > >You may be right, could you send an URL to that point of the
> > >documentation or copy-paste?
> > 
> > The link I followed from your feature page:
> > http://libvirt.org/formatdomain.html#elementsCPU :
> > 
> > host-model
> > The host-model mode is essentially a shortcut to copying host CPU
> > definition from capabilities XML into domain XML. Since the CPU
> > definition is copied just before starting a domain, exactly the
> > same
> > XML can be used on different hosts while still providing the best
> > guest CPU each host supports. Neither match attribute nor any
> > feature elements can be used in this mode. Specifying CPU model is
> > not supported either, but model's fallback attribute may still be
> > used. Libvirt does not model every aspect of each CPU so the guest
> > CPU will not match the host CPU exactly. On the other hand, the ABI
> > provided to the guest is reproducible. During migration, complete
> > CPU model definition is transferred to the destination host so the
> > migrated guest will see exactly the same CPU model even if the
> > destination host contains more capable CPUs for the running
> > instance
> > of the guest; but shutting down and restarting the guest may
> > present
> > different hardware to the guest according to the capabilities of
> > the
> > new host.
> > host-passthrough
> > With this mode, the CPU visible to the guest should be exactly the
> > same as the host CPU even in the aspects that libvirt does not
> > understand. Though the downside of this mode is that the guest
> > environment cannot be reproduced on different hardware. Thus, if
> > you
> > hit any bugs, you are on your own.
> 
> That's exactly where AllowMigrateCPUHost fits well: when a user ticks
> this for a cluster he says "yeah, I like to be on my own."
> 

cpu mode="host-passthrough" migration: I talked to the libvirt guys and they said it is OK if the hardware and the software are the same, and it will work, but they would not recommend.

So if they do not recommend it, I would drop this from the feature spec. Anyone against it?

Laszlo


From ykaul at redhat.com  Wed Dec  5 14:05:00 2012
From: ykaul at redhat.com (Yaniv Kaul)
Date: Wed, 05 Dec 2012 16:05:00 +0200
Subject: [Engine-devel] host cpu feature
In-Reply-To: <310221607.5986866.1354715753473.JavaMail.root@redhat.com>
References: <310221607.5986866.1354715753473.JavaMail.root@redhat.com>
Message-ID: <50BF548C.5090909@redhat.com>

On 12/05/2012 03:55 PM, Laszlo Hornyak wrote:
>
> ----- Original Message -----
>> From: "Yaniv Kaul" <ykaul at redhat.com>
>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>> Cc: "Dan Kenigsberg" <danken at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
>> Sent: Wednesday, December 5, 2012 2:45:46 PM
>> Subject: Re: [Engine-devel] host cpu feature
>>
>> On 12/05/2012 03:39 PM, Laszlo Hornyak wrote:
>>> ----- Original Message -----
>>>> From: "Dan Kenigsberg" <danken at redhat.com>
>>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>> Cc: "Yaniv Kaul" <ykaul at redhat.com>, "engine-devel"
>>>> <engine-devel at ovirt.org>
>>>> Sent: Wednesday, December 5, 2012 1:55:19 PM
>>>> Subject: Re: [Engine-devel] host cpu feature
>>>>
>>>> On Wed, Dec 05, 2012 at 06:46:09AM -0500, Laszlo Hornyak wrote:
>>>>> ----- Original Message -----
>>>>>> From: "Yaniv Kaul" <ykaul at redhat.com>
>>>>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>>>> Sent: Wednesday, December 5, 2012 12:23:47 PM
>>>>>> Subject: Re: [Engine-devel] host cpu feature
>>>>>>
>>>>>> On 12/05/2012 12:32 PM, Laszlo Hornyak wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> CPU-Host support allows the virtual machines to see and utilize
>>>>>>> the
>>>>>>> host's CPU flags, this enables better performance in VM's, at
>>>>>>> the
>>>>>>> price of worse portablity.
>>>>>>>
>>>>>>> http://www.ovirt.org/Features/Cpu-host_Support
>>>>>>>
>>>>>>> Your feedback is welcome!
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Laszlo
>>>>>>> _______________________________________________
>>>>>>> Engine-devel mailing list
>>>>>>> Engine-devel at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>>>>>> - I assume that when you allow migration, you'd use host-model?
>>>>>> This
>>>>>> is
>>>>>> not clear from the design. It seems like we VDSM developers can
>>>>>> choose
>>>>>> to use either this or passthrough, while in practice we should
>>>>>> support both.
>>>> I join Kaul's question: it is an ovirt-level question whether
>>>> hostPassthrough or hostModel or both should be supported. It
>>>> should
>>>> not
>>>> be a unilateral vdsm decision.
>>> Ah, possibly misunderstanding, I did not mean that VDSM should
>>> decide whether to use host-passthrough or host-model. The engine
>>> should decide.
>>> I meant _you_ should decide which version of vdsm api modification
>>> do you want :)
>>>
>>>>> If AllowMigrateCPUHost is set to true (in case you have the same
>>>>> cpu model everywhere in your DC) migration of such hsots will be
>>>>> enabled. Otherwise it will not be enabled.
>>>> What is the breadth of AllowMigrateCPUHost? Engine wide? Per DC?
>>>> Per
>>>> cluster?
>>> I thought of eninge-wide. The of course you can have different
>>> models in two different DC, but they should be unique in one.
>>> We can add this to DC or cluster level, imho it would be just
>>> another checkbox on the UI that most users would not use.
>>>
>>>> I favor the latter; a user may have a cluster of exact-same hosts,
>>>> where
>>>> hostcpu migration is allowed, and other cluster where it is
>>>> forbiden.
>>>>
>>>> The nice thing about hostModel (unlike hostPassthrough) is that
>>>> once
>>>> we
>>>> created the VM we can migrate it to stronger hosts, and back to
>>>> the
>>>> original host. I suppose that it complicates the scheduler.
>>> Yes with host-model you get the features that libvirt handles. In
>>> such cases the engine could decide, if you want this
>>> functionality. Well the scheduler architecture is just being
>>> reinvented.
>>>
>>> For the host-passthrough, I think the AllowMigrateCPUHost
>>> configuration option would be a simple decision for the
>>> administrator: set it to true if all hosts are uniform. If it is
>>> not set to true, then we will not allow migration of such VMs.
>> That's not what I understood from libvirt's documentation. I
> You may be right, could you send an URL to that point of the documentation or copy-paste?

The link I followed from your feature page: 
http://libvirt.org/formatdomain.html#elementsCPU :

host-model
The host-model mode is essentially a shortcut to copying host CPU 
definition from capabilities XML into domain XML. Since the CPU 
definition is copied just before starting a domain, exactly the same XML 
can be used on different hosts while still providing the best guest CPU 
each host supports. Neither match attribute nor any feature elements can 
be used in this mode. Specifying CPU model is not supported either, but 
model's fallback attribute may still be used. Libvirt does not model 
every aspect of each CPU so the guest CPU will not match the host CPU 
exactly. On the other hand, the ABI provided to the guest is 
reproducible. During migration, complete CPU model definition is 
transferred to the destination host so the migrated guest will see 
exactly the same CPU model even if the destination host contains more 
capable CPUs for the running instance of the guest; but shutting down 
and restarting the guest may present different hardware to the guest 
according to the capabilities of the new host.
host-passthrough
With this mode, the CPU visible to the guest should be exactly the same 
as the host CPU even in the aspects that libvirt does not understand. 
Though the downside of this mode is that the guest environment cannot be 
reproduced on different hardware. Thus, if you hit any bugs, you are on 
your own. Neither model nor feature elements are allowed in this mode.


Y.

>
>> understood
>> that if you want host+migration, you need to use host-model.
>> Otherwise -
>> host-passthrough.
>> Y.
>>
>>>>>> - I'm still convinced and commented on both relevat oVirt and
>>>>>> libvirt
>>>>>> BZs that we need to add x2apic support to the CPU, regardless of
>>>>>> what
>>>>>> the host CPU exposes.
>>>>>> AFAIK, the KVM developers agree with me.
>>>>> Not quite sure how is this related... could you send some URL's
>>>>> for
>>>>> the bugreports?
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121205/8aee88d9/attachment.html>

From ykaul at redhat.com  Wed Dec  5 15:10:13 2012
From: ykaul at redhat.com (Yaniv Kaul)
Date: Wed, 05 Dec 2012 17:10:13 +0200
Subject: [Engine-devel] host cpu feature
In-Reply-To: <1135933549.6037019.1354719568977.JavaMail.root@redhat.com>
References: <1135933549.6037019.1354719568977.JavaMail.root@redhat.com>
Message-ID: <50BF63D5.9030009@redhat.com>

On 12/05/2012 04:59 PM, Laszlo Hornyak wrote:
>
> ----- Original Message -----
>> From: "Dan Kenigsberg" <danken at redhat.com>
>> To: "Yaniv Kaul" <ykaul at redhat.com>
>> Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
>> Sent: Wednesday, December 5, 2012 3:22:18 PM
>> Subject: Re: [Engine-devel] host cpu feature
>>
>> On Wed, Dec 05, 2012 at 04:05:00PM +0200, Yaniv Kaul wrote:
>>> On 12/05/2012 03:55 PM, Laszlo Hornyak wrote:
>>>>>>> The nice thing about hostModel (unlike hostPassthrough) is that
>>>>>>> once
>>>>>>> we
>>>>>>> created the VM we can migrate it to stronger hosts, and back to
>>>>>>> the
>>>>>>> original host. I suppose that it complicates the scheduler.
>>>>>> Yes with host-model you get the features that libvirt handles.
>>>>>> In
>>>>>> such cases the engine could decide, if you want this
>>>>>> functionality. Well the scheduler architecture is just being
>>>>>> reinvented.
>>>>>>
>>>>>> For the host-passthrough, I think the AllowMigrateCPUHost
>>>>>> configuration option would be a simple decision for the
>>>>>> administrator: set it to true if all hosts are uniform.
>> If it is THAT simple, Engine could take this decision without human
>> intervension.
> Is there a way engine can figure out if the cpu-models in all the hosts are the same?
> I mean even if some host flags are not handled by libvirt and therefore vdsm and engine...
> So I would really need that permission from the user.
>
>>>>>> If it is
>>>>>> not set to true, then we will not allow migration of such VMs.
>>>>> That's not what I understood from libvirt's documentation. I
>>>> You may be right, could you send an URL to that point of the
>>>> documentation or copy-paste?
>>> The link I followed from your feature page:
>>> http://libvirt.org/formatdomain.html#elementsCPU :
>>>
>>> host-model
>>> The host-model mode is essentially a shortcut to copying host CPU
>>> definition from capabilities XML into domain XML. Since the CPU
>>> definition is copied just before starting a domain, exactly the
>>> same
>>> XML can be used on different hosts while still providing the best
>>> guest CPU each host supports. Neither match attribute nor any
>>> feature elements can be used in this mode. Specifying CPU model is
>>> not supported either, but model's fallback attribute may still be
>>> used. Libvirt does not model every aspect of each CPU so the guest
>>> CPU will not match the host CPU exactly. On the other hand, the ABI
>>> provided to the guest is reproducible. During migration, complete
>>> CPU model definition is transferred to the destination host so the
>>> migrated guest will see exactly the same CPU model even if the
>>> destination host contains more capable CPUs for the running
>>> instance
>>> of the guest; but shutting down and restarting the guest may
>>> present
>>> different hardware to the guest according to the capabilities of
>>> the
>>> new host.
>>> host-passthrough
>>> With this mode, the CPU visible to the guest should be exactly the
>>> same as the host CPU even in the aspects that libvirt does not
>>> understand. Though the downside of this mode is that the guest
>>> environment cannot be reproduced on different hardware. Thus, if
>>> you
>>> hit any bugs, you are on your own.
>> That's exactly where AllowMigrateCPUHost fits well: when a user ticks
>> this for a cluster he says "yeah, I like to be on my own."
>>
> cpu mode="host-passthrough" migration: I talked to the libvirt guys and they said it is OK if the hardware and the software are the same, and it will work, but they would not recommend.
>
> So if they do not recommend it, I would drop this from the feature spec. Anyone against it?
>
> Laszlo

I'm a bit against it. I don't see why it's that complicated:
Allow migration -> use 'host-model'
Do not allow -> use 'host-passthrough'.

The reason of why we need host-passthrough is that otherwise I suspect 
we depend on libvirt for newer features to be somehow exposed to the 
guest (not sure about it).
Y.





From lhornyak at redhat.com  Wed Dec  5 15:24:59 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Wed, 5 Dec 2012 10:24:59 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <50BF63D5.9030009@redhat.com>
Message-ID: <1904927813.6054833.1354721099562.JavaMail.root@redhat.com>


----- Original Message -----
> From: "Yaniv Kaul" <ykaul at redhat.com>
> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> Cc: "Dan Kenigsberg" <danken at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 4:10:13 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> On 12/05/2012 04:59 PM, Laszlo Hornyak wrote:
> >
> > ----- Original Message -----
> >> From: "Dan Kenigsberg" <danken at redhat.com>
> >> To: "Yaniv Kaul" <ykaul at redhat.com>
> >> Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel"
> >> <engine-devel at ovirt.org>
> >> Sent: Wednesday, December 5, 2012 3:22:18 PM
> >> Subject: Re: [Engine-devel] host cpu feature
> >>
> >> On Wed, Dec 05, 2012 at 04:05:00PM +0200, Yaniv Kaul wrote:
> >>> On 12/05/2012 03:55 PM, Laszlo Hornyak wrote:
> >>>>>>> The nice thing about hostModel (unlike hostPassthrough) is
> >>>>>>> that
> >>>>>>> once
> >>>>>>> we
> >>>>>>> created the VM we can migrate it to stronger hosts, and back
> >>>>>>> to
> >>>>>>> the
> >>>>>>> original host. I suppose that it complicates the scheduler.
> >>>>>> Yes with host-model you get the features that libvirt handles.
> >>>>>> In
> >>>>>> such cases the engine could decide, if you want this
> >>>>>> functionality. Well the scheduler architecture is just being
> >>>>>> reinvented.
> >>>>>>
> >>>>>> For the host-passthrough, I think the AllowMigrateCPUHost
> >>>>>> configuration option would be a simple decision for the
> >>>>>> administrator: set it to true if all hosts are uniform.
> >> If it is THAT simple, Engine could take this decision without
> >> human
> >> intervension.
> > Is there a way engine can figure out if the cpu-models in all the
> > hosts are the same?
> > I mean even if some host flags are not handled by libvirt and
> > therefore vdsm and engine...
> > So I would really need that permission from the user.
> >
> >>>>>> If it is
> >>>>>> not set to true, then we will not allow migration of such VMs.
> >>>>> That's not what I understood from libvirt's documentation. I
> >>>> You may be right, could you send an URL to that point of the
> >>>> documentation or copy-paste?
> >>> The link I followed from your feature page:
> >>> http://libvirt.org/formatdomain.html#elementsCPU :
> >>>
> >>> host-model
> >>> The host-model mode is essentially a shortcut to copying host CPU
> >>> definition from capabilities XML into domain XML. Since the CPU
> >>> definition is copied just before starting a domain, exactly the
> >>> same
> >>> XML can be used on different hosts while still providing the best
> >>> guest CPU each host supports. Neither match attribute nor any
> >>> feature elements can be used in this mode. Specifying CPU model
> >>> is
> >>> not supported either, but model's fallback attribute may still be
> >>> used. Libvirt does not model every aspect of each CPU so the
> >>> guest
> >>> CPU will not match the host CPU exactly. On the other hand, the
> >>> ABI
> >>> provided to the guest is reproducible. During migration, complete
> >>> CPU model definition is transferred to the destination host so
> >>> the
> >>> migrated guest will see exactly the same CPU model even if the
> >>> destination host contains more capable CPUs for the running
> >>> instance
> >>> of the guest; but shutting down and restarting the guest may
> >>> present
> >>> different hardware to the guest according to the capabilities of
> >>> the
> >>> new host.
> >>> host-passthrough
> >>> With this mode, the CPU visible to the guest should be exactly
> >>> the
> >>> same as the host CPU even in the aspects that libvirt does not
> >>> understand. Though the downside of this mode is that the guest
> >>> environment cannot be reproduced on different hardware. Thus, if
> >>> you
> >>> hit any bugs, you are on your own.
> >> That's exactly where AllowMigrateCPUHost fits well: when a user
> >> ticks
> >> this for a cluster he says "yeah, I like to be on my own."
> >>
> > cpu mode="host-passthrough" migration: I talked to the libvirt guys
> > and they said it is OK if the hardware and the software are the
> > same, and it will work, but they would not recommend.
> >
> > So if they do not recommend it, I would drop this from the feature
> > spec. Anyone against it?
> >
> > Laszlo
> 
> I'm a bit against it. I don't see why it's that complicated:
> Allow migration -> use 'host-model'
> Do not allow -> use 'host-passthrough'.

So the actual cpu capabilities would depend not only on the 'host cpu' checkbox, but also on the 'pinned to host' checkbox. I think this logical trick would be both funny and confusing from the user's perspective.

> 
> The reason of why we need host-passthrough is that otherwise I
> suspect
> we depend on libvirt for newer features to be somehow exposed to the
> guest (not sure about it).

Yes, with other words: this is a tuning feature.

> Y.
> 
> 
> 
> 


From dfediuck at redhat.com  Wed Dec  5 16:33:35 2012
From: dfediuck at redhat.com (Doron Fediuck)
Date: Wed, 5 Dec 2012 11:33:35 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <1904927813.6054833.1354721099562.JavaMail.root@redhat.com>
Message-ID: <358793067.41316136.1354725215298.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Laszlo Hornyak" <lhornyak at redhat.com>
> To: "Yaniv Kaul" <ykaul at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 5:24:59 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> 
> ----- Original Message -----
> > From: "Yaniv Kaul" <ykaul at redhat.com>
> > To: "Laszlo Hornyak" <lhornyak at redhat.com>
> > Cc: "Dan Kenigsberg" <danken at redhat.com>, "engine-devel"
> > <engine-devel at ovirt.org>
> > Sent: Wednesday, December 5, 2012 4:10:13 PM
> > Subject: Re: [Engine-devel] host cpu feature
> > 
> > On 12/05/2012 04:59 PM, Laszlo Hornyak wrote:
> > >
> > > ----- Original Message -----
> > >> From: "Dan Kenigsberg" <danken at redhat.com>
> > >> To: "Yaniv Kaul" <ykaul at redhat.com>
> > >> Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel"
> > >> <engine-devel at ovirt.org>
> > >> Sent: Wednesday, December 5, 2012 3:22:18 PM
> > >> Subject: Re: [Engine-devel] host cpu feature
> > >>
> > >> On Wed, Dec 05, 2012 at 04:05:00PM +0200, Yaniv Kaul wrote:
> > >>> On 12/05/2012 03:55 PM, Laszlo Hornyak wrote:
> > >>>>>>> The nice thing about hostModel (unlike hostPassthrough) is
> > >>>>>>> that
> > >>>>>>> once
> > >>>>>>> we
> > >>>>>>> created the VM we can migrate it to stronger hosts, and
> > >>>>>>> back
> > >>>>>>> to
> > >>>>>>> the
> > >>>>>>> original host. I suppose that it complicates the scheduler.
> > >>>>>> Yes with host-model you get the features that libvirt
> > >>>>>> handles.
> > >>>>>> In
> > >>>>>> such cases the engine could decide, if you want this
> > >>>>>> functionality. Well the scheduler architecture is just being
> > >>>>>> reinvented.
> > >>>>>>
> > >>>>>> For the host-passthrough, I think the AllowMigrateCPUHost
> > >>>>>> configuration option would be a simple decision for the
> > >>>>>> administrator: set it to true if all hosts are uniform.
> > >> If it is THAT simple, Engine could take this decision without
> > >> human
> > >> intervension.
> > > Is there a way engine can figure out if the cpu-models in all the
> > > hosts are the same?
> > > I mean even if some host flags are not handled by libvirt and
> > > therefore vdsm and engine...
> > > So I would really need that permission from the user.
> > >
> > >>>>>> If it is
> > >>>>>> not set to true, then we will not allow migration of such
> > >>>>>> VMs.
> > >>>>> That's not what I understood from libvirt's documentation. I
> > >>>> You may be right, could you send an URL to that point of the
> > >>>> documentation or copy-paste?
> > >>> The link I followed from your feature page:
> > >>> http://libvirt.org/formatdomain.html#elementsCPU :
> > >>>
> > >>> host-model
> > >>> The host-model mode is essentially a shortcut to copying host
> > >>> CPU
> > >>> definition from capabilities XML into domain XML. Since the CPU
> > >>> definition is copied just before starting a domain, exactly the
> > >>> same
> > >>> XML can be used on different hosts while still providing the
> > >>> best
> > >>> guest CPU each host supports. Neither match attribute nor any
> > >>> feature elements can be used in this mode. Specifying CPU model
> > >>> is
> > >>> not supported either, but model's fallback attribute may still
> > >>> be
> > >>> used. Libvirt does not model every aspect of each CPU so the
> > >>> guest
> > >>> CPU will not match the host CPU exactly. On the other hand, the
> > >>> ABI
> > >>> provided to the guest is reproducible. During migration,
> > >>> complete
> > >>> CPU model definition is transferred to the destination host so
> > >>> the
> > >>> migrated guest will see exactly the same CPU model even if the
> > >>> destination host contains more capable CPUs for the running
> > >>> instance
> > >>> of the guest; but shutting down and restarting the guest may
> > >>> present
> > >>> different hardware to the guest according to the capabilities
> > >>> of
> > >>> the
> > >>> new host.
> > >>> host-passthrough
> > >>> With this mode, the CPU visible to the guest should be exactly
> > >>> the
> > >>> same as the host CPU even in the aspects that libvirt does not
> > >>> understand. Though the downside of this mode is that the guest
> > >>> environment cannot be reproduced on different hardware. Thus,
> > >>> if
> > >>> you
> > >>> hit any bugs, you are on your own.
> > >> That's exactly where AllowMigrateCPUHost fits well: when a user
> > >> ticks
> > >> this for a cluster he says "yeah, I like to be on my own."
> > >>
> > > cpu mode="host-passthrough" migration: I talked to the libvirt
> > > guys
> > > and they said it is OK if the hardware and the software are the
> > > same, and it will work, but they would not recommend.
> > >
> > > So if they do not recommend it, I would drop this from the
> > > feature
> > > spec. Anyone against it?
> > >
> > > Laszlo
> > 
> > I'm a bit against it. I don't see why it's that complicated:
> > Allow migration -> use 'host-model'
> > Do not allow -> use 'host-passthrough'.
> 
> So the actual cpu capabilities would depend not only on the 'host
> cpu' checkbox, but also on the 'pinned to host' checkbox. I think
> this logical trick would be both funny and confusing from the user's
> perspective.
> 
> > 
> > The reason of why we need host-passthrough is that otherwise I
> > suspect
> > we depend on libvirt for newer features to be somehow exposed to
> > the
> > guest (not sure about it).
> 
> Yes, with other words: this is a tuning feature.
> 

Let's keep it simple.
1. Please remove AllowMigrateCPUHost. No reason for us to do what libvirt is asking to void.
2. host-passthrough should be available only for non migratable VMs.


> > Y.
> > 
> > 
> > 
> > 


From lhornyak at redhat.com  Wed Dec  5 16:41:24 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Wed, 5 Dec 2012 11:41:24 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <358793067.41316136.1354725215298.JavaMail.root@redhat.com>
Message-ID: <1101558148.6093644.1354725684844.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Doron Fediuck" <dfediuck at redhat.com>
> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, "Yaniv Kaul" <ykaul at redhat.com>
> Sent: Wednesday, December 5, 2012 5:33:35 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> 
> 
> ----- Original Message -----
> > From: "Laszlo Hornyak" <lhornyak at redhat.com>
> > To: "Yaniv Kaul" <ykaul at redhat.com>
> > Cc: "engine-devel" <engine-devel at ovirt.org>
> > Sent: Wednesday, December 5, 2012 5:24:59 PM
> > Subject: Re: [Engine-devel] host cpu feature
> > 
> > 
> > ----- Original Message -----
> > > From: "Yaniv Kaul" <ykaul at redhat.com>
> > > To: "Laszlo Hornyak" <lhornyak at redhat.com>
> > > Cc: "Dan Kenigsberg" <danken at redhat.com>, "engine-devel"
> > > <engine-devel at ovirt.org>
> > > Sent: Wednesday, December 5, 2012 4:10:13 PM
> > > Subject: Re: [Engine-devel] host cpu feature
> > > 
> > > On 12/05/2012 04:59 PM, Laszlo Hornyak wrote:
> > > >
> > > > ----- Original Message -----
> > > >> From: "Dan Kenigsberg" <danken at redhat.com>
> > > >> To: "Yaniv Kaul" <ykaul at redhat.com>
> > > >> Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel"
> > > >> <engine-devel at ovirt.org>
> > > >> Sent: Wednesday, December 5, 2012 3:22:18 PM
> > > >> Subject: Re: [Engine-devel] host cpu feature
> > > >>
> > > >> On Wed, Dec 05, 2012 at 04:05:00PM +0200, Yaniv Kaul wrote:
> > > >>> On 12/05/2012 03:55 PM, Laszlo Hornyak wrote:
> > > >>>>>>> The nice thing about hostModel (unlike hostPassthrough)
> > > >>>>>>> is
> > > >>>>>>> that
> > > >>>>>>> once
> > > >>>>>>> we
> > > >>>>>>> created the VM we can migrate it to stronger hosts, and
> > > >>>>>>> back
> > > >>>>>>> to
> > > >>>>>>> the
> > > >>>>>>> original host. I suppose that it complicates the
> > > >>>>>>> scheduler.
> > > >>>>>> Yes with host-model you get the features that libvirt
> > > >>>>>> handles.
> > > >>>>>> In
> > > >>>>>> such cases the engine could decide, if you want this
> > > >>>>>> functionality. Well the scheduler architecture is just
> > > >>>>>> being
> > > >>>>>> reinvented.
> > > >>>>>>
> > > >>>>>> For the host-passthrough, I think the AllowMigrateCPUHost
> > > >>>>>> configuration option would be a simple decision for the
> > > >>>>>> administrator: set it to true if all hosts are uniform.
> > > >> If it is THAT simple, Engine could take this decision without
> > > >> human
> > > >> intervension.
> > > > Is there a way engine can figure out if the cpu-models in all
> > > > the
> > > > hosts are the same?
> > > > I mean even if some host flags are not handled by libvirt and
> > > > therefore vdsm and engine...
> > > > So I would really need that permission from the user.
> > > >
> > > >>>>>> If it is
> > > >>>>>> not set to true, then we will not allow migration of such
> > > >>>>>> VMs.
> > > >>>>> That's not what I understood from libvirt's documentation.
> > > >>>>> I
> > > >>>> You may be right, could you send an URL to that point of the
> > > >>>> documentation or copy-paste?
> > > >>> The link I followed from your feature page:
> > > >>> http://libvirt.org/formatdomain.html#elementsCPU :
> > > >>>
> > > >>> host-model
> > > >>> The host-model mode is essentially a shortcut to copying host
> > > >>> CPU
> > > >>> definition from capabilities XML into domain XML. Since the
> > > >>> CPU
> > > >>> definition is copied just before starting a domain, exactly
> > > >>> the
> > > >>> same
> > > >>> XML can be used on different hosts while still providing the
> > > >>> best
> > > >>> guest CPU each host supports. Neither match attribute nor any
> > > >>> feature elements can be used in this mode. Specifying CPU
> > > >>> model
> > > >>> is
> > > >>> not supported either, but model's fallback attribute may
> > > >>> still
> > > >>> be
> > > >>> used. Libvirt does not model every aspect of each CPU so the
> > > >>> guest
> > > >>> CPU will not match the host CPU exactly. On the other hand,
> > > >>> the
> > > >>> ABI
> > > >>> provided to the guest is reproducible. During migration,
> > > >>> complete
> > > >>> CPU model definition is transferred to the destination host
> > > >>> so
> > > >>> the
> > > >>> migrated guest will see exactly the same CPU model even if
> > > >>> the
> > > >>> destination host contains more capable CPUs for the running
> > > >>> instance
> > > >>> of the guest; but shutting down and restarting the guest may
> > > >>> present
> > > >>> different hardware to the guest according to the capabilities
> > > >>> of
> > > >>> the
> > > >>> new host.
> > > >>> host-passthrough
> > > >>> With this mode, the CPU visible to the guest should be
> > > >>> exactly
> > > >>> the
> > > >>> same as the host CPU even in the aspects that libvirt does
> > > >>> not
> > > >>> understand. Though the downside of this mode is that the
> > > >>> guest
> > > >>> environment cannot be reproduced on different hardware. Thus,
> > > >>> if
> > > >>> you
> > > >>> hit any bugs, you are on your own.
> > > >> That's exactly where AllowMigrateCPUHost fits well: when a
> > > >> user
> > > >> ticks
> > > >> this for a cluster he says "yeah, I like to be on my own."
> > > >>
> > > > cpu mode="host-passthrough" migration: I talked to the libvirt
> > > > guys
> > > > and they said it is OK if the hardware and the software are the
> > > > same, and it will work, but they would not recommend.
> > > >
> > > > So if they do not recommend it, I would drop this from the
> > > > feature
> > > > spec. Anyone against it?
> > > >
> > > > Laszlo
> > > 
> > > I'm a bit against it. I don't see why it's that complicated:
> > > Allow migration -> use 'host-model'
> > > Do not allow -> use 'host-passthrough'.
> > 
> > So the actual cpu capabilities would depend not only on the 'host
> > cpu' checkbox, but also on the 'pinned to host' checkbox. I think
> > this logical trick would be both funny and confusing from the
> > user's
> > perspective.
> > 
> > > 
> > > The reason of why we need host-passthrough is that otherwise I
> > > suspect
> > > we depend on libvirt for newer features to be somehow exposed to
> > > the
> > > guest (not sure about it).
> > 
> > Yes, with other words: this is a tuning feature.
> > 
> 
> Let's keep it simple.
> 1. Please remove AllowMigrateCPUHost. No reason for us to do what
> libvirt is asking to void.

With pleasure :)

> 2. host-passthrough should be available only for non migratable VMs.

Right.

And no host-model support for now, right?

> 
> 
> > > Y.
> > > 
> > > 
> > > 
> > > 
> 


From dfediuck at redhat.com  Wed Dec  5 16:46:00 2012
From: dfediuck at redhat.com (Doron Fediuck)
Date: Wed, 5 Dec 2012 11:46:00 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <1101558148.6093644.1354725684844.JavaMail.root@redhat.com>
Message-ID: <1413711406.41321044.1354725960945.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Laszlo Hornyak" <lhornyak at redhat.com>
> To: "Doron Fediuck" <dfediuck at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, "Yaniv Kaul" <ykaul at redhat.com>
> Sent: Wednesday, December 5, 2012 6:41:24 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> 
> 
> ----- Original Message -----
> > From: "Doron Fediuck" <dfediuck at redhat.com>
> > To: "Laszlo Hornyak" <lhornyak at redhat.com>
> > Cc: "engine-devel" <engine-devel at ovirt.org>, "Yaniv Kaul"
> > <ykaul at redhat.com>
> > Sent: Wednesday, December 5, 2012 5:33:35 PM
> > Subject: Re: [Engine-devel] host cpu feature
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Laszlo Hornyak" <lhornyak at redhat.com>
> > > To: "Yaniv Kaul" <ykaul at redhat.com>
> > > Cc: "engine-devel" <engine-devel at ovirt.org>
> > > Sent: Wednesday, December 5, 2012 5:24:59 PM
> > > Subject: Re: [Engine-devel] host cpu feature
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Yaniv Kaul" <ykaul at redhat.com>
> > > > To: "Laszlo Hornyak" <lhornyak at redhat.com>
> > > > Cc: "Dan Kenigsberg" <danken at redhat.com>, "engine-devel"
> > > > <engine-devel at ovirt.org>
> > > > Sent: Wednesday, December 5, 2012 4:10:13 PM
> > > > Subject: Re: [Engine-devel] host cpu feature
> > > > 
> > > > On 12/05/2012 04:59 PM, Laszlo Hornyak wrote:
> > > > >
> > > > > ----- Original Message -----
> > > > >> From: "Dan Kenigsberg" <danken at redhat.com>
> > > > >> To: "Yaniv Kaul" <ykaul at redhat.com>
> > > > >> Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel"
> > > > >> <engine-devel at ovirt.org>
> > > > >> Sent: Wednesday, December 5, 2012 3:22:18 PM
> > > > >> Subject: Re: [Engine-devel] host cpu feature
> > > > >>
> > > > >> On Wed, Dec 05, 2012 at 04:05:00PM +0200, Yaniv Kaul wrote:
> > > > >>> On 12/05/2012 03:55 PM, Laszlo Hornyak wrote:
> > > > >>>>>>> The nice thing about hostModel (unlike hostPassthrough)
> > > > >>>>>>> is
> > > > >>>>>>> that
> > > > >>>>>>> once
> > > > >>>>>>> we
> > > > >>>>>>> created the VM we can migrate it to stronger hosts, and
> > > > >>>>>>> back
> > > > >>>>>>> to
> > > > >>>>>>> the
> > > > >>>>>>> original host. I suppose that it complicates the
> > > > >>>>>>> scheduler.
> > > > >>>>>> Yes with host-model you get the features that libvirt
> > > > >>>>>> handles.
> > > > >>>>>> In
> > > > >>>>>> such cases the engine could decide, if you want this
> > > > >>>>>> functionality. Well the scheduler architecture is just
> > > > >>>>>> being
> > > > >>>>>> reinvented.
> > > > >>>>>>
> > > > >>>>>> For the host-passthrough, I think the
> > > > >>>>>> AllowMigrateCPUHost
> > > > >>>>>> configuration option would be a simple decision for the
> > > > >>>>>> administrator: set it to true if all hosts are uniform.
> > > > >> If it is THAT simple, Engine could take this decision
> > > > >> without
> > > > >> human
> > > > >> intervension.
> > > > > Is there a way engine can figure out if the cpu-models in all
> > > > > the
> > > > > hosts are the same?
> > > > > I mean even if some host flags are not handled by libvirt and
> > > > > therefore vdsm and engine...
> > > > > So I would really need that permission from the user.
> > > > >
> > > > >>>>>> If it is
> > > > >>>>>> not set to true, then we will not allow migration of
> > > > >>>>>> such
> > > > >>>>>> VMs.
> > > > >>>>> That's not what I understood from libvirt's
> > > > >>>>> documentation.
> > > > >>>>> I
> > > > >>>> You may be right, could you send an URL to that point of
> > > > >>>> the
> > > > >>>> documentation or copy-paste?
> > > > >>> The link I followed from your feature page:
> > > > >>> http://libvirt.org/formatdomain.html#elementsCPU :
> > > > >>>
> > > > >>> host-model
> > > > >>> The host-model mode is essentially a shortcut to copying
> > > > >>> host
> > > > >>> CPU
> > > > >>> definition from capabilities XML into domain XML. Since the
> > > > >>> CPU
> > > > >>> definition is copied just before starting a domain, exactly
> > > > >>> the
> > > > >>> same
> > > > >>> XML can be used on different hosts while still providing
> > > > >>> the
> > > > >>> best
> > > > >>> guest CPU each host supports. Neither match attribute nor
> > > > >>> any
> > > > >>> feature elements can be used in this mode. Specifying CPU
> > > > >>> model
> > > > >>> is
> > > > >>> not supported either, but model's fallback attribute may
> > > > >>> still
> > > > >>> be
> > > > >>> used. Libvirt does not model every aspect of each CPU so
> > > > >>> the
> > > > >>> guest
> > > > >>> CPU will not match the host CPU exactly. On the other hand,
> > > > >>> the
> > > > >>> ABI
> > > > >>> provided to the guest is reproducible. During migration,
> > > > >>> complete
> > > > >>> CPU model definition is transferred to the destination host
> > > > >>> so
> > > > >>> the
> > > > >>> migrated guest will see exactly the same CPU model even if
> > > > >>> the
> > > > >>> destination host contains more capable CPUs for the running
> > > > >>> instance
> > > > >>> of the guest; but shutting down and restarting the guest
> > > > >>> may
> > > > >>> present
> > > > >>> different hardware to the guest according to the
> > > > >>> capabilities
> > > > >>> of
> > > > >>> the
> > > > >>> new host.
> > > > >>> host-passthrough
> > > > >>> With this mode, the CPU visible to the guest should be
> > > > >>> exactly
> > > > >>> the
> > > > >>> same as the host CPU even in the aspects that libvirt does
> > > > >>> not
> > > > >>> understand. Though the downside of this mode is that the
> > > > >>> guest
> > > > >>> environment cannot be reproduced on different hardware.
> > > > >>> Thus,
> > > > >>> if
> > > > >>> you
> > > > >>> hit any bugs, you are on your own.
> > > > >> That's exactly where AllowMigrateCPUHost fits well: when a
> > > > >> user
> > > > >> ticks
> > > > >> this for a cluster he says "yeah, I like to be on my own."
> > > > >>
> > > > > cpu mode="host-passthrough" migration: I talked to the
> > > > > libvirt
> > > > > guys
> > > > > and they said it is OK if the hardware and the software are
> > > > > the
> > > > > same, and it will work, but they would not recommend.
> > > > >
> > > > > So if they do not recommend it, I would drop this from the
> > > > > feature
> > > > > spec. Anyone against it?
> > > > >
> > > > > Laszlo
> > > > 
> > > > I'm a bit against it. I don't see why it's that complicated:
> > > > Allow migration -> use 'host-model'
> > > > Do not allow -> use 'host-passthrough'.
> > > 
> > > So the actual cpu capabilities would depend not only on the 'host
> > > cpu' checkbox, but also on the 'pinned to host' checkbox. I think
> > > this logical trick would be both funny and confusing from the
> > > user's
> > > perspective.
> > > 
> > > > 
> > > > The reason of why we need host-passthrough is that otherwise I
> > > > suspect
> > > > we depend on libvirt for newer features to be somehow exposed
> > > > to
> > > > the
> > > > guest (not sure about it).
> > > 
> > > Yes, with other words: this is a tuning feature.
> > > 
> > 
> > Let's keep it simple.
> > 1. Please remove AllowMigrateCPUHost. No reason for us to do what
> > libvirt is asking to void.
> 
> With pleasure :)
> 
> > 2. host-passthrough should be available only for non migratable
> > VMs.
> 
> Right.
> 
> And no host-model support for now, right?
> 

Right.
In can later be added if we have a good reasoning for it.

> > 
> > 
> > > > Y.
> > > > 
> > > > 
> > > > 
> > > > 
> > 
> 


From ykaul at redhat.com  Wed Dec  5 16:48:33 2012
From: ykaul at redhat.com (Yaniv Kaul)
Date: Wed, 05 Dec 2012 18:48:33 +0200
Subject: [Engine-devel] host cpu feature
In-Reply-To: <1413711406.41321044.1354725960945.JavaMail.root@redhat.com>
References: <1413711406.41321044.1354725960945.JavaMail.root@redhat.com>
Message-ID: <50BF7AE1.7080803@redhat.com>

On 12/05/2012 06:46 PM, Doron Fediuck wrote:
>
> ----- Original Message -----
>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>> To: "Doron Fediuck" <dfediuck at redhat.com>
>> Cc: "engine-devel" <engine-devel at ovirt.org>, "Yaniv Kaul" <ykaul at redhat.com>
>> Sent: Wednesday, December 5, 2012 6:41:24 PM
>> Subject: Re: [Engine-devel] host cpu feature
>>
>>
>>
>> ----- Original Message -----
>>> From: "Doron Fediuck" <dfediuck at redhat.com>
>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>>> Cc: "engine-devel" <engine-devel at ovirt.org>, "Yaniv Kaul"
>>> <ykaul at redhat.com>
>>> Sent: Wednesday, December 5, 2012 5:33:35 PM
>>> Subject: Re: [Engine-devel] host cpu feature
>>>
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>> To: "Yaniv Kaul" <ykaul at redhat.com>
>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>> Sent: Wednesday, December 5, 2012 5:24:59 PM
>>>> Subject: Re: [Engine-devel] host cpu feature
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Yaniv Kaul" <ykaul at redhat.com>
>>>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>> Cc: "Dan Kenigsberg" <danken at redhat.com>, "engine-devel"
>>>>> <engine-devel at ovirt.org>
>>>>> Sent: Wednesday, December 5, 2012 4:10:13 PM
>>>>> Subject: Re: [Engine-devel] host cpu feature
>>>>>
>>>>> On 12/05/2012 04:59 PM, Laszlo Hornyak wrote:
>>>>>> ----- Original Message -----
>>>>>>> From: "Dan Kenigsberg" <danken at redhat.com>
>>>>>>> To: "Yaniv Kaul" <ykaul at redhat.com>
>>>>>>> Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel"
>>>>>>> <engine-devel at ovirt.org>
>>>>>>> Sent: Wednesday, December 5, 2012 3:22:18 PM
>>>>>>> Subject: Re: [Engine-devel] host cpu feature
>>>>>>>
>>>>>>> On Wed, Dec 05, 2012 at 04:05:00PM +0200, Yaniv Kaul wrote:
>>>>>>>> On 12/05/2012 03:55 PM, Laszlo Hornyak wrote:
>>>>>>>>>>>> The nice thing about hostModel (unlike hostPassthrough)
>>>>>>>>>>>> is
>>>>>>>>>>>> that
>>>>>>>>>>>> once
>>>>>>>>>>>> we
>>>>>>>>>>>> created the VM we can migrate it to stronger hosts, and
>>>>>>>>>>>> back
>>>>>>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>> original host. I suppose that it complicates the
>>>>>>>>>>>> scheduler.
>>>>>>>>>>> Yes with host-model you get the features that libvirt
>>>>>>>>>>> handles.
>>>>>>>>>>> In
>>>>>>>>>>> such cases the engine could decide, if you want this
>>>>>>>>>>> functionality. Well the scheduler architecture is just
>>>>>>>>>>> being
>>>>>>>>>>> reinvented.
>>>>>>>>>>>
>>>>>>>>>>> For the host-passthrough, I think the
>>>>>>>>>>> AllowMigrateCPUHost
>>>>>>>>>>> configuration option would be a simple decision for the
>>>>>>>>>>> administrator: set it to true if all hosts are uniform.
>>>>>>> If it is THAT simple, Engine could take this decision
>>>>>>> without
>>>>>>> human
>>>>>>> intervension.
>>>>>> Is there a way engine can figure out if the cpu-models in all
>>>>>> the
>>>>>> hosts are the same?
>>>>>> I mean even if some host flags are not handled by libvirt and
>>>>>> therefore vdsm and engine...
>>>>>> So I would really need that permission from the user.
>>>>>>
>>>>>>>>>>> If it is
>>>>>>>>>>> not set to true, then we will not allow migration of
>>>>>>>>>>> such
>>>>>>>>>>> VMs.
>>>>>>>>>> That's not what I understood from libvirt's
>>>>>>>>>> documentation.
>>>>>>>>>> I
>>>>>>>>> You may be right, could you send an URL to that point of
>>>>>>>>> the
>>>>>>>>> documentation or copy-paste?
>>>>>>>> The link I followed from your feature page:
>>>>>>>> http://libvirt.org/formatdomain.html#elementsCPU :
>>>>>>>>
>>>>>>>> host-model
>>>>>>>> The host-model mode is essentially a shortcut to copying
>>>>>>>> host
>>>>>>>> CPU
>>>>>>>> definition from capabilities XML into domain XML. Since the
>>>>>>>> CPU
>>>>>>>> definition is copied just before starting a domain, exactly
>>>>>>>> the
>>>>>>>> same
>>>>>>>> XML can be used on different hosts while still providing
>>>>>>>> the
>>>>>>>> best
>>>>>>>> guest CPU each host supports. Neither match attribute nor
>>>>>>>> any
>>>>>>>> feature elements can be used in this mode. Specifying CPU
>>>>>>>> model
>>>>>>>> is
>>>>>>>> not supported either, but model's fallback attribute may
>>>>>>>> still
>>>>>>>> be
>>>>>>>> used. Libvirt does not model every aspect of each CPU so
>>>>>>>> the
>>>>>>>> guest
>>>>>>>> CPU will not match the host CPU exactly. On the other hand,
>>>>>>>> the
>>>>>>>> ABI
>>>>>>>> provided to the guest is reproducible. During migration,
>>>>>>>> complete
>>>>>>>> CPU model definition is transferred to the destination host
>>>>>>>> so
>>>>>>>> the
>>>>>>>> migrated guest will see exactly the same CPU model even if
>>>>>>>> the
>>>>>>>> destination host contains more capable CPUs for the running
>>>>>>>> instance
>>>>>>>> of the guest; but shutting down and restarting the guest
>>>>>>>> may
>>>>>>>> present
>>>>>>>> different hardware to the guest according to the
>>>>>>>> capabilities
>>>>>>>> of
>>>>>>>> the
>>>>>>>> new host.
>>>>>>>> host-passthrough
>>>>>>>> With this mode, the CPU visible to the guest should be
>>>>>>>> exactly
>>>>>>>> the
>>>>>>>> same as the host CPU even in the aspects that libvirt does
>>>>>>>> not
>>>>>>>> understand. Though the downside of this mode is that the
>>>>>>>> guest
>>>>>>>> environment cannot be reproduced on different hardware.
>>>>>>>> Thus,
>>>>>>>> if
>>>>>>>> you
>>>>>>>> hit any bugs, you are on your own.
>>>>>>> That's exactly where AllowMigrateCPUHost fits well: when a
>>>>>>> user
>>>>>>> ticks
>>>>>>> this for a cluster he says "yeah, I like to be on my own."
>>>>>>>
>>>>>> cpu mode="host-passthrough" migration: I talked to the
>>>>>> libvirt
>>>>>> guys
>>>>>> and they said it is OK if the hardware and the software are
>>>>>> the
>>>>>> same, and it will work, but they would not recommend.
>>>>>>
>>>>>> So if they do not recommend it, I would drop this from the
>>>>>> feature
>>>>>> spec. Anyone against it?
>>>>>>
>>>>>> Laszlo
>>>>> I'm a bit against it. I don't see why it's that complicated:
>>>>> Allow migration -> use 'host-model'
>>>>> Do not allow -> use 'host-passthrough'.
>>>> So the actual cpu capabilities would depend not only on the 'host
>>>> cpu' checkbox, but also on the 'pinned to host' checkbox. I think
>>>> this logical trick would be both funny and confusing from the
>>>> user's
>>>> perspective.
>>>>
>>>>> The reason of why we need host-passthrough is that otherwise I
>>>>> suspect
>>>>> we depend on libvirt for newer features to be somehow exposed
>>>>> to
>>>>> the
>>>>> guest (not sure about it).
>>>> Yes, with other words: this is a tuning feature.
>>>>
>>> Let's keep it simple.
>>> 1. Please remove AllowMigrateCPUHost. No reason for us to do what
>>> libvirt is asking to void.
>> With pleasure :)
>>
>>> 2. host-passthrough should be available only for non migratable
>>> VMs.
>> Right.
>>
>> And no host-model support for now, right?
>>
> Right.
> In can later be added if we have a good reasoning for it.

Alternative idea, inspired by "Thus, if you hit any bugs, you are on 
your own" (http://libvirt.org/formatdomain.html#elementsCPU wrt 
'host-passthrough'):
A config option to determine if we use host-model or host-passthrough.
Y.

>
>>>
>>>>> Y.
>>>>>
>>>>>
>>>>>
>>>>>



From dfediuck at redhat.com  Wed Dec  5 17:10:55 2012
From: dfediuck at redhat.com (Doron Fediuck)
Date: Wed, 5 Dec 2012 12:10:55 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <50BF7AE1.7080803@redhat.com>
Message-ID: <718821734.41337789.1354727455679.JavaMail.root@redhat.com>

----- Original Message -----
> From: "Yaniv Kaul" <ykaul at redhat.com>
> To: "Doron Fediuck" <dfediuck at redhat.com>
> Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 6:48:33 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> On 12/05/2012 06:46 PM, Doron Fediuck wrote:
> >
> > ----- Original Message -----
> >> From: "Laszlo Hornyak" <lhornyak at redhat.com>
> >> To: "Doron Fediuck" <dfediuck at redhat.com>
> >> Cc: "engine-devel" <engine-devel at ovirt.org>, "Yaniv Kaul"
> >> <ykaul at redhat.com>
> >> Sent: Wednesday, December 5, 2012 6:41:24 PM
> >> Subject: Re: [Engine-devel] host cpu feature
> >>
> >>
> >>
> >> ----- Original Message -----
> >>> From: "Doron Fediuck" <dfediuck at redhat.com>
> >>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> >>> Cc: "engine-devel" <engine-devel at ovirt.org>, "Yaniv Kaul"
> >>> <ykaul at redhat.com>
> >>> Sent: Wednesday, December 5, 2012 5:33:35 PM
> >>> Subject: Re: [Engine-devel] host cpu feature
> >>>
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
> >>>> To: "Yaniv Kaul" <ykaul at redhat.com>
> >>>> Cc: "engine-devel" <engine-devel at ovirt.org>
> >>>> Sent: Wednesday, December 5, 2012 5:24:59 PM
> >>>> Subject: Re: [Engine-devel] host cpu feature
> >>>>
> >>>>
> >>>> ----- Original Message -----
> >>>>> From: "Yaniv Kaul" <ykaul at redhat.com>
> >>>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> >>>>> Cc: "Dan Kenigsberg" <danken at redhat.com>, "engine-devel"
> >>>>> <engine-devel at ovirt.org>
> >>>>> Sent: Wednesday, December 5, 2012 4:10:13 PM
> >>>>> Subject: Re: [Engine-devel] host cpu feature
> >>>>>
> >>>>> On 12/05/2012 04:59 PM, Laszlo Hornyak wrote:
> >>>>>> ----- Original Message -----
> >>>>>>> From: "Dan Kenigsberg" <danken at redhat.com>
> >>>>>>> To: "Yaniv Kaul" <ykaul at redhat.com>
> >>>>>>> Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel"
> >>>>>>> <engine-devel at ovirt.org>
> >>>>>>> Sent: Wednesday, December 5, 2012 3:22:18 PM
> >>>>>>> Subject: Re: [Engine-devel] host cpu feature
> >>>>>>>
> >>>>>>> On Wed, Dec 05, 2012 at 04:05:00PM +0200, Yaniv Kaul wrote:
> >>>>>>>> On 12/05/2012 03:55 PM, Laszlo Hornyak wrote:
> >>>>>>>>>>>> The nice thing about hostModel (unlike hostPassthrough)
> >>>>>>>>>>>> is
> >>>>>>>>>>>> that
> >>>>>>>>>>>> once
> >>>>>>>>>>>> we
> >>>>>>>>>>>> created the VM we can migrate it to stronger hosts, and
> >>>>>>>>>>>> back
> >>>>>>>>>>>> to
> >>>>>>>>>>>> the
> >>>>>>>>>>>> original host. I suppose that it complicates the
> >>>>>>>>>>>> scheduler.
> >>>>>>>>>>> Yes with host-model you get the features that libvirt
> >>>>>>>>>>> handles.
> >>>>>>>>>>> In
> >>>>>>>>>>> such cases the engine could decide, if you want this
> >>>>>>>>>>> functionality. Well the scheduler architecture is just
> >>>>>>>>>>> being
> >>>>>>>>>>> reinvented.
> >>>>>>>>>>>
> >>>>>>>>>>> For the host-passthrough, I think the
> >>>>>>>>>>> AllowMigrateCPUHost
> >>>>>>>>>>> configuration option would be a simple decision for the
> >>>>>>>>>>> administrator: set it to true if all hosts are uniform.
> >>>>>>> If it is THAT simple, Engine could take this decision
> >>>>>>> without
> >>>>>>> human
> >>>>>>> intervension.
> >>>>>> Is there a way engine can figure out if the cpu-models in all
> >>>>>> the
> >>>>>> hosts are the same?
> >>>>>> I mean even if some host flags are not handled by libvirt and
> >>>>>> therefore vdsm and engine...
> >>>>>> So I would really need that permission from the user.
> >>>>>>
> >>>>>>>>>>> If it is
> >>>>>>>>>>> not set to true, then we will not allow migration of
> >>>>>>>>>>> such
> >>>>>>>>>>> VMs.
> >>>>>>>>>> That's not what I understood from libvirt's
> >>>>>>>>>> documentation.
> >>>>>>>>>> I
> >>>>>>>>> You may be right, could you send an URL to that point of
> >>>>>>>>> the
> >>>>>>>>> documentation or copy-paste?
> >>>>>>>> The link I followed from your feature page:
> >>>>>>>> http://libvirt.org/formatdomain.html#elementsCPU :
> >>>>>>>>
> >>>>>>>> host-model
> >>>>>>>> The host-model mode is essentially a shortcut to copying
> >>>>>>>> host
> >>>>>>>> CPU
> >>>>>>>> definition from capabilities XML into domain XML. Since the
> >>>>>>>> CPU
> >>>>>>>> definition is copied just before starting a domain, exactly
> >>>>>>>> the
> >>>>>>>> same
> >>>>>>>> XML can be used on different hosts while still providing
> >>>>>>>> the
> >>>>>>>> best
> >>>>>>>> guest CPU each host supports. Neither match attribute nor
> >>>>>>>> any
> >>>>>>>> feature elements can be used in this mode. Specifying CPU
> >>>>>>>> model
> >>>>>>>> is
> >>>>>>>> not supported either, but model's fallback attribute may
> >>>>>>>> still
> >>>>>>>> be
> >>>>>>>> used. Libvirt does not model every aspect of each CPU so
> >>>>>>>> the
> >>>>>>>> guest
> >>>>>>>> CPU will not match the host CPU exactly. On the other hand,
> >>>>>>>> the
> >>>>>>>> ABI
> >>>>>>>> provided to the guest is reproducible. During migration,
> >>>>>>>> complete
> >>>>>>>> CPU model definition is transferred to the destination host
> >>>>>>>> so
> >>>>>>>> the
> >>>>>>>> migrated guest will see exactly the same CPU model even if
> >>>>>>>> the
> >>>>>>>> destination host contains more capable CPUs for the running
> >>>>>>>> instance
> >>>>>>>> of the guest; but shutting down and restarting the guest
> >>>>>>>> may
> >>>>>>>> present
> >>>>>>>> different hardware to the guest according to the
> >>>>>>>> capabilities
> >>>>>>>> of
> >>>>>>>> the
> >>>>>>>> new host.
> >>>>>>>> host-passthrough
> >>>>>>>> With this mode, the CPU visible to the guest should be
> >>>>>>>> exactly
> >>>>>>>> the
> >>>>>>>> same as the host CPU even in the aspects that libvirt does
> >>>>>>>> not
> >>>>>>>> understand. Though the downside of this mode is that the
> >>>>>>>> guest
> >>>>>>>> environment cannot be reproduced on different hardware.
> >>>>>>>> Thus,
> >>>>>>>> if
> >>>>>>>> you
> >>>>>>>> hit any bugs, you are on your own.
> >>>>>>> That's exactly where AllowMigrateCPUHost fits well: when a
> >>>>>>> user
> >>>>>>> ticks
> >>>>>>> this for a cluster he says "yeah, I like to be on my own."
> >>>>>>>
> >>>>>> cpu mode="host-passthrough" migration: I talked to the
> >>>>>> libvirt
> >>>>>> guys
> >>>>>> and they said it is OK if the hardware and the software are
> >>>>>> the
> >>>>>> same, and it will work, but they would not recommend.
> >>>>>>
> >>>>>> So if they do not recommend it, I would drop this from the
> >>>>>> feature
> >>>>>> spec. Anyone against it?
> >>>>>>
> >>>>>> Laszlo
> >>>>> I'm a bit against it. I don't see why it's that complicated:
> >>>>> Allow migration -> use 'host-model'
> >>>>> Do not allow -> use 'host-passthrough'.
> >>>> So the actual cpu capabilities would depend not only on the
> >>>> 'host
> >>>> cpu' checkbox, but also on the 'pinned to host' checkbox. I
> >>>> think
> >>>> this logical trick would be both funny and confusing from the
> >>>> user's
> >>>> perspective.
> >>>>
> >>>>> The reason of why we need host-passthrough is that otherwise I
> >>>>> suspect
> >>>>> we depend on libvirt for newer features to be somehow exposed
> >>>>> to
> >>>>> the
> >>>>> guest (not sure about it).
> >>>> Yes, with other words: this is a tuning feature.
> >>>>
> >>> Let's keep it simple.
> >>> 1. Please remove AllowMigrateCPUHost. No reason for us to do what
> >>> libvirt is asking to void.
> >> With pleasure :)
> >>
> >>> 2. host-passthrough should be available only for non migratable
> >>> VMs.
> >> Right.
> >>
> >> And no host-model support for now, right?
> >>
> > Right.
> > In can later be added if we have a good reasoning for it.
> 
> Alternative idea, inspired by "Thus, if you hit any bugs, you are on
> your own" (http://libvirt.org/formatdomain.html#elementsCPU wrt
> 'host-passthrough'):
> A config option to determine if we use host-model or
> host-passthrough.
> Y.
> 

I do not think the engine should go to this level.
ie- it can ask for passthrough as a feature, and the
actual implementation is handled by vdsm.


From ykaul at redhat.com  Wed Dec  5 17:13:47 2012
From: ykaul at redhat.com (Yaniv Kaul)
Date: Wed, 05 Dec 2012 19:13:47 +0200
Subject: [Engine-devel] host cpu feature
In-Reply-To: <718821734.41337789.1354727455679.JavaMail.root@redhat.com>
References: <718821734.41337789.1354727455679.JavaMail.root@redhat.com>
Message-ID: <50BF80CB.7010302@redhat.com>

On 12/05/2012 07:10 PM, Doron Fediuck wrote:
> ----- Original Message -----
>> From: "Yaniv Kaul" <ykaul at redhat.com>
>> To: "Doron Fediuck" <dfediuck at redhat.com>
>> Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
>> Sent: Wednesday, December 5, 2012 6:48:33 PM
>> Subject: Re: [Engine-devel] host cpu feature
>>
>> On 12/05/2012 06:46 PM, Doron Fediuck wrote:
>>> ----- Original Message -----
>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>> To: "Doron Fediuck" <dfediuck at redhat.com>
>>>> Cc: "engine-devel" <engine-devel at ovirt.org>, "Yaniv Kaul"
>>>> <ykaul at redhat.com>
>>>> Sent: Wednesday, December 5, 2012 6:41:24 PM
>>>> Subject: Re: [Engine-devel] host cpu feature
>>>>
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Doron Fediuck" <dfediuck at redhat.com>
>>>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>, "Yaniv Kaul"
>>>>> <ykaul at redhat.com>
>>>>> Sent: Wednesday, December 5, 2012 5:33:35 PM
>>>>> Subject: Re: [Engine-devel] host cpu feature
>>>>>
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>> To: "Yaniv Kaul" <ykaul at redhat.com>
>>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>>>> Sent: Wednesday, December 5, 2012 5:24:59 PM
>>>>>> Subject: Re: [Engine-devel] host cpu feature
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Yaniv Kaul" <ykaul at redhat.com>
>>>>>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>> Cc: "Dan Kenigsberg" <danken at redhat.com>, "engine-devel"
>>>>>>> <engine-devel at ovirt.org>
>>>>>>> Sent: Wednesday, December 5, 2012 4:10:13 PM
>>>>>>> Subject: Re: [Engine-devel] host cpu feature
>>>>>>>
>>>>>>> On 12/05/2012 04:59 PM, Laszlo Hornyak wrote:
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Dan Kenigsberg" <danken at redhat.com>
>>>>>>>>> To: "Yaniv Kaul" <ykaul at redhat.com>
>>>>>>>>> Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel"
>>>>>>>>> <engine-devel at ovirt.org>
>>>>>>>>> Sent: Wednesday, December 5, 2012 3:22:18 PM
>>>>>>>>> Subject: Re: [Engine-devel] host cpu feature
>>>>>>>>>
>>>>>>>>> On Wed, Dec 05, 2012 at 04:05:00PM +0200, Yaniv Kaul wrote:
>>>>>>>>>> On 12/05/2012 03:55 PM, Laszlo Hornyak wrote:
>>>>>>>>>>>>>> The nice thing about hostModel (unlike hostPassthrough)
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> once
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> created the VM we can migrate it to stronger hosts, and
>>>>>>>>>>>>>> back
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> original host. I suppose that it complicates the
>>>>>>>>>>>>>> scheduler.
>>>>>>>>>>>>> Yes with host-model you get the features that libvirt
>>>>>>>>>>>>> handles.
>>>>>>>>>>>>> In
>>>>>>>>>>>>> such cases the engine could decide, if you want this
>>>>>>>>>>>>> functionality. Well the scheduler architecture is just
>>>>>>>>>>>>> being
>>>>>>>>>>>>> reinvented.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For the host-passthrough, I think the
>>>>>>>>>>>>> AllowMigrateCPUHost
>>>>>>>>>>>>> configuration option would be a simple decision for the
>>>>>>>>>>>>> administrator: set it to true if all hosts are uniform.
>>>>>>>>> If it is THAT simple, Engine could take this decision
>>>>>>>>> without
>>>>>>>>> human
>>>>>>>>> intervension.
>>>>>>>> Is there a way engine can figure out if the cpu-models in all
>>>>>>>> the
>>>>>>>> hosts are the same?
>>>>>>>> I mean even if some host flags are not handled by libvirt and
>>>>>>>> therefore vdsm and engine...
>>>>>>>> So I would really need that permission from the user.
>>>>>>>>
>>>>>>>>>>>>> If it is
>>>>>>>>>>>>> not set to true, then we will not allow migration of
>>>>>>>>>>>>> such
>>>>>>>>>>>>> VMs.
>>>>>>>>>>>> That's not what I understood from libvirt's
>>>>>>>>>>>> documentation.
>>>>>>>>>>>> I
>>>>>>>>>>> You may be right, could you send an URL to that point of
>>>>>>>>>>> the
>>>>>>>>>>> documentation or copy-paste?
>>>>>>>>>> The link I followed from your feature page:
>>>>>>>>>> http://libvirt.org/formatdomain.html#elementsCPU :
>>>>>>>>>>
>>>>>>>>>> host-model
>>>>>>>>>> The host-model mode is essentially a shortcut to copying
>>>>>>>>>> host
>>>>>>>>>> CPU
>>>>>>>>>> definition from capabilities XML into domain XML. Since the
>>>>>>>>>> CPU
>>>>>>>>>> definition is copied just before starting a domain, exactly
>>>>>>>>>> the
>>>>>>>>>> same
>>>>>>>>>> XML can be used on different hosts while still providing
>>>>>>>>>> the
>>>>>>>>>> best
>>>>>>>>>> guest CPU each host supports. Neither match attribute nor
>>>>>>>>>> any
>>>>>>>>>> feature elements can be used in this mode. Specifying CPU
>>>>>>>>>> model
>>>>>>>>>> is
>>>>>>>>>> not supported either, but model's fallback attribute may
>>>>>>>>>> still
>>>>>>>>>> be
>>>>>>>>>> used. Libvirt does not model every aspect of each CPU so
>>>>>>>>>> the
>>>>>>>>>> guest
>>>>>>>>>> CPU will not match the host CPU exactly. On the other hand,
>>>>>>>>>> the
>>>>>>>>>> ABI
>>>>>>>>>> provided to the guest is reproducible. During migration,
>>>>>>>>>> complete
>>>>>>>>>> CPU model definition is transferred to the destination host
>>>>>>>>>> so
>>>>>>>>>> the
>>>>>>>>>> migrated guest will see exactly the same CPU model even if
>>>>>>>>>> the
>>>>>>>>>> destination host contains more capable CPUs for the running
>>>>>>>>>> instance
>>>>>>>>>> of the guest; but shutting down and restarting the guest
>>>>>>>>>> may
>>>>>>>>>> present
>>>>>>>>>> different hardware to the guest according to the
>>>>>>>>>> capabilities
>>>>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>> new host.
>>>>>>>>>> host-passthrough
>>>>>>>>>> With this mode, the CPU visible to the guest should be
>>>>>>>>>> exactly
>>>>>>>>>> the
>>>>>>>>>> same as the host CPU even in the aspects that libvirt does
>>>>>>>>>> not
>>>>>>>>>> understand. Though the downside of this mode is that the
>>>>>>>>>> guest
>>>>>>>>>> environment cannot be reproduced on different hardware.
>>>>>>>>>> Thus,
>>>>>>>>>> if
>>>>>>>>>> you
>>>>>>>>>> hit any bugs, you are on your own.
>>>>>>>>> That's exactly where AllowMigrateCPUHost fits well: when a
>>>>>>>>> user
>>>>>>>>> ticks
>>>>>>>>> this for a cluster he says "yeah, I like to be on my own."
>>>>>>>>>
>>>>>>>> cpu mode="host-passthrough" migration: I talked to the
>>>>>>>> libvirt
>>>>>>>> guys
>>>>>>>> and they said it is OK if the hardware and the software are
>>>>>>>> the
>>>>>>>> same, and it will work, but they would not recommend.
>>>>>>>>
>>>>>>>> So if they do not recommend it, I would drop this from the
>>>>>>>> feature
>>>>>>>> spec. Anyone against it?
>>>>>>>>
>>>>>>>> Laszlo
>>>>>>> I'm a bit against it. I don't see why it's that complicated:
>>>>>>> Allow migration -> use 'host-model'
>>>>>>> Do not allow -> use 'host-passthrough'.
>>>>>> So the actual cpu capabilities would depend not only on the
>>>>>> 'host
>>>>>> cpu' checkbox, but also on the 'pinned to host' checkbox. I
>>>>>> think
>>>>>> this logical trick would be both funny and confusing from the
>>>>>> user's
>>>>>> perspective.
>>>>>>
>>>>>>> The reason of why we need host-passthrough is that otherwise I
>>>>>>> suspect
>>>>>>> we depend on libvirt for newer features to be somehow exposed
>>>>>>> to
>>>>>>> the
>>>>>>> guest (not sure about it).
>>>>>> Yes, with other words: this is a tuning feature.
>>>>>>
>>>>> Let's keep it simple.
>>>>> 1. Please remove AllowMigrateCPUHost. No reason for us to do what
>>>>> libvirt is asking to void.
>>>> With pleasure :)
>>>>
>>>>> 2. host-passthrough should be available only for non migratable
>>>>> VMs.
>>>> Right.
>>>>
>>>> And no host-model support for now, right?
>>>>
>>> Right.
>>> In can later be added if we have a good reasoning for it.
>> Alternative idea, inspired by "Thus, if you hit any bugs, you are on
>> your own" (http://libvirt.org/formatdomain.html#elementsCPU wrt
>> 'host-passthrough'):
>> A config option to determine if we use host-model or
>> host-passthrough.
>> Y.
>>
> I do not think the engine should go to this level.

IMHO, it's not the engine, it's the user - the advanced user. But even 
an advanced user won't go and change VDSM's logic.
Lets hope that host-passthrough is well tested.
Y.

> ie- it can ask for passthrough as a feature, and the
> actual implementation is handled by vdsm.



From lhornyak at redhat.com  Wed Dec  5 17:14:46 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Wed, 5 Dec 2012 12:14:46 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <718821734.41337789.1354727455679.JavaMail.root@redhat.com>
Message-ID: <2071872846.6102754.1354727686717.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Doron Fediuck" <dfediuck at redhat.com>
> To: "Yaniv Kaul" <ykaul at redhat.com>
> Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 6:10:55 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> > Alternative idea, inspired by "Thus, if you hit any bugs, you are
> > on
> > your own" (http://libvirt.org/formatdomain.html#elementsCPU wrt
> > 'host-passthrough'):
> > A config option to determine if we use host-model or
> > host-passthrough.
> > Y.
> > 
> 
> I do not think the engine should go to this level.
> ie- it can ask for passthrough as a feature, and the
> actual implementation is handled by vdsm.
> 

If vdsm decides over host-passthrough or host-model, then how will the engine user know what exactly his VM gets. I think vdsm must be told exactly what to do.



From dfediuck at redhat.com  Wed Dec  5 17:37:40 2012
From: dfediuck at redhat.com (Doron Fediuck)
Date: Wed, 5 Dec 2012 12:37:40 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <2071872846.6102754.1354727686717.JavaMail.root@redhat.com>
Message-ID: <936058925.41365777.1354729060291.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Laszlo Hornyak" <lhornyak at redhat.com>
> To: "Doron Fediuck" <dfediuck at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 7:14:46 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> 
> 
> ----- Original Message -----
> > From: "Doron Fediuck" <dfediuck at redhat.com>
> > To: "Yaniv Kaul" <ykaul at redhat.com>
> > Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel"
> > <engine-devel at ovirt.org>
> > Sent: Wednesday, December 5, 2012 6:10:55 PM
> > Subject: Re: [Engine-devel] host cpu feature
> > 
> > > Alternative idea, inspired by "Thus, if you hit any bugs, you are
> > > on
> > > your own" (http://libvirt.org/formatdomain.html#elementsCPU wrt
> > > 'host-passthrough'):
> > > A config option to determine if we use host-model or
> > > host-passthrough.
> > > Y.
> > > 
> > 
> > I do not think the engine should go to this level.
> > ie- it can ask for passthrough as a feature, and the
> > actual implementation is handled by vdsm.
> > 
> 
> If vdsm decides over host-passthrough or host-model, then how will
> the engine user know what exactly his VM gets. I think vdsm must be
> told exactly what to do.
> 

VDSM maintains some level of independence. This is why it the engine
should be able to ask for passthrough as a feature. Otherwize vdsm will
handle it. So for now I suggest we stick to passthrough only, and if
we get an RFE for advanced mode we'll support the host model.


From snmishra at linux.vnet.ibm.com  Wed Dec  5 19:15:33 2012
From: snmishra at linux.vnet.ibm.com (snmishra at linux.vnet.ibm.com)
Date: Wed, 05 Dec 2012 11:15:33 -0800
Subject: [Engine-devel] host cpu feature
In-Reply-To: <936058925.41365777.1354729060291.JavaMail.root@redhat.com>
References: <936058925.41365777.1354729060291.JavaMail.root@redhat.com>
Message-ID: <20121205111533.Horde.F2LiPpir309Qv51VFznXIdA@imap.linux.ibm.com>


Quoting Doron Fediuck <dfediuck at redhat.com>:

> ----- Original Message -----
>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>> To: "Doron Fediuck" <dfediuck at redhat.com>
>> Cc: "engine-devel" <engine-devel at ovirt.org>
>> Sent: Wednesday, December 5, 2012 7:14:46 PM
>> Subject: Re: [Engine-devel] host cpu feature
>>
>>
>>
>> ----- Original Message -----
>> > From: "Doron Fediuck" <dfediuck at redhat.com>
>> > To: "Yaniv Kaul" <ykaul at redhat.com>
>> > Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel"
>> > <engine-devel at ovirt.org>
>> > Sent: Wednesday, December 5, 2012 6:10:55 PM
>> > Subject: Re: [Engine-devel] host cpu feature
>> >
>> > > Alternative idea, inspired by "Thus, if you hit any bugs, you are
>> > > on
>> > > your own" (http://libvirt.org/formatdomain.html#elementsCPU wrt
>> > > 'host-passthrough'):
>> > > A config option to determine if we use host-model or
>> > > host-passthrough.
>> > > Y.
>> > >
>> >
>> > I do not think the engine should go to this level.
>> > ie- it can ask for passthrough as a feature, and the
>> > actual implementation is handled by vdsm.
>> >
>>
>> If vdsm decides over host-passthrough or host-model, then how will
>> the engine user know what exactly his VM gets. I think vdsm must be
>> told exactly what to do.
>>
>
> VDSM maintains some level of independence. This is why it the engine
> should be able to ask for passthrough as a feature. Otherwize vdsm will
> handle it. So for now I suggest we stick to passthrough only, and if
> we get an RFE for advanced mode we'll support the host model.

What are we gaining by using passthrough over host-model? Looking at  
libvirt documentation, it seems that both modes give host CPU  
capabilities to guest VM. Whereas the downside of passthrough is that  
it limits migration. Whereas host-model will migrate to other hardware  
and if the destination hardware is better than source then the guest  
VM performance can be improved by rebooting guest.

As a stretch goal, ovirt can keep track of host capabilities and  
inform the user after migrating to a better host, that a reboot may  
improve guest performance.

Regards
Sharad Mishra

> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel





From iheim at redhat.com  Wed Dec  5 20:56:02 2012
From: iheim at redhat.com (Itamar Heim)
Date: Wed, 05 Dec 2012 22:56:02 +0200
Subject: [Engine-devel] host cpu feature
In-Reply-To: <20121205111533.Horde.F2LiPpir309Qv51VFznXIdA@imap.linux.ibm.com>
References: <936058925.41365777.1354729060291.JavaMail.root@redhat.com>
	<20121205111533.Horde.F2LiPpir309Qv51VFznXIdA@imap.linux.ibm.com>
Message-ID: <50BFB4E2.7060500@redhat.com>

On 12/05/2012 09:15 PM, snmishra at linux.vnet.ibm.com wrote:
>
> Quoting Doron Fediuck <dfediuck at redhat.com>:
>
>> ----- Original Message -----
>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>> To: "Doron Fediuck" <dfediuck at redhat.com>
>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>> Sent: Wednesday, December 5, 2012 7:14:46 PM
>>> Subject: Re: [Engine-devel] host cpu feature
>>>
>>>
>>>
>>> ----- Original Message -----
>>> > From: "Doron Fediuck" <dfediuck at redhat.com>
>>> > To: "Yaniv Kaul" <ykaul at redhat.com>
>>> > Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel"
>>> > <engine-devel at ovirt.org>
>>> > Sent: Wednesday, December 5, 2012 6:10:55 PM
>>> > Subject: Re: [Engine-devel] host cpu feature
>>> >
>>> > > Alternative idea, inspired by "Thus, if you hit any bugs, you are
>>> > > on
>>> > > your own" (http://libvirt.org/formatdomain.html#elementsCPU wrt
>>> > > 'host-passthrough'):
>>> > > A config option to determine if we use host-model or
>>> > > host-passthrough.
>>> > > Y.
>>> > >
>>> >
>>> > I do not think the engine should go to this level.
>>> > ie- it can ask for passthrough as a feature, and the
>>> > actual implementation is handled by vdsm.
>>> >
>>>
>>> If vdsm decides over host-passthrough or host-model, then how will
>>> the engine user know what exactly his VM gets. I think vdsm must be
>>> told exactly what to do.
>>>
>>
>> VDSM maintains some level of independence. This is why it the engine
>> should be able to ask for passthrough as a feature. Otherwize vdsm will
>> handle it. So for now I suggest we stick to passthrough only, and if
>> we get an RFE for advanced mode we'll support the host model.
>
> What are we gaining by using passthrough over host-model? Looking at
> libvirt documentation, it seems that both modes give host CPU
> capabilities to guest VM. Whereas the downside of passthrough is that it
> limits migration. Whereas host-model will migrate to other hardware and
> if the destination hardware is better than source then the guest VM
> performance can be improved by rebooting guest.
>
> As a stretch goal, ovirt can keep track of host capabilities and inform
> the user after migrating to a better host, that a reboot may improve
> guest performance.

pass-through may give better performance.
host-model would be relevant when we can support live migration inside 
the cluster for some of the nodes, which will be relevant when the 
scheduler is more pluggable/extendable than today.


From amureini at redhat.com  Thu Dec  6 06:47:17 2012
From: amureini at redhat.com (Allon Mureinik)
Date: Thu, 6 Dec 2012 01:47:17 -0500 (EST)
Subject: [Engine-devel] [Users] RHEVM 3.1 DB Schema diagram
In-Reply-To: <1679431457.18711815.1354704680489.JavaMail.root@redhat.com>
Message-ID: <1007363270.11241750.1354776437084.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Eli Mesika" <emesika at redhat.com>
> To: "Allon Mureinik" <amureini at redhat.com>
> Cc: "users" <users at ovirt.org>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 12:51:20 PM
> Subject: Re: [Users] RHEVM 3.1 DB Schema diagram
> 
> 
> 
> ----- Original Message -----
> > From: "Allon Mureinik" <amureini at redhat.com>
> > To: "Eli Mesika" <emesika at redhat.com>
> > Cc: "users" <users at ovirt.org>, "engine-devel"
> > <engine-devel at ovirt.org>
> > Sent: Wednesday, December 5, 2012 9:21:34 AM
> > Subject: Re: [Users] RHEVM 3.1 DB Schema diagram
> > 
> > Nice!
> > 
> > How are the names of the blocks generated?
> > Some names strike me a bit there (e.g., images is in the
> > storage_domains_static block).
> 
> Auto-generated ... (nice work IMHO)
That I get `-)
I was asking if you understand where the names of the blocks are taken from, because I couldn't.
> 
> > 
> > ----- Original Message -----
> > > From: "Eli Mesika" <emesika at redhat.com>
> > > To: "users" <users at ovirt.org>, "engine-devel"
> > > <engine-devel at ovirt.org>
> > > Sent: Tuesday, December 4, 2012 7:12:28 PM
> > > Subject: [Users] RHEVM 3.1 DB Schema diagram
> > > 
> > > Hi
> > > 
> > > There were some requests on the users list to get our DB schema
> > > diagram.
> > > After looking a while for a tool for generating that , I have
> > > found
> > > the DBSchema tool.
> > > 
> > > Please find attached the RHEVM 3.1 DB Schema diagram.
> > > 
> > > I intend to get it into the oVirt site but currently it's too
> > > large
> > > and does not fit well.
> > > 
> > > Regards
> > > 
> > > Eli Mesika
> > > 
> > > _______________________________________________
> > > Users mailing list
> > > Users at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/users
> > > 
> > 
> 


From shuming at linux.vnet.ibm.com  Thu Dec  6 16:02:02 2012
From: shuming at linux.vnet.ibm.com (Shu Ming)
Date: Fri, 07 Dec 2012 00:02:02 +0800
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <1661336497.13921817.1354657960654.JavaMail.root@redhat.com>
References: <1661336497.13921817.1354657960654.JavaMail.root@redhat.com>
Message-ID: <50C0C17A.9070000@linux.vnet.ibm.com>

Saggi,

Thanks for sharing your thought and I get some comments below.


Saggi Mizrahi:
> I've been throwing a lot of bits out about the new storage API and I think it's time to talk a bit.
> I will purposefully try and keep implementation details away and concentrate about how the API looks and how you use it.
>
> First major change is in terminology, there is no long a storage domain but a storage repository.
> This change is done because so many things are already called domain in the system and this will make things less confusing for new-commers with a libvirt background.
>
> One other changes is that repositories no longer have a UUID.
> The UUID was only used in the pool members manifest and is no longer needed.
>
>
> connectStorageRepository(repoId, repoFormat, connectionParameters={}):
> repoId - is a transient name that will be used to refer to the connected domain, it is not persisted and doesn't have to be the same across the cluster.
> repoFormat - Similar to what used to be type (eg. localfs-1.0, nfs-3.4, clvm-1.2).
> connectionParameters - This is format specific and will used to tell VDSM how to connect to the repo.


Where does repoID come from? I think repoID doesn't exist before 
connectStorageRepository() return.  Isn't repoID a return value of 
connectStorageRepository()?

>
> disconnectStorageRepository(self, repoId)
>
>
> In the new API there are only images, some images are mutable and some are not.
> mutable images are also called VirtualDisks
> immutable images are also called Snapshots
>
> There are no explicit templates, you can create as many images as you want from any snapshot.
>
> There are 4 major image operations:
>
>
> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
>                    userData={}, options={}):
>
> targetRepoId - ID of a connected repo where the disk will be created
> size - The size of the image you wish to create
> baseSnapshotId - the ID of the snapshot you want the base the new virtual disk on
> userData - optional data that will be attached to the new VD, could be anything that the user desires.
> options - options to modify VDSMs default behavior
>
> returns the id of the new VD

I think we will also need a function to check if a a VirtualDisk is 
based on a specific snapshot.
Like: isSnapshotOf(virtualDiskId, baseSnapshotID):

>
> createSnapshot(targetRepoId, baseVirtualDiskId,
>                 userData={}, options={}):
> targetRepoId - The ID of a connected repo where the new sanpshot will be created and the original image exists as well.
> size - The size of the image you wish to create
> baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want to snapshot
> userData - optional data that will be attached to the new Snapshot, could be anything that the user desires.
> options - options to modify VDSMs default behavior
>
> returns the id of the new Snapshot
>
> copyImage(targetRepoId, imageId, baseImageId=None, userData={}, options={})
> targetRepoId - The ID of a connected repo where the new image will be created
> imageId - The image you wish to copy
> baseImageId - if specified, the new image will contain only the diff between image and Id.
>                If None the new image will contain all the bits of image Id. This can be used to copy partial parts of images for export.
> userData - optional data that will be attached to the new image, could be anything that the user desires.
> options - options to modify VDSMs default behavior

Does this function mean that we can copy the image from one repository 
to another repository? Does it cover the semantics of storage migration, 
storage backup, storage incremental backup?

>
> return the Id of the new image. In case of copying an immutable image the ID will be identical to the original image as they contain the same data. However the user should not assume that and always use the value returned from the method.
>
> removeImage(repositoryId, imageId, options={}):
> repositoryId - The ID of a connected repo where the image to delete resides
> imageId - The id of the image you wish to delete.
>
>
> ----
> getImageStatus(repositoryId, imageId)
> repositoryId - The ID of a connected repo where the image to check resides
> imageId - The id of the image you wish to check.
>
> All operations return once the operations has been committed to disk NOT when the operation actually completes.
> This is done so that:
> - operation come to a stable state as quickly as possible.
> - In case where there is an SDM, only small portion of the operation actually needs to be performed on the SDM host.
> - No matter how many times the operation fails and on how many hosts, you can always resume the operation and choose when to do it.
> - You can stop an operation at any time and remove the resulting object making a distinction between "stop because the host is overloaded" to "I don't want that image"
>
> This means that after calling any operation that creates a new image the user must then call getImageStatus() to check what is the status of the image.
> The status of the image can be either optimized, degraded, or broken.
> "Optimized" means that the image is available and you can run VMs of it.
> "Degraded" means that the image is available and will run VMs but it might be a better way VDSM can represent the underlying data.

What does the "represent" mean here?
> "Broken" means that the image can't be used at the moment, probably because not all the data has been set up on the volume.
>
> Apart from that VDSM will also return the last persisted status information which will conatin
> hostID - the last host to try and optimize of fix the image
Any host can optimize the image? No need to be SDM?

> stage - X/Y (eg. 1/10) the last persisted stage of the fix.
> percent_complete - -1 or 0-100, the last persisted completion percentage of the aforementioned stage. -1 means that no progress is available for that operation.
> last_error - This will only be filled if the operation failed because of something other then IO or a VDSM crash for obvious reasons.
>               It will usually be set if the task was manually stopped
>
> The user can either be satisfied with that information or as the host specified in host ID if it is still working on that image by checking it's running tasks.

So we need a function to know what tasks are running on the image
>
> checkStorageRepository(self, repositoryId, options={}):
> A method to go over a storage repository and scan for any existing problems. This includes degraded\broken images and deleted images that have no yet been physically deleted\merged.
> It returns a list of Fix objects.
> Fix objects come in 4 types:
> clean - cleans data, run them to get more space.
> optimize - run them to optimize a degraded image
> merge - Merges two images together. Doing this sometimes
>          makes more images ready optimizing or cleaning.
>          The reason it is different from optimize is that
>          unmerged images are considered optimized.
> mend - mends a broken image
>
> The user can read these types and prioritize fixes. Fixes also contain opaque FIX data and they should be sent as received to
> fixStorageRepository(self, repositoryId, fix, options={}):
>
> That will start a fix operation.
>
>
> All major operations automatically start the appropriate "Fix" to bring the created object to an optimize\degraded state (the one that is quicker) unless one of the options is
> AutoFix=False. This is only useful for repos that might not be able to create volumes on all hosts (SDM) but would like to have the actual IO distributed in the cluster.
>
> Other common options is the strategy option:
> It has currently 2 possible values
> space and performance - In case VDSM has 2 ways of completing the same operation it will tell it to value one over the other. For example, whether to copy all the data or just create a qcow based of a snapshot.
> The default is space.
>
> You might have also noticed that it is never explicitly specified where to look for existing images. This is done purposefully, VDSM will always look in all connected repositories for existing objects.
> For very large setups this might be problematic. To mitigate the problem you have these options:
> participatingRepositories=[repoId, ...] which tell VDSM to narrow the search to just these repositories
> and
> imageHints={imgId: repoId} which will force VDSM to look for those image ID just in those repositories and fail if it doesn't find them there.
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


-- 
---
?? Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com or shuming at linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC




From amureini at redhat.com  Thu Dec  6 16:09:39 2012
From: amureini at redhat.com (Allon Mureinik)
Date: Thu, 6 Dec 2012 11:09:39 -0500 (EST)
Subject: [Engine-devel] JUnit 4.10 Upgrade
In-Reply-To: <119779564.11644107.1354810115342.JavaMail.root@redhat.com>
Message-ID: <418225466.11644727.1354810179043.JavaMail.root@redhat.com>

Hi guys,

Earlier this week I upgraded ovirt-engine to use JUnit 4.10.

The nifty new features can be found here:
https://github.com/KentBeck/junit/blob/master/doc/ReleaseNotes4.10.md


-Allon


From smizrahi at redhat.com  Thu Dec  6 16:52:01 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Thu, 6 Dec 2012 11:52:01 -0500 (EST)
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <50C0C17A.9070000@linux.vnet.ibm.com>
Message-ID: <1904075471.14922627.1354812721787.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "VDSM Project Development" <vdsm-devel at lists.fedorahosted.org>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Thursday, December 6, 2012 11:02:02 AM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> Saggi,
> 
> Thanks for sharing your thought and I get some comments below.
> 
> 
> Saggi Mizrahi:
> > I've been throwing a lot of bits out about the new storage API and
> > I think it's time to talk a bit.
> > I will purposefully try and keep implementation details away and
> > concentrate about how the API looks and how you use it.
> >
> > First major change is in terminology, there is no long a storage
> > domain but a storage repository.
> > This change is done because so many things are already called
> > domain in the system and this will make things less confusing for
> > new-commers with a libvirt background.
> >
> > One other changes is that repositories no longer have a UUID.
> > The UUID was only used in the pool members manifest and is no
> > longer needed.
> >
> >
> > connectStorageRepository(repoId, repoFormat,
> > connectionParameters={}):
> > repoId - is a transient name that will be used to refer to the
> > connected domain, it is not persisted and doesn't have to be the
> > same across the cluster.
> > repoFormat - Similar to what used to be type (eg. localfs-1.0,
> > nfs-3.4, clvm-1.2).
> > connectionParameters - This is format specific and will used to
> > tell VDSM how to connect to the repo.
> 
> 
> Where does repoID come from? I think repoID doesn't exist before
> connectStorageRepository() return.  Isn't repoID a return value of
> connectStorageRepository()?
No, repoIDs are no longer part of the domain, they are just a transient handle.
The user can put whatever it wants there as long as it isn't already taken by another currently connected domain.
> 
> >
> > disconnectStorageRepository(self, repoId)
> >
> >
> > In the new API there are only images, some images are mutable and
> > some are not.
> > mutable images are also called VirtualDisks
> > immutable images are also called Snapshots
> >
> > There are no explicit templates, you can create as many images as
> > you want from any snapshot.
> >
> > There are 4 major image operations:
> >
> >
> > createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> >                    userData={}, options={}):
> >
> > targetRepoId - ID of a connected repo where the disk will be
> > created
> > size - The size of the image you wish to create
> > baseSnapshotId - the ID of the snapshot you want the base the new
> > virtual disk on
> > userData - optional data that will be attached to the new VD, could
> > be anything that the user desires.
> > options - options to modify VDSMs default behavior
> >
> > returns the id of the new VD
> 
> I think we will also need a function to check if a a VirtualDisk is
> based on a specific snapshot.
> Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
No, the design is that volume dependencies are an implementation detail.
There is no reason for you to know that an image is physically a snapshot of another.
Logical snapshots, template information, and any other information can be set by the user by using the userData field available for every image.
> 
> >
> > createSnapshot(targetRepoId, baseVirtualDiskId,
> >                 userData={}, options={}):
> > targetRepoId - The ID of a connected repo where the new sanpshot
> > will be created and the original image exists as well.
> > size - The size of the image you wish to create
> > baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want
> > to snapshot
> > userData - optional data that will be attached to the new Snapshot,
> > could be anything that the user desires.
> > options - options to modify VDSMs default behavior
> >
> > returns the id of the new Snapshot
> >
> > copyImage(targetRepoId, imageId, baseImageId=None, userData={},
> > options={})
> > targetRepoId - The ID of a connected repo where the new image will
> > be created
> > imageId - The image you wish to copy
> > baseImageId - if specified, the new image will contain only the
> > diff between image and Id.
> >                If None the new image will contain all the bits of
> >                image Id. This can be used to copy partial parts of
> >                images for export.
> > userData - optional data that will be attached to the new image,
> > could be anything that the user desires.
> > options - options to modify VDSMs default behavior
> 
> Does this function mean that we can copy the image from one
> repository
> to another repository? Does it cover the semantics of storage
> migration,
> storage backup, storage incremental backup?
Yes, the main purpose is copying to another repo. and you can even do incremental backups.
Also the following flow
1. Run a VM using imageA
2. write to disk
3. Stop VM
4. copy imageA to repoB
5. Run a VM using imageA again
6. Write to disk
7. Stop VM
8. Copy imageA again basing it of imageA_copy1 on repoB creating a diff on repo diff without snapshotting the original image.

> 
> >
> > return the Id of the new image. In case of copying an immutable
> > image the ID will be identical to the original image as they
> > contain the same data. However the user should not assume that and
> > always use the value returned from the method.
> >
> > removeImage(repositoryId, imageId, options={}):
> > repositoryId - The ID of a connected repo where the image to delete
> > resides
> > imageId - The id of the image you wish to delete.
> >
> >
> > ----
> > getImageStatus(repositoryId, imageId)
> > repositoryId - The ID of a connected repo where the image to check
> > resides
> > imageId - The id of the image you wish to check.
> >
> > All operations return once the operations has been committed to
> > disk NOT when the operation actually completes.
> > This is done so that:
> > - operation come to a stable state as quickly as possible.
> > - In case where there is an SDM, only small portion of the
> > operation actually needs to be performed on the SDM host.
> > - No matter how many times the operation fails and on how many
> > hosts, you can always resume the operation and choose when to do
> > it.
> > - You can stop an operation at any time and remove the resulting
> > object making a distinction between "stop because the host is
> > overloaded" to "I don't want that image"
> >
> > This means that after calling any operation that creates a new
> > image the user must then call getImageStatus() to check what is
> > the status of the image.
> > The status of the image can be either optimized, degraded, or
> > broken.
> > "Optimized" means that the image is available and you can run VMs
> > of it.
> > "Degraded" means that the image is available and will run VMs but
> > it might be a better way VDSM can represent the underlying data.
> 
> What does the "represent" mean here?
Anything, but mostly image formate RAW\QCOW2 when performance strategy has been selected.
> > "Broken" means that the image can't be used at the moment, probably
> > because not all the data has been set up on the volume.
> >
> > Apart from that VDSM will also return the last persisted status
> > information which will conatin
> > hostID - the last host to try and optimize of fix the image
> Any host can optimize the image? No need to be SDM?
On anything but lvm based block domains there will not even be an SDM.
On SDM based domains we will try as hard as we can to have as many operations executable on any host.
> 
> > stage - X/Y (eg. 1/10) the last persisted stage of the fix.
> > percent_complete - -1 or 0-100, the last persisted completion
> > percentage of the aforementioned stage. -1 means that no progress
> > is available for that operation.
> > last_error - This will only be filled if the operation failed
> > because of something other then IO or a VDSM crash for obvious
> > reasons.
> >               It will usually be set if the task was manually
> >               stopped
> >
> > The user can either be satisfied with that information or as the
> > host specified in host ID if it is still working on that image by
> > checking it's running tasks.
> 
> So we need a function to know what tasks are running on the image
getImageStatus()
> >
> > checkStorageRepository(self, repositoryId, options={}):
> > A method to go over a storage repository and scan for any existing
> > problems. This includes degraded\broken images and deleted images
> > that have no yet been physically deleted\merged.
> > It returns a list of Fix objects.
> > Fix objects come in 4 types:
> > clean - cleans data, run them to get more space.
> > optimize - run them to optimize a degraded image
> > merge - Merges two images together. Doing this sometimes
> >          makes more images ready optimizing or cleaning.
> >          The reason it is different from optimize is that
> >          unmerged images are considered optimized.
> > mend - mends a broken image
> >
> > The user can read these types and prioritize fixes. Fixes also
> > contain opaque FIX data and they should be sent as received to
> > fixStorageRepository(self, repositoryId, fix, options={}):
> >
> > That will start a fix operation.
> >
> >
> > All major operations automatically start the appropriate "Fix" to
> > bring the created object to an optimize\degraded state (the one
> > that is quicker) unless one of the options is
> > AutoFix=False. This is only useful for repos that might not be able
> > to create volumes on all hosts (SDM) but would like to have the
> > actual IO distributed in the cluster.
> >
> > Other common options is the strategy option:
> > It has currently 2 possible values
> > space and performance - In case VDSM has 2 ways of completing the
> > same operation it will tell it to value one over the other. For
> > example, whether to copy all the data or just create a qcow based
> > of a snapshot.
> > The default is space.
> >
> > You might have also noticed that it is never explicitly specified
> > where to look for existing images. This is done purposefully, VDSM
> > will always look in all connected repositories for existing
> > objects.
> > For very large setups this might be problematic. To mitigate the
> > problem you have these options:
> > participatingRepositories=[repoId, ...] which tell VDSM to narrow
> > the search to just these repositories
> > and
> > imageHints={imgId: repoId} which will force VDSM to look for those
> > image ID just in those repositories and fail if it doesn't find
> > them there.
> > _______________________________________________
> > vdsm-devel mailing list
> > vdsm-devel at lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 
> 
> --
> ---
> ?? Shu Ming
> Open Virtualization Engineerning; CSTL, IBM Corp.
> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com or
> shuming at linux.vnet.ibm.com
> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> District, Beijing 100193, PRC
> 
> 
> 


From vszocs at redhat.com  Thu Dec  6 23:50:17 2012
From: vszocs at redhat.com (Vojtech Szocs)
Date: Thu, 6 Dec 2012 18:50:17 -0500 (EST)
Subject: [Engine-devel] UI Plugins: PoC patch revision 7 update 2 is now
	available
In-Reply-To: <314409108.15026919.1354831183281.JavaMail.root@redhat.com>
Message-ID: <571020179.15061949.1354837817205.JavaMail.root@redhat.com>

Hi guys, 

an updated version of UI Plugins proof-of-c on cept patch is now available for you to experiment with. It's basically revision 7 with new features, driven mostly by the feedback received so far. 

You can download and apply latest UI Plugins patches from oVirt Gerrit code review system: 

    * rev is ion 6 - http://gerrit.ovirt.org/#/c/8120/ 
    * rev is ion 7 update 2 - http://gerrit.ovirt.org/#/c/9250/ 

Please read on to learn what's new. If you have any comments, questi on s or ideas, please let me know! 



Engine REST API integration 

UI plugin infrastructure no longer closes current REST API session upon WebAdmin user logout. 

Since REST API session management uses JSESSIONID cookie to transmit session ID from client to server, the cookie will be present even after user signs out of WebAdmin. Upon subsequent login, this cookie will be sent as part of another "acquire REST API session" request, which means REST API will potentially reuse existing session based on JSESSIONID cookie value. 

Known limitation: due to REST API integration requiring WebAdmin user password when acquiring session, RestApiSessionAcquired function will NOT be called in case of auto login. This happens when the user signs into WebAdmin, closes the browser window (tab) without signing out, and opens it again - WebAdmin signs the user in automatically. In this case, there is no way (yet) to query for user password, which is required when acquiring REST API session. To work around this limitation, just sign out and sign in again. We're planning to address this limitation in future update. 

Other than that, REST API integration works the same as before, calling RestApiSessionAcquired function on plugins each time new or existing session is acquired, and keeping the session alive through heartbeat requests while the user stays authenticated. It is therefore guaranteed that RestApiSessionAcquired function will be called each time the user signs into WebAdmin. 



New API function: addSubTab 

It's now possible to add custom sub tabs under existing main tabs. 

For example, to add custom sub tab under Host main tab: 

api.addSubTab('Host', 'My Host Sub Tab', 'my-host-sub-tab', 'http://www.ovirt.org/'); 

The signature of addSubTab function is following: 

addSubTab(entityTypeName, label, historyToken, contentUrl) 

entityTypeName indicates the main tab entity type, i.e. under which main tab the sub tab should be added. See org.ovirt.engine.ui.webadmin.plugin.entity.EntityType enum for supported values. 

label , historyToken and contentUrl have same semantics as in addMainTab function. 



New API function: setTabContentUrl 

Use this function to update custom main or sub tab content URL, for example: 

api.setTabContentUrl( 'my-host-sub-tab' , 'http://www.example.org/' ); 



New API function: setTabAccessible 

Use this function to control access to custom main or sub tab, for example: 

api.setTabAccessible( 'my-host-sub-tab' , false); 

In the example above, the custom tab is set as not accessible. In practice, this means two things: 

    * corresponding tab header will be hidden on GUI 
    * any attempt to navigate to this tab (e.g. by modifying URL with token representing the tab) will be denied by the plugin infrastructure 



New events: UserLogin and UserLogout 

Plugins can now be notified when user logs into or out of WebAdmin GUI. 

For example: 

api.register({ 
UserLogin: function(userName, userId) { 
... 
}, 
UserLogout: function() { 
... 
} 
}); 



New events: {EntityTypeName}SelectionChange 

Plugins can now be notified whenever existing main tab selection changes . 

For example, a plugin could listen to Host main tab selection change like so: 

api.register({ 
HostSelectionChange: function() { 
var hostsSelected = arguments.length; 
var firstHostId = arguments[0].entityId; 
} 
}); 

For each supported entity type, {EntityTypeName}SelectionChange event is defined: ClusterSelectionChange, DataCenterSelectionChange, DiskSelectionChange, HostSelectionChange, StorageSelectionChange, TemplateSelectionChange, VirtualMachineSelectionChange. 

Note that each {EntityTypeName}SelectionChange event handler function receives currently selected items as arguments. 



Minor improvements 

Browser popup window triggered via showDialog API function now shows scrollbars if the content doesn't fit the window. 

Note on showDialog function implementation: we're planning to replace the current (browser-based) popup implementation with one that properly integrates with WebAdmin dialog infrastructure. This is something that's still on my TODO list, planning to do this in near future. 



Regards, 
Vojtech 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121206/adc39b0d/attachment.html>

From deepakcs at linux.vnet.ibm.com  Fri Dec  7 05:23:15 2012
From: deepakcs at linux.vnet.ibm.com (Deepak C Shetty)
Date: Fri, 07 Dec 2012 10:53:15 +0530
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <1904075471.14922627.1354812721787.JavaMail.root@redhat.com>
References: <1904075471.14922627.1354812721787.JavaMail.root@redhat.com>
Message-ID: <50C17D43.7010509@linux.vnet.ibm.com>

On 12/06/2012 10:22 PM, Saggi Mizrahi wrote:
>
> ----- Original Message -----
>> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
>> Cc: "VDSM Project Development" <vdsm-devel at lists.fedorahosted.org>, "engine-devel" <engine-devel at ovirt.org>
>> Sent: Thursday, December 6, 2012 11:02:02 AM
>> Subject: Re: [vdsm] RFC: New Storage API
>>
>> Saggi,
>>
>> Thanks for sharing your thought and I get some comments below.
>>
>>
>> Saggi Mizrahi:
>>> I've been throwing a lot of bits out about the new storage API and
>>> I think it's time to talk a bit.
>>> I will purposefully try and keep implementation details away and
>>> concentrate about how the API looks and how you use it.
>>>
>>> First major change is in terminology, there is no long a storage
>>> domain but a storage repository.
>>> This change is done because so many things are already called
>>> domain in the system and this will make things less confusing for
>>> new-commers with a libvirt background.
>>>
>>> One other changes is that repositories no longer have a UUID.
>>> The UUID was only used in the pool members manifest and is no
>>> longer needed.
>>>
>>>
>>> connectStorageRepository(repoId, repoFormat,
>>> connectionParameters={}):
>>> repoId - is a transient name that will be used to refer to the
>>> connected domain, it is not persisted and doesn't have to be the
>>> same across the cluster.
>>> repoFormat - Similar to what used to be type (eg. localfs-1.0,
>>> nfs-3.4, clvm-1.2).
>>> connectionParameters - This is format specific and will used to
>>> tell VDSM how to connect to the repo.
>>
>> Where does repoID come from? I think repoID doesn't exist before
>> connectStorageRepository() return.  Isn't repoID a return value of
>> connectStorageRepository()?
> No, repoIDs are no longer part of the domain, they are just a transient handle.
> The user can put whatever it wants there as long as it isn't already taken by another currently connected domain.

So what happens when user mistakenly gives a repoID that is in use 
before.. there should be something in the return value that specifies 
the error and/or reason for error so that user can try with a new/diff 
repoID ?

>>> disconnectStorageRepository(self, repoId)
>>>
>>>
>>> In the new API there are only images, some images are mutable and
>>> some are not.
>>> mutable images are also called VirtualDisks
>>> immutable images are also called Snapshots
>>>
>>> There are no explicit templates, you can create as many images as
>>> you want from any snapshot.
>>>
>>> There are 4 major image operations:
>>>
>>>
>>> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
>>>                     userData={}, options={}):
>>>
>>> targetRepoId - ID of a connected repo where the disk will be
>>> created
>>> size - The size of the image you wish to create
>>> baseSnapshotId - the ID of the snapshot you want the base the new
>>> virtual disk on
>>> userData - optional data that will be attached to the new VD, could
>>> be anything that the user desires.
>>> options - options to modify VDSMs default behavior

IIUC, i can use options to do storage offloads ? For eg. I can create a 
LUN that represents this VD on my storage array based on the 'options' 
parameter ? Is this the intended way to use 'options' ?

>>>
>>> returns the id of the new VD
>> I think we will also need a function to check if a a VirtualDisk is
>> based on a specific snapshot.
>> Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
> No, the design is that volume dependencies are an implementation detail.
> There is no reason for you to know that an image is physically a snapshot of another.
> Logical snapshots, template information, and any other information can be set by the user by using the userData field available for every image.
>>> createSnapshot(targetRepoId, baseVirtualDiskId,
>>>                  userData={}, options={}):
>>> targetRepoId - The ID of a connected repo where the new sanpshot
>>> will be created and the original image exists as well.
>>> size - The size of the image you wish to create
>>> baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want
>>> to snapshot
>>> userData - optional data that will be attached to the new Snapshot,
>>> could be anything that the user desires.
>>> options - options to modify VDSMs default behavior
>>>
>>> returns the id of the new Snapshot
>>>
>>> copyImage(targetRepoId, imageId, baseImageId=None, userData={},
>>> options={})
>>> targetRepoId - The ID of a connected repo where the new image will
>>> be created
>>> imageId - The image you wish to copy
>>> baseImageId - if specified, the new image will contain only the
>>> diff between image and Id.
>>>                 If None the new image will contain all the bits of
>>>                 image Id. This can be used to copy partial parts of
>>>                 images for export.
>>> userData - optional data that will be attached to the new image,
>>> could be anything that the user desires.
>>> options - options to modify VDSMs default behavior
>> Does this function mean that we can copy the image from one
>> repository
>> to another repository? Does it cover the semantics of storage
>> migration,
>> storage backup, storage incremental backup?
> Yes, the main purpose is copying to another repo. and you can even do incremental backups.
> Also the following flow
> 1. Run a VM using imageA
> 2. write to disk
> 3. Stop VM
> 4. copy imageA to repoB
> 5. Run a VM using imageA again
> 6. Write to disk
> 7. Stop VM
> 8. Copy imageA again basing it of imageA_copy1 on repoB creating a diff on repo diff without snapshotting the original image.
>
>>> return the Id of the new image. In case of copying an immutable
>>> image the ID will be identical to the original image as they
>>> contain the same data. However the user should not assume that and
>>> always use the value returned from the method.
>>>
>>> removeImage(repositoryId, imageId, options={}):
>>> repositoryId - The ID of a connected repo where the image to delete
>>> resides
>>> imageId - The id of the image you wish to delete.
>>>
>>>
>>> ----
>>> getImageStatus(repositoryId, imageId)
>>> repositoryId - The ID of a connected repo where the image to check
>>> resides
>>> imageId - The id of the image you wish to check.
>>>
>>> All operations return once the operations has been committed to
>>> disk NOT when the operation actually completes.
>>> This is done so that:
>>> - operation come to a stable state as quickly as possible.
>>> - In case where there is an SDM, only small portion of the
>>> operation actually needs to be performed on the SDM host.
>>> - No matter how many times the operation fails and on how many
>>> hosts, you can always resume the operation and choose when to do
>>> it.
>>> - You can stop an operation at any time and remove the resulting
>>> object making a distinction between "stop because the host is
>>> overloaded" to "I don't want that image"
>>>
>>> This means that after calling any operation that creates a new
>>> image the user must then call getImageStatus() to check what is
>>> the status of the image.
>>> The status of the image can be either optimized, degraded, or
>>> broken.
>>> "Optimized" means that the image is available and you can run VMs
>>> of it.
>>> "Degraded" means that the image is available and will run VMs but
>>> it might be a better way VDSM can represent the underlying data.

Calling qcow2 based snapshot degraded (meaning its degraded in perf, as 
its space optimzed ) and calling raw images as optimised ( meaning its 
optimised for perf as its space in-efficient) is confusing. Degraded 
sounds like a bad thing when seen by the end-user :) I think there is 
scope for having some better and less confusing terminology here ?

>> What does the "represent" mean here?
> Anything, but mostly image formate RAW\QCOW2 when performance strategy has been selected.
>>> "Broken" means that the image can't be used at the moment, probably
>>> because not all the data has been set up on the volume.
>>>
>>> Apart from that VDSM will also return the last persisted status
>>> information which will conatin
>>> hostID - the last host to try and optimize of fix the image
>> Any host can optimize the image? No need to be SDM?
> On anything but lvm based block domains there will not even be an SDM.
> On SDM based domains we will try as hard as we can to have as many operations executable on any host.

1) Can you provide more info on why there is a exception for 'lvm based 
block domain'. Its not coming out clearly.
2) Based on the terminology change, domain is now replaced by 
repository, SDM should now be more aptly called SRM (storage repo 
manager) so that we are consistent in the usage of terminology
3) Can you provide some example flow / scenario to understnad how with 
and without SDM domains work ? Especially how the disk based lock is 
taken if no SDM ?

>>> stage - X/Y (eg. 1/10) the last persisted stage of the fix.
>>> percent_complete - -1 or 0-100, the last persisted completion
>>> percentage of the aforementioned stage. -1 means that no progress
>>> is available for that operation.
>>> last_error - This will only be filled if the operation failed
>>> because of something other then IO or a VDSM crash for obvious
>>> reasons.
>>>                It will usually be set if the task was manually
>>>                stopped
>>>
>>> The user can either be satisfied with that information or as the
>>> host specified in host ID if it is still working on that image by
>>> checking it's running tasks.
>> So we need a function to know what tasks are running on the image
> getImageStatus()
>>> checkStorageRepository(self, repositoryId, options={}):
>>> A method to go over a storage repository and scan for any existing
>>> problems. This includes degraded\broken images and deleted images
>>> that have no yet been physically deleted\merged.
>>> It returns a list of Fix objects.
>>> Fix objects come in 4 types:
>>> clean - cleans data, run them to get more space.
>>> optimize - run them to optimize a degraded image
>>> merge - Merges two images together. Doing this sometimes
>>>           makes more images ready optimizing or cleaning.
>>>           The reason it is different from optimize is that
>>>           unmerged images are considered optimized.
>>> mend - mends a broken image
>>>
>>> The user can read these types and prioritize fixes. Fixes also
>>> contain opaque FIX data and they should be sent as received to
>>> fixStorageRepository(self, repositoryId, fix, options={}):
>>>
>>> That will start a fix operation.

It would be good if you can provide some example or flow of "fix" operation.
When and Why would somebody want to do it ?

Does 'Fix' here mean that i move from raw to qcow2 format or vice-versa, 
or there is more to it ?

>>>
>>>
>>> All major operations automatically start the appropriate "Fix" to
>>> bring the created object to an optimize\degraded state (the one
>>> that is quicker) unless one of the options is
>>> AutoFix=False. This is only useful for repos that might not be able
>>> to create volumes on all hosts (SDM) but would like to have the
>>> actual IO distributed in the cluster.
>>>
>>> Other common options is the strategy option:
>>> It has currently 2 possible values
>>> space and performance - In case VDSM has 2 ways of completing the
>>> same operation it will tell it to value one over the other. For
>>> example, whether to copy all the data or just create a qcow based
>>> of a snapshot.
>>> The default is space.
>>>
>>> You might have also noticed that it is never explicitly specified
>>> where to look for existing images. This is done purposefully, VDSM
>>> will always look in all connected repositories for existing
>>> objects.
>>> For very large setups this might be problematic. To mitigate the
>>> problem you have these options:
>>> participatingRepositories=[repoId, ...] which tell VDSM to narrow
>>> the search to just these repositories
>>> and
>>> imageHints={imgId: repoId} which will force VDSM to look for those
>>> image ID just in those repositories and fail if it doesn't find
>>> them there.
>>> _______________________________________________
>>> vdsm-devel mailing list
>>> vdsm-devel at lists.fedorahosted.org
>>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>>
>> --
>> ---
>> ?? Shu Ming
>> Open Virtualization Engineerning; CSTL, IBM Corp.
>> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com or
>> shuming at linux.vnet.ibm.com
>> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
>> District, Beijing 100193, PRC
>>
>>
>>
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel



From shuming at linux.vnet.ibm.com  Fri Dec  7 06:37:20 2012
From: shuming at linux.vnet.ibm.com (Shu Ming)
Date: Fri, 07 Dec 2012 14:37:20 +0800
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <50C17D43.7010509@linux.vnet.ibm.com>
References: <1904075471.14922627.1354812721787.JavaMail.root@redhat.com>
	<50C17D43.7010509@linux.vnet.ibm.com>
Message-ID: <50C18EA0.6080400@linux.vnet.ibm.com>

? 2012-12-7 13:23, Deepak C Shetty:
> On 12/06/2012 10:22 PM, Saggi Mizrahi wrote:
>>
>> ----- Original Message -----
>>> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
>>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
>>> Cc: "VDSM Project Development" <vdsm-devel at lists.fedorahosted.org>, 
>>> "engine-devel" <engine-devel at ovirt.org>
>>> Sent: Thursday, December 6, 2012 11:02:02 AM
>>> Subject: Re: [vdsm] RFC: New Storage API
>>>
>>> Saggi,
>>>
>>> Thanks for sharing your thought and I get some comments below.
>>>
>>>
>>> Saggi Mizrahi:
>>>> I've been throwing a lot of bits out about the new storage API and
>>>> I think it's time to talk a bit.
>>>> I will purposefully try and keep implementation details away and
>>>> concentrate about how the API looks and how you use it.
>>>>
>>>> First major change is in terminology, there is no long a storage
>>>> domain but a storage repository.
>>>> This change is done because so many things are already called
>>>> domain in the system and this will make things less confusing for
>>>> new-commers with a libvirt background.
>>>>
>>>> One other changes is that repositories no longer have a UUID.
>>>> The UUID was only used in the pool members manifest and is no
>>>> longer needed.
>>>>
>>>>
>>>> connectStorageRepository(repoId, repoFormat,
>>>> connectionParameters={}):
>>>> repoId - is a transient name that will be used to refer to the
>>>> connected domain, it is not persisted and doesn't have to be the
>>>> same across the cluster.
>>>> repoFormat - Similar to what used to be type (eg. localfs-1.0,
>>>> nfs-3.4, clvm-1.2).
>>>> connectionParameters - This is format specific and will used to
>>>> tell VDSM how to connect to the repo.
>>>
>>> Where does repoID come from? I think repoID doesn't exist before
>>> connectStorageRepository() return.  Isn't repoID a return value of
>>> connectStorageRepository()?
>> No, repoIDs are no longer part of the domain, they are just a 
>> transient handle.
>> The user can put whatever it wants there as long as it isn't already 
>> taken by another currently connected domain.
>
> So what happens when user mistakenly gives a repoID that is in use 
> before.. there should be something in the return value that specifies 
> the error and/or reason for error so that user can try with a new/diff 
> repoID ?

I think let the user to give the repoID is meaningless and error-prune.  
Developer must maintain a a unique ID list for every storage repository 
connected.

>
>>>> disconnectStorageRepository(self, repoId)
>>>>
>>>>
>>>> In the new API there are only images, some images are mutable and
>>>> some are not.
>>>> mutable images are also called VirtualDisks
>>>> immutable images are also called Snapshots
>>>>
>>>> There are no explicit templates, you can create as many images as
>>>> you want from any snapshot.
>>>>
>>>> There are 4 major image operations:
>>>>
>>>>
>>>> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
>>>>                     userData={}, options={}):
>>>>
>>>> targetRepoId - ID of a connected repo where the disk will be
>>>> created
>>>> size - The size of the image you wish to create
>>>> baseSnapshotId - the ID of the snapshot you want the base the new
>>>> virtual disk on
>>>> userData - optional data that will be attached to the new VD, could
>>>> be anything that the user desires.
>>>> options - options to modify VDSMs default behavior
>
> IIUC, i can use options to do storage offloads ? For eg. I can create 
> a LUN that represents this VD on my storage array based on the 
> 'options' parameter ? Is this the intended way to use 'options' ?
>
>>>>
>>>> returns the id of the new VD
>>> I think we will also need a function to check if a a VirtualDisk is
>>> based on a specific snapshot.
>>> Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
>> No, the design is that volume dependencies are an implementation detail.
>> There is no reason for you to know that an image is physically a 
>> snapshot of another.
>> Logical snapshots, template information, and any other information 
>> can be set by the user by using the userData field available for 
>> every image.
>>>> createSnapshot(targetRepoId, baseVirtualDiskId,
>>>>                  userData={}, options={}):
>>>> targetRepoId - The ID of a connected repo where the new sanpshot
>>>> will be created and the original image exists as well.
>>>> size - The size of the image you wish to create
>>>> baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want
>>>> to snapshot
>>>> userData - optional data that will be attached to the new Snapshot,
>>>> could be anything that the user desires.
>>>> options - options to modify VDSMs default behavior
>>>>
>>>> returns the id of the new Snapshot
>>>>
>>>> copyImage(targetRepoId, imageId, baseImageId=None, userData={},
>>>> options={})
>>>> targetRepoId - The ID of a connected repo where the new image will
>>>> be created
>>>> imageId - The image you wish to copy
>>>> baseImageId - if specified, the new image will contain only the
>>>> diff between image and Id.
>>>>                 If None the new image will contain all the bits of
>>>>                 image Id. This can be used to copy partial parts of
>>>>                 images for export.
>>>> userData - optional data that will be attached to the new image,
>>>> could be anything that the user desires.
>>>> options - options to modify VDSMs default behavior
>>> Does this function mean that we can copy the image from one
>>> repository
>>> to another repository? Does it cover the semantics of storage
>>> migration,
>>> storage backup, storage incremental backup?
>> Yes, the main purpose is copying to another repo. and you can even do 
>> incremental backups.
>> Also the following flow
>> 1. Run a VM using imageA
>> 2. write to disk
>> 3. Stop VM
>> 4. copy imageA to repoB
>> 5. Run a VM using imageA again
>> 6. Write to disk
>> 7. Stop VM
>> 8. Copy imageA again basing it of imageA_copy1 on repoB creating a 
>> diff on repo diff without snapshotting the original image.
>>>> return the Id of the new image. In case of copying an immutable
>>>> image the ID will be identical to the original image as they
>>>> contain the same data. However the user should not assume that and
>>>> always use the value returned from the method.
Do you mean the ID returned by copyImage() will be the same as the 
origin ID?  It is okay for copying to a different repository than the 
original repository because they have different namespace. However, if 
we do the copy inside in the same repository, we should get a new 
different ID.


>>>>
>>>> removeImage(repositoryId, imageId, options={}):
>>>> repositoryId - The ID of a connected repo where the image to delete
>>>> resides
>>>> imageId - The id of the image you wish to delete.
>>>>
>>>>
>>>> ----
>>>> getImageStatus(repositoryId, imageId)
>>>> repositoryId - The ID of a connected repo where the image to check
>>>> resides
>>>> imageId - The id of the image you wish to check.
>>>>
>>>> All operations return once the operations has been committed to
>>>> disk NOT when the operation actually completes.
>>>> This is done so that:
>>>> - operation come to a stable state as quickly as possible.
>>>> - In case where there is an SDM, only small portion of the
>>>> operation actually needs to be performed on the SDM host.
>>>> - No matter how many times the operation fails and on how many
>>>> hosts, you can always resume the operation and choose when to do
>>>> it.
>>>> - You can stop an operation at any time and remove the resulting
>>>> object making a distinction between "stop because the host is
>>>> overloaded" to "I don't want that image"
>>>>
>>>> This means that after calling any operation that creates a new
>>>> image the user must then call getImageStatus() to check what is
>>>> the status of the image.
>>>> The status of the image can be either optimized, degraded, or
>>>> broken.
>>>> "Optimized" means that the image is available and you can run VMs
>>>> of it.
>>>> "Degraded" means that the image is available and will run VMs but
>>>> it might be a better way VDSM can represent the underlying data.
>
> Calling qcow2 based snapshot degraded (meaning its degraded in perf, 
> as its space optimzed ) and calling raw images as optimised ( meaning 
> its optimised for perf as its space in-efficient) is confusing. 
> Degraded sounds like a bad thing when seen by the end-user :) I think 
> there is scope for having some better and less confusing terminology 
> here ?
>
>>> What does the "represent" mean here?
>> Anything, but mostly image formate RAW\QCOW2 when performance 
>> strategy has been selected.
>>>> "Broken" means that the image can't be used at the moment, probably
>>>> because not all the data has been set up on the volume.
>>>>
>>>> Apart from that VDSM will also return the last persisted status
>>>> information which will conatin
>>>> hostID - the last host to try and optimize of fix the image
>>> Any host can optimize the image? No need to be SDM?
>> On anything but lvm based block domains there will not even be an SDM.
>> On SDM based domains we will try as hard as we can to have as many 
>> operations executable on any host.
>
> 1) Can you provide more info on why there is a exception for 'lvm 
> based block domain'. Its not coming out clearly.

I guess Saggi means we can leverage CLVM to build a cluster based LVM 
group and no need to have SDM.  If I understand correctly, in this 
proposal,  we manage different types of the repositories at the same 
type, say, one localFS repository, one iSCSI repository, one nfs 
repository at the same time, also the images can be copied to each other 
back and forth.  When you copy a image from repository without SDM to a 
repository with SDM, we should use the later SDM to execute the copy 
operation.

> 2) Based on the terminology change, domain is now replaced by 
> repository, SDM should now be more aptly called SRM (storage repo 
> manager) so that we are consistent in the usage of terminology
> 3) Can you provide some example flow / scenario to understnad how with 
> and without SDM domains work ? Especially how the disk based lock is 
> taken if no SDM ?
>
>>>> stage - X/Y (eg. 1/10) the last persisted stage of the fix.
>>>> percent_complete - -1 or 0-100, the last persisted completion
>>>> percentage of the aforementioned stage. -1 means that no progress
>>>> is available for that operation.
>>>> last_error - This will only be filled if the operation failed
>>>> because of something other then IO or a VDSM crash for obvious
>>>> reasons.
>>>>                It will usually be set if the task was manually
>>>>                stopped
>>>>
>>>> The user can either be satisfied with that information or as the
>>>> host specified in host ID if it is still working on that image by
>>>> checking it's running tasks.
>>> So we need a function to know what tasks are running on the image
>> getImageStatus()
>>>> checkStorageRepository(self, repositoryId, options={}):
>>>> A method to go over a storage repository and scan for any existing
>>>> problems. This includes degraded\broken images and deleted images
>>>> that have no yet been physically deleted\merged.
>>>> It returns a list of Fix objects.
>>>> Fix objects come in 4 types:
>>>> clean - cleans data, run them to get more space.
>>>> optimize - run them to optimize a degraded image
>>>> merge - Merges two images together. Doing this sometimes
>>>>           makes more images ready optimizing or cleaning.
>>>>           The reason it is different from optimize is that
>>>>           unmerged images are considered optimized.
>>>> mend - mends a broken image

What is a broken image? Does broken mean corruption?  If it is 
corruption, we may need some recoverable checksum for each image.

>>>>
>>>> The user can read these types and prioritize fixes. Fixes also
>>>> contain opaque FIX data and they should be sent as received to
>>>> fixStorageRepository(self, repositoryId, fix, options={}):
>>>>
>>>> That will start a fix operation.
>
> It would be good if you can provide some example or flow of "fix" 
> operation.
> When and Why would somebody want to do it ?
>
> Does 'Fix' here mean that i move from raw to qcow2 format or 
> vice-versa, or there is more to it ?
>
>>>>
>>>>
>>>> All major operations automatically start the appropriate "Fix" to
>>>> bring the created object to an optimize\degraded state (the one
>>>> that is quicker) unless one of the options is
>>>> AutoFix=False. This is only useful for repos that might not be able
>>>> to create volumes on all hosts (SDM) but would like to have the
>>>> actual IO distributed in the cluster.
>>>>
>>>> Other common options is the strategy option:
>>>> It has currently 2 possible values
>>>> space and performance - In case VDSM has 2 ways of completing the
>>>> same operation it will tell it to value one over the other. For
>>>> example, whether to copy all the data or just create a qcow based
>>>> of a snapshot.
>>>> The default is space.
>>>>
>>>> You might have also noticed that it is never explicitly specified
>>>> where to look for existing images. This is done purposefully, VDSM
>>>> will always look in all connected repositories for existing
>>>> objects.
>>>> For very large setups this might be problematic. To mitigate the
>>>> problem you have these options:
>>>> participatingRepositories=[repoId, ...] which tell VDSM to narrow
>>>> the search to just these repositories
>>>> and
>>>> imageHints={imgId: repoId} which will force VDSM to look for those
>>>> image ID just in those repositories and fail if it doesn't find
>>>> them there.
>>>> _______________________________________________
>>>> vdsm-devel mailing list
>>>> vdsm-devel at lists.fedorahosted.org
>>>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>>>
>>> -- 
>>> ---
>>> ?? Shu Ming
>>> Open Virtualization Engineerning; CSTL, IBM Corp.
>>> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com or
>>> shuming at linux.vnet.ibm.com
>>> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
>>> District, Beijing 100193, PRC
>>>
>>>
>>>
>> _______________________________________________
>> vdsm-devel mailing list
>> vdsm-devel at lists.fedorahosted.org
>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>


-- 
---
?? Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com or shuming at linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC




From lhornyak at redhat.com  Fri Dec  7 09:20:51 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Fri, 7 Dec 2012 04:20:51 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <50BFB4E2.7060500@redhat.com>
Message-ID: <149108836.6671114.1354872051114.JavaMail.root@redhat.com>

Thank you all for your feedback, I moved the feature to implementation status.

Patches:
 - engine http://gerrit.ovirt.org/#/c/9753/
 - vdsm: http://gerrit.ovirt.org/#/c/9507/ (engine patch is built on this) 
 - or alternative vdsm: http://gerrit.ovirt.org/#/c/9367/ (vdsm guys still need to decide which one do they want)


----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: snmishra at linux.vnet.ibm.com
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 9:56:02 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> On 12/05/2012 09:15 PM, snmishra at linux.vnet.ibm.com wrote:
> >
> > Quoting Doron Fediuck <dfediuck at redhat.com>:
> >
> >> ----- Original Message -----
> >>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
> >>> To: "Doron Fediuck" <dfediuck at redhat.com>
> >>> Cc: "engine-devel" <engine-devel at ovirt.org>
> >>> Sent: Wednesday, December 5, 2012 7:14:46 PM
> >>> Subject: Re: [Engine-devel] host cpu feature
> >>>
> >>>
> >>>
> >>> ----- Original Message -----
> >>> > From: "Doron Fediuck" <dfediuck at redhat.com>
> >>> > To: "Yaniv Kaul" <ykaul at redhat.com>
> >>> > Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel"
> >>> > <engine-devel at ovirt.org>
> >>> > Sent: Wednesday, December 5, 2012 6:10:55 PM
> >>> > Subject: Re: [Engine-devel] host cpu feature
> >>> >
> >>> > > Alternative idea, inspired by "Thus, if you hit any bugs, you
> >>> > > are
> >>> > > on
> >>> > > your own" (http://libvirt.org/formatdomain.html#elementsCPU
> >>> > > wrt
> >>> > > 'host-passthrough'):
> >>> > > A config option to determine if we use host-model or
> >>> > > host-passthrough.
> >>> > > Y.
> >>> > >
> >>> >
> >>> > I do not think the engine should go to this level.
> >>> > ie- it can ask for passthrough as a feature, and the
> >>> > actual implementation is handled by vdsm.
> >>> >
> >>>
> >>> If vdsm decides over host-passthrough or host-model, then how
> >>> will
> >>> the engine user know what exactly his VM gets. I think vdsm must
> >>> be
> >>> told exactly what to do.
> >>>
> >>
> >> VDSM maintains some level of independence. This is why it the
> >> engine
> >> should be able to ask for passthrough as a feature. Otherwize vdsm
> >> will
> >> handle it. So for now I suggest we stick to passthrough only, and
> >> if
> >> we get an RFE for advanced mode we'll support the host model.
> >
> > What are we gaining by using passthrough over host-model? Looking
> > at
> > libvirt documentation, it seems that both modes give host CPU
> > capabilities to guest VM. Whereas the downside of passthrough is
> > that it
> > limits migration. Whereas host-model will migrate to other hardware
> > and
> > if the destination hardware is better than source then the guest VM
> > performance can be improved by rebooting guest.
> >
> > As a stretch goal, ovirt can keep track of host capabilities and
> > inform
> > the user after migrating to a better host, that a reboot may
> > improve
> > guest performance.
> 
> pass-through may give better performance.
> host-model would be relevant when we can support live migration
> inside
> the cluster for some of the nodes, which will be relevant when the
> scheduler is more pluggable/extendable than today.
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From acathrow at redhat.com  Fri Dec  7 12:21:03 2012
From: acathrow at redhat.com (Andrew Cathrow)
Date: Fri, 7 Dec 2012 07:21:03 -0500 (EST)
Subject: [Engine-devel] host cpu feature
In-Reply-To: <50BFB4E2.7060500@redhat.com>
Message-ID: <2023727127.42619962.1354882863562.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: snmishra at linux.vnet.ibm.com
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 5, 2012 3:56:02 PM
> Subject: Re: [Engine-devel] host cpu feature
> 
> On 12/05/2012 09:15 PM, snmishra at linux.vnet.ibm.com wrote:
> >
> > Quoting Doron Fediuck <dfediuck at redhat.com>:
> >
> >> ----- Original Message -----
> >>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
> >>> To: "Doron Fediuck" <dfediuck at redhat.com>
> >>> Cc: "engine-devel" <engine-devel at ovirt.org>
> >>> Sent: Wednesday, December 5, 2012 7:14:46 PM
> >>> Subject: Re: [Engine-devel] host cpu feature
> >>>
> >>>
> >>>
> >>> ----- Original Message -----
> >>> > From: "Doron Fediuck" <dfediuck at redhat.com>
> >>> > To: "Yaniv Kaul" <ykaul at redhat.com>
> >>> > Cc: "Laszlo Hornyak" <lhornyak at redhat.com>, "engine-devel"
> >>> > <engine-devel at ovirt.org>
> >>> > Sent: Wednesday, December 5, 2012 6:10:55 PM
> >>> > Subject: Re: [Engine-devel] host cpu feature
> >>> >
> >>> > > Alternative idea, inspired by "Thus, if you hit any bugs, you
> >>> > > are
> >>> > > on
> >>> > > your own" (http://libvirt.org/formatdomain.html#elementsCPU
> >>> > > wrt
> >>> > > 'host-passthrough'):
> >>> > > A config option to determine if we use host-model or
> >>> > > host-passthrough.
> >>> > > Y.
> >>> > >
> >>> >
> >>> > I do not think the engine should go to this level.
> >>> > ie- it can ask for passthrough as a feature, and the
> >>> > actual implementation is handled by vdsm.
> >>> >
> >>>
> >>> If vdsm decides over host-passthrough or host-model, then how
> >>> will
> >>> the engine user know what exactly his VM gets. I think vdsm must
> >>> be
> >>> told exactly what to do.
> >>>
> >>
> >> VDSM maintains some level of independence. This is why it the
> >> engine
> >> should be able to ask for passthrough as a feature. Otherwize vdsm
> >> will
> >> handle it. So for now I suggest we stick to passthrough only, and
> >> if
> >> we get an RFE for advanced mode we'll support the host model.
> >
> > What are we gaining by using passthrough over host-model? Looking
> > at
> > libvirt documentation, it seems that both modes give host CPU
> > capabilities to guest VM. Whereas the downside of passthrough is
> > that it
> > limits migration. Whereas host-model will migrate to other hardware
> > and
> > if the destination hardware is better than source then the guest VM
> > performance can be improved by rebooting guest.
> >
> > As a stretch goal, ovirt can keep track of host capabilities and
> > inform
> > the user after migrating to a better host, that a reboot may
> > improve
> > guest performance.
> 
> pass-through may give better performance.

We need to be using -cpu host aka pass-through for performance. Selecting -cpu host on a Westmere cpu is different to -cpu Westmere on a Westmere cpu in terms of what the guest sees


> host-model would be relevant when we can support live migration
> inside
> the cluster for some of the nodes, which will be relevant when the
> scheduler is more pluggable/extendable than today.
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From smizrahi at redhat.com  Fri Dec  7 19:53:41 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Fri, 7 Dec 2012 14:53:41 -0500 (EST)
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <50C17D43.7010509@linux.vnet.ibm.com>
Message-ID: <2088165976.15522700.1354910021174.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Deepak C Shetty" <deepakcs at linux.vnet.ibm.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "Shu Ming" <shuming at linux.vnet.ibm.com>, "engine-devel" <engine-devel at ovirt.org>, "VDSM Project Development"
> <vdsm-devel at lists.fedorahosted.org>, "Deepak C Shetty" <deepakcs at linux.vnet.ibm.com>
> Sent: Friday, December 7, 2012 12:23:15 AM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> On 12/06/2012 10:22 PM, Saggi Mizrahi wrote:
> >
> > ----- Original Message -----
> >> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
> >> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> >> Cc: "VDSM Project Development"
> >> <vdsm-devel at lists.fedorahosted.org>, "engine-devel"
> >> <engine-devel at ovirt.org>
> >> Sent: Thursday, December 6, 2012 11:02:02 AM
> >> Subject: Re: [vdsm] RFC: New Storage API
> >>
> >> Saggi,
> >>
> >> Thanks for sharing your thought and I get some comments below.
> >>
> >>
> >> Saggi Mizrahi:
> >>> I've been throwing a lot of bits out about the new storage API
> >>> and
> >>> I think it's time to talk a bit.
> >>> I will purposefully try and keep implementation details away and
> >>> concentrate about how the API looks and how you use it.
> >>>
> >>> First major change is in terminology, there is no long a storage
> >>> domain but a storage repository.
> >>> This change is done because so many things are already called
> >>> domain in the system and this will make things less confusing for
> >>> new-commers with a libvirt background.
> >>>
> >>> One other changes is that repositories no longer have a UUID.
> >>> The UUID was only used in the pool members manifest and is no
> >>> longer needed.
> >>>
> >>>
> >>> connectStorageRepository(repoId, repoFormat,
> >>> connectionParameters={}):
> >>> repoId - is a transient name that will be used to refer to the
> >>> connected domain, it is not persisted and doesn't have to be the
> >>> same across the cluster.
> >>> repoFormat - Similar to what used to be type (eg. localfs-1.0,
> >>> nfs-3.4, clvm-1.2).
> >>> connectionParameters - This is format specific and will used to
> >>> tell VDSM how to connect to the repo.
> >>
> >> Where does repoID come from? I think repoID doesn't exist before
> >> connectStorageRepository() return.  Isn't repoID a return value of
> >> connectStorageRepository()?
> > No, repoIDs are no longer part of the domain, they are just a
> > transient handle.
> > The user can put whatever it wants there as long as it isn't
> > already taken by another currently connected domain.
> 
> So what happens when user mistakenly gives a repoID that is in use
> before.. there should be something in the return value that specifies
> the error and/or reason for error so that user can try with a
> new/diff
> repoID ?
Asi I said, connect fails if the repoId is in use ATM.
> 
> >>> disconnectStorageRepository(self, repoId)
> >>>
> >>>
> >>> In the new API there are only images, some images are mutable and
> >>> some are not.
> >>> mutable images are also called VirtualDisks
> >>> immutable images are also called Snapshots
> >>>
> >>> There are no explicit templates, you can create as many images as
> >>> you want from any snapshot.
> >>>
> >>> There are 4 major image operations:
> >>>
> >>>
> >>> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> >>>                     userData={}, options={}):
> >>>
> >>> targetRepoId - ID of a connected repo where the disk will be
> >>> created
> >>> size - The size of the image you wish to create
> >>> baseSnapshotId - the ID of the snapshot you want the base the new
> >>> virtual disk on
> >>> userData - optional data that will be attached to the new VD,
> >>> could
> >>> be anything that the user desires.
> >>> options - options to modify VDSMs default behavior
> 
> IIUC, i can use options to do storage offloads ? For eg. I can create
> a
> LUN that represents this VD on my storage array based on the
> 'options'
> parameter ? Is this the intended way to use 'options' ?
No, this has nothing to do with offloads.
If by "offloads" you mean having other VDSM hosts to the heavy lifting then this is what the option autoFix=False and the fix mechanism is for.
If you are talking about advanced scsi features (ie. write same) they will be used automatically whenever possible.
In any case, how we manage LUNs (if they are even used) is an implementation detail.
> 
> >>>
> >>> returns the id of the new VD
> >> I think we will also need a function to check if a a VirtualDisk
> >> is
> >> based on a specific snapshot.
> >> Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
> > No, the design is that volume dependencies are an implementation
> > detail.
> > There is no reason for you to know that an image is physically a
> > snapshot of another.
> > Logical snapshots, template information, and any other information
> > can be set by the user by using the userData field available for
> > every image.
> >>> createSnapshot(targetRepoId, baseVirtualDiskId,
> >>>                  userData={}, options={}):
> >>> targetRepoId - The ID of a connected repo where the new sanpshot
> >>> will be created and the original image exists as well.
> >>> size - The size of the image you wish to create
> >>> baseVirtualDisk - the ID of a mutable image (Virtual Disk) you
> >>> want
> >>> to snapshot
> >>> userData - optional data that will be attached to the new
> >>> Snapshot,
> >>> could be anything that the user desires.
> >>> options - options to modify VDSMs default behavior
> >>>
> >>> returns the id of the new Snapshot
> >>>
> >>> copyImage(targetRepoId, imageId, baseImageId=None, userData={},
> >>> options={})
> >>> targetRepoId - The ID of a connected repo where the new image
> >>> will
> >>> be created
> >>> imageId - The image you wish to copy
> >>> baseImageId - if specified, the new image will contain only the
> >>> diff between image and Id.
> >>>                 If None the new image will contain all the bits
> >>>                 of
> >>>                 image Id. This can be used to copy partial parts
> >>>                 of
> >>>                 images for export.
> >>> userData - optional data that will be attached to the new image,
> >>> could be anything that the user desires.
> >>> options - options to modify VDSMs default behavior
> >> Does this function mean that we can copy the image from one
> >> repository
> >> to another repository? Does it cover the semantics of storage
> >> migration,
> >> storage backup, storage incremental backup?
> > Yes, the main purpose is copying to another repo. and you can even
> > do incremental backups.
> > Also the following flow
> > 1. Run a VM using imageA
> > 2. write to disk
> > 3. Stop VM
> > 4. copy imageA to repoB
> > 5. Run a VM using imageA again
> > 6. Write to disk
> > 7. Stop VM
> > 8. Copy imageA again basing it of imageA_copy1 on repoB creating a
> > diff on repo diff without snapshotting the original image.
> >
> >>> return the Id of the new image. In case of copying an immutable
> >>> image the ID will be identical to the original image as they
> >>> contain the same data. However the user should not assume that
> >>> and
> >>> always use the value returned from the method.
> >>>
> >>> removeImage(repositoryId, imageId, options={}):
> >>> repositoryId - The ID of a connected repo where the image to
> >>> delete
> >>> resides
> >>> imageId - The id of the image you wish to delete.
> >>>
> >>>
> >>> ----
> >>> getImageStatus(repositoryId, imageId)
> >>> repositoryId - The ID of a connected repo where the image to
> >>> check
> >>> resides
> >>> imageId - The id of the image you wish to check.
> >>>
> >>> All operations return once the operations has been committed to
> >>> disk NOT when the operation actually completes.
> >>> This is done so that:
> >>> - operation come to a stable state as quickly as possible.
> >>> - In case where there is an SDM, only small portion of the
> >>> operation actually needs to be performed on the SDM host.
> >>> - No matter how many times the operation fails and on how many
> >>> hosts, you can always resume the operation and choose when to do
> >>> it.
> >>> - You can stop an operation at any time and remove the resulting
> >>> object making a distinction between "stop because the host is
> >>> overloaded" to "I don't want that image"
> >>>
> >>> This means that after calling any operation that creates a new
> >>> image the user must then call getImageStatus() to check what is
> >>> the status of the image.
> >>> The status of the image can be either optimized, degraded, or
> >>> broken.
> >>> "Optimized" means that the image is available and you can run VMs
> >>> of it.
> >>> "Degraded" means that the image is available and will run VMs but
> >>> it might be a better way VDSM can represent the underlying data.
> 
> Calling qcow2 based snapshot degraded (meaning its degraded in perf,
> as
> its space optimzed ) and calling raw images as optimised ( meaning
> its
> optimised for perf as its space in-efficient) is confusing. Degraded
> sounds like a bad thing when seen by the end-user :) I think there is
> scope for having some better and less confusing terminology here ?
No, that is not what I meant. If the user asked for a space driven image then the qcow2 version is the optimized version.
I chose my words carefully when I originally wrote "a better way VDSM can represent the underlying data".
"better" can mean different things in different circumstances.
How VDSM makes it "better" is an implementation detail. It can even change between VDSM versions.
An extreme case is if we ever decide to make QED the preferred format.
That could potentially suddenly mark all images that are internally represented as qcow2 as degraded because VDSM has found a "better way to represent the data".
All the user has to know is that "degraded images" can be used but they are in a what VDSM considers to be a sub-optimal state.
It's up to the user to decide whether to optimize now, later or never.
> 
> >> What does the "represent" mean here?
> > Anything, but mostly image formate RAW\QCOW2 when performance
> > strategy has been selected.
> >>> "Broken" means that the image can't be used at the moment,
> >>> probably
> >>> because not all the data has been set up on the volume.
> >>>
> >>> Apart from that VDSM will also return the last persisted status
> >>> information which will conatin
> >>> hostID - the last host to try and optimize of fix the image
> >> Any host can optimize the image? No need to be SDM?
> > On anything but lvm based block domains there will not even be an
> > SDM.
> > On SDM based domains we will try as hard as we can to have as many
> > operations executable on any host.
> 
> 1) Can you provide more info on why there is a exception for 'lvm
> based
> block domain'. Its not coming out clearly.
File based domains are responsible for syncing up object manipulation (creation\deletion)
The backend is responsible for making sure it all works either by having a single writer (NFS) or having it's own locking mechanism (gluster).
In our LVM based domains VDSM is responsible for basic object manipulation.
The current design uses an approach where there is a single host responsible for object creation\deleteion it is the SRM\SDM\SPM\S?M.
If we ever find a way to make it fully clustered without a big hit in performance the S?M requirement will be removed form that type of domain.
> 2) Based on the terminology change, domain is now replaced by
> repository, SDM should now be more aptly called SRM (storage repo
> manager) so that we are consistent in the usage of terminology
OK
> 3) Can you provide some example flow / scenario to understnad how
> with
> and without SDM domains work ? Especially how the disk based lock is
> taken if no SDM ?
This is part of the repo API which is out of the scope of this document but in general terms.
getStorageRepositoryContraints() will return the NEED_S?M flag which will notify the manager that it needs to elect an S?M for this repository instance.
All image createVirtualDisk() createSnapshot() and copyImage() and some fixes() will need to be performed on the S?M, removeImage() and some of the fixes could be performed on any host.
As previously noted sending the option autoFix=False will make VDSM not automatically start a fix operation after the operation finished persisting the request and creating the object.
This means you could offload the actual date IO to another host.
Because what fixes can run on what host can change between versions you will just have to try and see if VDSM rejects it on hosts other then the SPM.

There are also plans on abstracting further and making hosts that are not the S?M ask the S?M to create\delete objects for them so you can run any request on any host.
I would very much like to implement that but I am still not sure how efficient inter VDSM communication is to commit to that.
This can be easily implemented in the future by a new flag that marks that even though this repo requires an S?M it doesn't matter where you perform the operation.
> 
> >>> stage - X/Y (eg. 1/10) the last persisted stage of the fix.
> >>> percent_complete - -1 or 0-100, the last persisted completion
> >>> percentage of the aforementioned stage. -1 means that no progress
> >>> is available for that operation.
> >>> last_error - This will only be filled if the operation failed
> >>> because of something other then IO or a VDSM crash for obvious
> >>> reasons.
> >>>                It will usually be set if the task was manually
> >>>                stopped
> >>>
> >>> The user can either be satisfied with that information or as the
> >>> host specified in host ID if it is still working on that image by
> >>> checking it's running tasks.
> >> So we need a function to know what tasks are running on the image
> > getImageStatus()
> >>> checkStorageRepository(self, repositoryId, options={}):
> >>> A method to go over a storage repository and scan for any
> >>> existing
> >>> problems. This includes degraded\broken images and deleted images
> >>> that have no yet been physically deleted\merged.
> >>> It returns a list of Fix objects.
> >>> Fix objects come in 4 types:
> >>> clean - cleans data, run them to get more space.
> >>> optimize - run them to optimize a degraded image
> >>> merge - Merges two images together. Doing this sometimes
> >>>           makes more images ready optimizing or cleaning.
> >>>           The reason it is different from optimize is that
> >>>           unmerged images are considered optimized.
> >>> mend - mends a broken image
> >>>
> >>> The user can read these types and prioritize fixes. Fixes also
> >>> contain opaque FIX data and they should be sent as received to
> >>> fixStorageRepository(self, repositoryId, fix, options={}):
> >>>
> >>> That will start a fix operation.
> 
> It would be good if you can provide some example or flow of "fix"
> operation.
> When and Why would somebody want to do it ?
You would want to do it to get your images to a better state broken->(degraded|optimized) or degraded->optimized.
Getting an image to a functional (degraded|optimized) state is obviously beneficial and getting to an optimized is better by definition.
As an example.
The user called copyImage() that created the all the objects involved and persisted the command and returned.
The snapshot is not yet ready as the actual copy of the data didn't really happen.
Lets say that the method was called with autoFix=False so you now have a broken image (the new image).
You can't use it at the moment. Running a Fix will continue the actual operation moving it to a better state.
VDSM will choose whatever is more likely to get the image to a functional state faster. This means that the image might find itself in a degraded state.
This is good because if the user want to run the VM ASAP it can now do it for the price of performance.
If the user has resource to spare (time\IO\hosts) it can choose to perform another fix and get the image to an optimized state.
This ensures that the images is in the best state according to the users options.
This of course doesn't mean best performance as the user might ask that VDSM value other things over performance.

This system means that you can allocate and deallocate resources at will with minimal consequences.
In some cases advanced users can use this system in conjunction with live merge \ live migration to actually "optimize" while it's running.

This system means that VDSM can change it's policy and behavior between versions without that breaking the interface.
We find a way to skip degraded state for some use case, great.
We found a way to make some use case finish faster but in degraded mode, awesome.
We found a way to make some operations run on all hosts, fantastic.
The user can always handle it because it never assumes the physical result of any operation just the logical.

> 
> Does 'Fix' here mean that i move from raw to qcow2 format or
> vice-versa,
> or there is more to it ?
Fix could literally mean anything, format conversion, compression\decompression, cleanups, merges.
> 
> >>>
> >>>
> >>> All major operations automatically start the appropriate "Fix" to
> >>> bring the created object to an optimize\degraded state (the one
> >>> that is quicker) unless one of the options is
> >>> AutoFix=False. This is only useful for repos that might not be
> >>> able
> >>> to create volumes on all hosts (SDM) but would like to have the
> >>> actual IO distributed in the cluster.
> >>>
> >>> Other common options is the strategy option:
> >>> It has currently 2 possible values
> >>> space and performance - In case VDSM has 2 ways of completing the
> >>> same operation it will tell it to value one over the other. For
> >>> example, whether to copy all the data or just create a qcow based
> >>> of a snapshot.
> >>> The default is space.
> >>>
> >>> You might have also noticed that it is never explicitly specified
> >>> where to look for existing images. This is done purposefully,
> >>> VDSM
> >>> will always look in all connected repositories for existing
> >>> objects.
> >>> For very large setups this might be problematic. To mitigate the
> >>> problem you have these options:
> >>> participatingRepositories=[repoId, ...] which tell VDSM to narrow
> >>> the search to just these repositories
> >>> and
> >>> imageHints={imgId: repoId} which will force VDSM to look for
> >>> those
> >>> image ID just in those repositories and fail if it doesn't find
> >>> them there.
> >>> _______________________________________________
> >>> vdsm-devel mailing list
> >>> vdsm-devel at lists.fedorahosted.org
> >>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> >>
> >> --
> >> ---
> >> ?? Shu Ming
> >> Open Virtualization Engineerning; CSTL, IBM Corp.
> >> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com
> >> or
> >> shuming at linux.vnet.ibm.com
> >> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> >> District, Beijing 100193, PRC
> >>
> >>
> >>
> > _______________________________________________
> > vdsm-devel mailing list
> > vdsm-devel at lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 
>


From smizrahi at redhat.com  Fri Dec  7 20:07:09 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Fri, 7 Dec 2012 15:07:09 -0500 (EST)
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <50C18EA0.6080400@linux.vnet.ibm.com>
Message-ID: <2110662052.15534826.1354910829612.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
> To: "Deepak C Shetty" <deepakcs at linux.vnet.ibm.com>
> Cc: "Saggi Mizrahi" <smizrahi at redhat.com>, "engine-devel" <engine-devel at ovirt.org>, "VDSM Project Development"
> <vdsm-devel at lists.fedorahosted.org>
> Sent: Friday, December 7, 2012 1:37:20 AM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> ? 2012-12-7 13:23, Deepak C Shetty:
> > On 12/06/2012 10:22 PM, Saggi Mizrahi wrote:
> >>
> >> ----- Original Message -----
> >>> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
> >>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> >>> Cc: "VDSM Project Development"
> >>> <vdsm-devel at lists.fedorahosted.org>,
> >>> "engine-devel" <engine-devel at ovirt.org>
> >>> Sent: Thursday, December 6, 2012 11:02:02 AM
> >>> Subject: Re: [vdsm] RFC: New Storage API
> >>>
> >>> Saggi,
> >>>
> >>> Thanks for sharing your thought and I get some comments below.
> >>>
> >>>
> >>> Saggi Mizrahi:
> >>>> I've been throwing a lot of bits out about the new storage API
> >>>> and
> >>>> I think it's time to talk a bit.
> >>>> I will purposefully try and keep implementation details away and
> >>>> concentrate about how the API looks and how you use it.
> >>>>
> >>>> First major change is in terminology, there is no long a storage
> >>>> domain but a storage repository.
> >>>> This change is done because so many things are already called
> >>>> domain in the system and this will make things less confusing
> >>>> for
> >>>> new-commers with a libvirt background.
> >>>>
> >>>> One other changes is that repositories no longer have a UUID.
> >>>> The UUID was only used in the pool members manifest and is no
> >>>> longer needed.
> >>>>
> >>>>
> >>>> connectStorageRepository(repoId, repoFormat,
> >>>> connectionParameters={}):
> >>>> repoId - is a transient name that will be used to refer to the
> >>>> connected domain, it is not persisted and doesn't have to be the
> >>>> same across the cluster.
> >>>> repoFormat - Similar to what used to be type (eg. localfs-1.0,
> >>>> nfs-3.4, clvm-1.2).
> >>>> connectionParameters - This is format specific and will used to
> >>>> tell VDSM how to connect to the repo.
> >>>
> >>> Where does repoID come from? I think repoID doesn't exist before
> >>> connectStorageRepository() return.  Isn't repoID a return value
> >>> of
> >>> connectStorageRepository()?
> >> No, repoIDs are no longer part of the domain, they are just a
> >> transient handle.
> >> The user can put whatever it wants there as long as it isn't
> >> already
> >> taken by another currently connected domain.
> >
> > So what happens when user mistakenly gives a repoID that is in use
> > before.. there should be something in the return value that
> > specifies
> > the error and/or reason for error so that user can try with a
> > new/diff
> > repoID ?
> 
> I think let the user to give the repoID is meaningless and
> error-prune.
The repo ID is meaningless, it's just a handle to the instance.
It's never persisted to disk and doesn't have to be unique across the cluster.
> Developer must maintain a a unique ID list for every storage
> repository
> connected.
Why? You could just use repoId = <progname>_<hostname>_<data_center>_<domain_number>
"ovirt_example.com_hosting_3" as an example
If you agree with all other users of the same VDSM instance that you are going to use this scheme you can cooperate.
You can use whatever scheme you want to make sure you don't hit anyone else's.
The point is, VDSM doesn't care how you provision repoIDs and how unique they are across the cluster.
It's the user's choice how and if to persist this information.
> 
> >
> >>>> disconnectStorageRepository(self, repoId)
> >>>>
> >>>>
> >>>> In the new API there are only images, some images are mutable
> >>>> and
> >>>> some are not.
> >>>> mutable images are also called VirtualDisks
> >>>> immutable images are also called Snapshots
> >>>>
> >>>> There are no explicit templates, you can create as many images
> >>>> as
> >>>> you want from any snapshot.
> >>>>
> >>>> There are 4 major image operations:
> >>>>
> >>>>
> >>>> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> >>>>                     userData={}, options={}):
> >>>>
> >>>> targetRepoId - ID of a connected repo where the disk will be
> >>>> created
> >>>> size - The size of the image you wish to create
> >>>> baseSnapshotId - the ID of the snapshot you want the base the
> >>>> new
> >>>> virtual disk on
> >>>> userData - optional data that will be attached to the new VD,
> >>>> could
> >>>> be anything that the user desires.
> >>>> options - options to modify VDSMs default behavior
> >
> > IIUC, i can use options to do storage offloads ? For eg. I can
> > create
> > a LUN that represents this VD on my storage array based on the
> > 'options' parameter ? Is this the intended way to use 'options' ?
> >
> >>>>
> >>>> returns the id of the new VD
> >>> I think we will also need a function to check if a a VirtualDisk
> >>> is
> >>> based on a specific snapshot.
> >>> Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
> >> No, the design is that volume dependencies are an implementation
> >> detail.
> >> There is no reason for you to know that an image is physically a
> >> snapshot of another.
> >> Logical snapshots, template information, and any other information
> >> can be set by the user by using the userData field available for
> >> every image.
> >>>> createSnapshot(targetRepoId, baseVirtualDiskId,
> >>>>                  userData={}, options={}):
> >>>> targetRepoId - The ID of a connected repo where the new sanpshot
> >>>> will be created and the original image exists as well.
> >>>> size - The size of the image you wish to create
> >>>> baseVirtualDisk - the ID of a mutable image (Virtual Disk) you
> >>>> want
> >>>> to snapshot
> >>>> userData - optional data that will be attached to the new
> >>>> Snapshot,
> >>>> could be anything that the user desires.
> >>>> options - options to modify VDSMs default behavior
> >>>>
> >>>> returns the id of the new Snapshot
> >>>>
> >>>> copyImage(targetRepoId, imageId, baseImageId=None, userData={},
> >>>> options={})
> >>>> targetRepoId - The ID of a connected repo where the new image
> >>>> will
> >>>> be created
> >>>> imageId - The image you wish to copy
> >>>> baseImageId - if specified, the new image will contain only the
> >>>> diff between image and Id.
> >>>>                 If None the new image will contain all the bits
> >>>>                 of
> >>>>                 image Id. This can be used to copy partial parts
> >>>>                 of
> >>>>                 images for export.
> >>>> userData - optional data that will be attached to the new image,
> >>>> could be anything that the user desires.
> >>>> options - options to modify VDSMs default behavior
> >>> Does this function mean that we can copy the image from one
> >>> repository
> >>> to another repository? Does it cover the semantics of storage
> >>> migration,
> >>> storage backup, storage incremental backup?
> >> Yes, the main purpose is copying to another repo. and you can even
> >> do
> >> incremental backups.
> >> Also the following flow
> >> 1. Run a VM using imageA
> >> 2. write to disk
> >> 3. Stop VM
> >> 4. copy imageA to repoB
> >> 5. Run a VM using imageA again
> >> 6. Write to disk
> >> 7. Stop VM
> >> 8. Copy imageA again basing it of imageA_copy1 on repoB creating a
> >> diff on repo diff without snapshotting the original image.
> >>>> return the Id of the new image. In case of copying an immutable
> >>>> image the ID will be identical to the original image as they
> >>>> contain the same data. However the user should not assume that
> >>>> and
> >>>> always use the value returned from the method.
> Do you mean the ID returned by copyImage() will be the same as the
> origin ID?  It is okay for copying to a different repository than the
> original repository because they have different namespace. However,
> if
> we do the copy inside in the same repository, we should get a new
> different ID.
As I said, copying immutable objects in the same repo doesn't work.
> 
> 
> >>>>
> >>>> removeImage(repositoryId, imageId, options={}):
> >>>> repositoryId - The ID of a connected repo where the image to
> >>>> delete
> >>>> resides
> >>>> imageId - The id of the image you wish to delete.
> >>>>
> >>>>
> >>>> ----
> >>>> getImageStatus(repositoryId, imageId)
> >>>> repositoryId - The ID of a connected repo where the image to
> >>>> check
> >>>> resides
> >>>> imageId - The id of the image you wish to check.
> >>>>
> >>>> All operations return once the operations has been committed to
> >>>> disk NOT when the operation actually completes.
> >>>> This is done so that:
> >>>> - operation come to a stable state as quickly as possible.
> >>>> - In case where there is an SDM, only small portion of the
> >>>> operation actually needs to be performed on the SDM host.
> >>>> - No matter how many times the operation fails and on how many
> >>>> hosts, you can always resume the operation and choose when to do
> >>>> it.
> >>>> - You can stop an operation at any time and remove the resulting
> >>>> object making a distinction between "stop because the host is
> >>>> overloaded" to "I don't want that image"
> >>>>
> >>>> This means that after calling any operation that creates a new
> >>>> image the user must then call getImageStatus() to check what is
> >>>> the status of the image.
> >>>> The status of the image can be either optimized, degraded, or
> >>>> broken.
> >>>> "Optimized" means that the image is available and you can run
> >>>> VMs
> >>>> of it.
> >>>> "Degraded" means that the image is available and will run VMs
> >>>> but
> >>>> it might be a better way VDSM can represent the underlying data.
> >
> > Calling qcow2 based snapshot degraded (meaning its degraded in
> > perf,
> > as its space optimzed ) and calling raw images as optimised (
> > meaning
> > its optimised for perf as its space in-efficient) is confusing.
> > Degraded sounds like a bad thing when seen by the end-user :) I
> > think
> > there is scope for having some better and less confusing
> > terminology
> > here ?
> >
> >>> What does the "represent" mean here?
> >> Anything, but mostly image formate RAW\QCOW2 when performance
> >> strategy has been selected.
> >>>> "Broken" means that the image can't be used at the moment,
> >>>> probably
> >>>> because not all the data has been set up on the volume.
> >>>>
> >>>> Apart from that VDSM will also return the last persisted status
> >>>> information which will conatin
> >>>> hostID - the last host to try and optimize of fix the image
> >>> Any host can optimize the image? No need to be SDM?
> >> On anything but lvm based block domains there will not even be an
> >> SDM.
> >> On SDM based domains we will try as hard as we can to have as many
> >> operations executable on any host.
> >
> > 1) Can you provide more info on why there is a exception for 'lvm
> > based block domain'. Its not coming out clearly.
> 
> I guess Saggi means we can leverage CLVM to build a cluster based LVM
> group and no need to have SDM.  If I understand correctly, in this
> proposal,  we manage different types of the repositories at the same
> type, say, one localFS repository, one iSCSI repository, one nfs
> repository at the same time, also the images can be copied to each
> other
> back and forth.  When you copy a image from repository without SDM to
> a
> repository with SDM, we should use the later SDM to execute the copy
> operation.
> 
> > 2) Based on the terminology change, domain is now replaced by
> > repository, SDM should now be more aptly called SRM (storage repo
> > manager) so that we are consistent in the usage of terminology
> > 3) Can you provide some example flow / scenario to understnad how
> > with
> > and without SDM domains work ? Especially how the disk based lock
> > is
> > taken if no SDM ?
> >
> >>>> stage - X/Y (eg. 1/10) the last persisted stage of the fix.
> >>>> percent_complete - -1 or 0-100, the last persisted completion
> >>>> percentage of the aforementioned stage. -1 means that no
> >>>> progress
> >>>> is available for that operation.
> >>>> last_error - This will only be filled if the operation failed
> >>>> because of something other then IO or a VDSM crash for obvious
> >>>> reasons.
> >>>>                It will usually be set if the task was manually
> >>>>                stopped
> >>>>
> >>>> The user can either be satisfied with that information or as the
> >>>> host specified in host ID if it is still working on that image
> >>>> by
> >>>> checking it's running tasks.
> >>> So we need a function to know what tasks are running on the image
> >> getImageStatus()
> >>>> checkStorageRepository(self, repositoryId, options={}):
> >>>> A method to go over a storage repository and scan for any
> >>>> existing
> >>>> problems. This includes degraded\broken images and deleted
> >>>> images
> >>>> that have no yet been physically deleted\merged.
> >>>> It returns a list of Fix objects.
> >>>> Fix objects come in 4 types:
> >>>> clean - cleans data, run them to get more space.
> >>>> optimize - run them to optimize a degraded image
> >>>> merge - Merges two images together. Doing this sometimes
> >>>>           makes more images ready optimizing or cleaning.
> >>>>           The reason it is different from optimize is that
> >>>>           unmerged images are considered optimized.
> >>>> mend - mends a broken image
> 
> What is a broken image? Does broken mean corruption?  If it is
> corruption, we may need some recoverable checksum for each image.
> 
> >>>>
> >>>> The user can read these types and prioritize fixes. Fixes also
> >>>> contain opaque FIX data and they should be sent as received to
> >>>> fixStorageRepository(self, repositoryId, fix, options={}):
> >>>>
> >>>> That will start a fix operation.
> >
> > It would be good if you can provide some example or flow of "fix"
> > operation.
> > When and Why would somebody want to do it ?
> >
> > Does 'Fix' here mean that i move from raw to qcow2 format or
> > vice-versa, or there is more to it ?
> >
> >>>>
> >>>>
> >>>> All major operations automatically start the appropriate "Fix"
> >>>> to
> >>>> bring the created object to an optimize\degraded state (the one
> >>>> that is quicker) unless one of the options is
> >>>> AutoFix=False. This is only useful for repos that might not be
> >>>> able
> >>>> to create volumes on all hosts (SDM) but would like to have the
> >>>> actual IO distributed in the cluster.
> >>>>
> >>>> Other common options is the strategy option:
> >>>> It has currently 2 possible values
> >>>> space and performance - In case VDSM has 2 ways of completing
> >>>> the
> >>>> same operation it will tell it to value one over the other. For
> >>>> example, whether to copy all the data or just create a qcow
> >>>> based
> >>>> of a snapshot.
> >>>> The default is space.
> >>>>
> >>>> You might have also noticed that it is never explicitly
> >>>> specified
> >>>> where to look for existing images. This is done purposefully,
> >>>> VDSM
> >>>> will always look in all connected repositories for existing
> >>>> objects.
> >>>> For very large setups this might be problematic. To mitigate the
> >>>> problem you have these options:
> >>>> participatingRepositories=[repoId, ...] which tell VDSM to
> >>>> narrow
> >>>> the search to just these repositories
> >>>> and
> >>>> imageHints={imgId: repoId} which will force VDSM to look for
> >>>> those
> >>>> image ID just in those repositories and fail if it doesn't find
> >>>> them there.
> >>>> _______________________________________________
> >>>> vdsm-devel mailing list
> >>>> vdsm-devel at lists.fedorahosted.org
> >>>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> >>>
> >>> --
> >>> ---
> >>> ?? Shu Ming
> >>> Open Virtualization Engineerning; CSTL, IBM Corp.
> >>> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com
> >>> or
> >>> shuming at linux.vnet.ibm.com
> >>> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> >>> District, Beijing 100193, PRC
> >>>
> >>>
> >>>
> >> _______________________________________________
> >> vdsm-devel mailing list
> >> vdsm-devel at lists.fedorahosted.org
> >> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> >
> 
> 
> --
> ---
> ?? Shu Ming
> Open Virtualization Engineerning; CSTL, IBM Corp.
> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com or
> shuming at linux.vnet.ibm.com
> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> District, Beijing 100193, PRC
> 
> 
> 


From masayag at redhat.com  Mon Dec 10 11:43:09 2012
From: masayag at redhat.com (Moti Asayag)
Date: Mon, 10 Dec 2012 13:43:09 +0200
Subject: [Engine-devel] Report vNic Information - Round 2
Message-ID: <50C5CACD.8000300@redhat.com>

Hi,

According to the latest discussion of vNic info feature, I've updated
its feature page [1].

Note that both feature page and detailed design page are updated.
The feature page contains also a suggested mockup for the vm interface
sub-tab.

[1] http://www.ovirt.org/Feature/ReportingVnicInformation

Thanks,
Moti


From fkobzik at redhat.com  Mon Dec 10 12:28:50 2012
From: fkobzik at redhat.com (Frantisek Kobzik)
Date: Mon, 10 Dec 2012 07:28:50 -0500 (EST)
Subject: [Engine-devel] Attaching a console should be controllable
In-Reply-To: <1089043911.23484370.1355141561452.JavaMail.root@redhat.com>
Message-ID: <15155534.23582417.1355142530672.JavaMail.root@redhat.com>

Hi guys,

Just a simple question. I'm writing an engine counterpart for this vdsm feature:
https://bugzilla.redhat.com/show_bug.cgi?id=878459
http://gerrit.ovirt.org/#/c/9359

Long story short, it adds the possibility of controlling adding the virtio serial console to a VM.

I'm going to add a new column "console_enabled" to vm_static and a new field "bool consoleEnabled" to appropriate entities. I'm just worried about the naming of the column and the field - could I use these names? Isn't it confusing? I just want to avoid unneeded refactoring when the patch is done. 

Any feedback is welcome!

Thanks,
Frank


From masayag at redhat.com  Mon Dec 10 14:06:21 2012
From: masayag at redhat.com (Moti Asayag)
Date: Mon, 10 Dec 2012 16:06:21 +0200
Subject: [Engine-devel] Creating New Data-Center and Permission on Network
Message-ID: <50C5EC5D.50503@redhat.com>

Hi,

When creating a new data-center, a management network is created for it.
By default the created management network is defined as VM network.

I'd like to consult from permissions perspective, what is the preferred
permission settings for that network.

The network is defined as management network, therefore it is designed
be used by VMs. However, the admin should grant permissions on that
networks to the target users (which one might find tedious).

We can grant permission on that network to 'everyone' with role
'NetworkUser', but in case the admin doesn't meant this network to be
used, the permission should be removed.

In 'Add Logical Network' dialog I've added a new checkbox to allow
granting 'everyone' a role for using that network ('NetworkUser').
We can embrace same method in 'Add Data-Center' dialog.

Thoughts ?

Thanks,
Moti


From rnori at redhat.com  Mon Dec 10 15:05:33 2012
From: rnori at redhat.com (Ravi Nori)
Date: Mon, 10 Dec 2012 17:05:33 +0200
Subject: [Engine-devel] Inherited Permissions from System
Message-ID: <50C5FA3D.70004@redhat.com>

I am currently working on a bug to show the inherited permissions on an 
object.

So if the user has permissions assigned on a Cluster, the user should 
have permissions on the VMs in the cluster.

My question is if the user has permissions on the System, should the 
permissions be inherited to all the entities?

Link to the bug

https://bugzilla.redhat.com/show_bug.cgi?id=871365

Thanks

Ravi


From vszocs at redhat.com  Mon Dec 10 18:10:09 2012
From: vszocs at redhat.com (Vojtech Szocs)
Date: Mon, 10 Dec 2012 13:10:09 -0500 (EST)
Subject: [Engine-devel] UI Plugins: PoC patch revision 7 update 3 is now
	available
In-Reply-To: <719755359.16651648.1355161522121.JavaMail.root@redhat.com>
Message-ID: <940461717.16659811.1355163009146.JavaMail.root@redhat.com>

Hi everyone, 

just a quick update on UI Plugins proof-of-c on cept patch - new patch for revision 7 (update 3) is now available. 

You can download and apply latest patches from oVirt Gerrit code review system: 

    * rev is ion 6 - http://gerrit.ovirt.org/#/c/8120/ 
    * rev is ion 7 update 3 - http://gerrit.ovirt.org/#/c/9250/ 
    * REST API session patch: http://gerrit.ovirt.org/#/c/9820/ 


Please read on to learn what's new. 




Engine REST API integration 


REST API session patch (see above) now ensures that new session is acquired each time the user logs into WebAdmin, despite the fact that UI plugin infrastructure doesn't close existing REST API session on server after user logout. This means each user login equals new REST API session, based on provided login credentials. This fixes the problem of reusing existing (un-closed) session from previous user login. Many thanks to Michael Pasternak and Oved Ourfalli for the patch! 



Additionally, known limitation with regard to auto login has been fixed - UI plugin infrastructure now remembers REST API session ID on client (using HTML5 storage / cookie fallback) and uses it for auto login process. UI plugin infrastructure assumes that for auto login, REST API session has already been acquired (via classic login) and is still active at the time of auto login. 





Updated API function: register 

To make it clear that api.register function should be called exactly once , the function behavior has been changed in the following way: 

    * previously, you could call api.register multiple times, last call would "win" with regard to passed event handler object 
    * now, you can call api.register only once, subsequent calls will have no effect (the event handler object has already been set ) 





Regards, 
Vojtech 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121210/18138f99/attachment.html>

From rgolan at redhat.com  Mon Dec 10 18:31:55 2012
From: rgolan at redhat.com (Roy Golan)
Date: Mon, 10 Dec 2012 20:31:55 +0200
Subject: [Engine-devel] Attaching a console should be controllable
In-Reply-To: <15155534.23582417.1355142530672.JavaMail.root@redhat.com>
References: <15155534.23582417.1355142530672.JavaMail.root@redhat.com>
Message-ID: <50C62A9B.6020208@redhat.com>

On 12/10/2012 02:28 PM, Frantisek Kobzik wrote:
> Hi guys,
>
> Just a simple question. I'm writing an engine counterpart for this vdsm feature:
> https://bugzilla.redhat.com/show_bug.cgi?id=878459
> http://gerrit.ovirt.org/#/c/9359
>
> Long story short, it adds the possibility of controlling adding the virtio serial console to a VM.
>
> I'm going to add a new column "console_enabled" to vm_static and a new field "bool consoleEnabled" to appropriate entities. I'm just worried about the naming of the column and the field - could I use these names? Isn't it confusing? I just want to avoid unneeded refactoring when the patch is done.
both fits the convention - db columns underscored and java field 
camelCased.
> Any feedback is welcome!
>
> Thanks,
> Frank
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel



From agl at us.ibm.com  Mon Dec 10 18:39:51 2012
From: agl at us.ibm.com (Adam Litke)
Date: Mon, 10 Dec 2012 12:39:51 -0600
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <1904075471.14922627.1354812721787.JavaMail.root@redhat.com>
References: <50C0C17A.9070000@linux.vnet.ibm.com>
	<1904075471.14922627.1354812721787.JavaMail.root@redhat.com>
Message-ID: <20121210183951.GE10131@aglitke>

On Thu, Dec 06, 2012 at 11:52:01AM -0500, Saggi Mizrahi wrote:
> 
> 
> ----- Original Message -----
> > From: "Shu Ming" <shuming at linux.vnet.ibm.com>
> > To: "Saggi Mizrahi" <smizrahi at redhat.com>
> > Cc: "VDSM Project Development" <vdsm-devel at lists.fedorahosted.org>, "engine-devel" <engine-devel at ovirt.org>
> > Sent: Thursday, December 6, 2012 11:02:02 AM
> > Subject: Re: [vdsm] RFC: New Storage API
> > 
> > Saggi,
> > 
> > Thanks for sharing your thought and I get some comments below.
> > 
> > 
> > Saggi Mizrahi:
> > > I've been throwing a lot of bits out about the new storage API and
> > > I think it's time to talk a bit.
> > > I will purposefully try and keep implementation details away and
> > > concentrate about how the API looks and how you use it.
> > >
> > > First major change is in terminology, there is no long a storage
> > > domain but a storage repository.
> > > This change is done because so many things are already called
> > > domain in the system and this will make things less confusing for
> > > new-commers with a libvirt background.
> > >
> > > One other changes is that repositories no longer have a UUID.
> > > The UUID was only used in the pool members manifest and is no
> > > longer needed.
> > >
> > >
> > > connectStorageRepository(repoId, repoFormat,
> > > connectionParameters={}):
> > > repoId - is a transient name that will be used to refer to the
> > > connected domain, it is not persisted and doesn't have to be the
> > > same across the cluster.
> > > repoFormat - Similar to what used to be type (eg. localfs-1.0,
> > > nfs-3.4, clvm-1.2).
> > > connectionParameters - This is format specific and will used to
> > > tell VDSM how to connect to the repo.
> > 
> > 
> > Where does repoID come from? I think repoID doesn't exist before
> > connectStorageRepository() return.  Isn't repoID a return value of
> > connectStorageRepository()?
> No, repoIDs are no longer part of the domain, they are just a transient handle.
> The user can put whatever it wants there as long as it isn't already taken by another currently connected domain.
> > 
> > >
> > > disconnectStorageRepository(self, repoId)
> > >
> > >
> > > In the new API there are only images, some images are mutable and
> > > some are not.
> > > mutable images are also called VirtualDisks
> > > immutable images are also called Snapshots
> > >
> > > There are no explicit templates, you can create as many images as
> > > you want from any snapshot.
> > >
> > > There are 4 major image operations:
> > >
> > >
> > > createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> > >                    userData={}, options={}):
> > >
> > > targetRepoId - ID of a connected repo where the disk will be
> > > created
> > > size - The size of the image you wish to create
> > > baseSnapshotId - the ID of the snapshot you want the base the new
> > > virtual disk on
> > > userData - optional data that will be attached to the new VD, could
> > > be anything that the user desires.
> > > options - options to modify VDSMs default behavior
> > >
> > > returns the id of the new VD
> > 
> > I think we will also need a function to check if a a VirtualDisk is
> > based on a specific snapshot.
> > Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
> No, the design is that volume dependencies are an implementation detail.
> There is no reason for you to know that an image is physically a snapshot of another.
> Logical snapshots, template information, and any other information can be set by the user by using the userData field available for every image.

Statements like this make me start to worry about your userData concept.  It's a
sign of a bad API if the user needs to invent a custom metadata scheme for
itself.  This reminds me of the abomination that is the 'custom' property in the
vm definition today.

> > > createSnapshot(targetRepoId, baseVirtualDiskId,
> > >                 userData={}, options={}):
> > > targetRepoId - The ID of a connected repo where the new sanpshot
> > > will be created and the original image exists as well.
> > > size - The size of the image you wish to create
> > > baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want
> > > to snapshot
> > > userData - optional data that will be attached to the new Snapshot,
> > > could be anything that the user desires.
> > > options - options to modify VDSMs default behavior
> > >
> > > returns the id of the new Snapshot
> > >
> > > copyImage(targetRepoId, imageId, baseImageId=None, userData={},
> > > options={})
> > > targetRepoId - The ID of a connected repo where the new image will
> > > be created
> > > imageId - The image you wish to copy
> > > baseImageId - if specified, the new image will contain only the
> > > diff between image and Id.
> > >                If None the new image will contain all the bits of
> > >                image Id. This can be used to copy partial parts of
> > >                images for export.
> > > userData - optional data that will be attached to the new image,
> > > could be anything that the user desires.
> > > options - options to modify VDSMs default behavior
> > 
> > Does this function mean that we can copy the image from one
> > repository
> > to another repository? Does it cover the semantics of storage
> > migration,
> > storage backup, storage incremental backup?
> Yes, the main purpose is copying to another repo. and you can even do incremental backups.
> Also the following flow
> 1. Run a VM using imageA
> 2. write to disk
> 3. Stop VM
> 4. copy imageA to repoB
> 5. Run a VM using imageA again
> 6. Write to disk
> 7. Stop VM
> 8. Copy imageA again basing it of imageA_copy1 on repoB creating a diff on repo diff without snapshotting the original image.
> 
> > 
> > >
> > > return the Id of the new image. In case of copying an immutable
> > > image the ID will be identical to the original image as they
> > > contain the same data. However the user should not assume that and
> > > always use the value returned from the method.
> > >
> > > removeImage(repositoryId, imageId, options={}):
> > > repositoryId - The ID of a connected repo where the image to delete
> > > resides
> > > imageId - The id of the image you wish to delete.
> > >
> > >
> > > ----
> > > getImageStatus(repositoryId, imageId)
> > > repositoryId - The ID of a connected repo where the image to check
> > > resides
> > > imageId - The id of the image you wish to check.
> > >
> > > All operations return once the operations has been committed to
> > > disk NOT when the operation actually completes.
> > > This is done so that:
> > > - operation come to a stable state as quickly as possible.
> > > - In case where there is an SDM, only small portion of the
> > > operation actually needs to be performed on the SDM host.
> > > - No matter how many times the operation fails and on how many
> > > hosts, you can always resume the operation and choose when to do
> > > it.
> > > - You can stop an operation at any time and remove the resulting
> > > object making a distinction between "stop because the host is
> > > overloaded" to "I don't want that image"
> > >
> > > This means that after calling any operation that creates a new
> > > image the user must then call getImageStatus() to check what is
> > > the status of the image.
> > > The status of the image can be either optimized, degraded, or
> > > broken.
> > > "Optimized" means that the image is available and you can run VMs
> > > of it.
> > > "Degraded" means that the image is available and will run VMs but
> > > it might be a better way VDSM can represent the underlying data.
> > 
> > What does the "represent" mean here?
> Anything, but mostly image formate RAW\QCOW2 when performance strategy has been selected.
> > > "Broken" means that the image can't be used at the moment, probably
> > > because not all the data has been set up on the volume.
> > >
> > > Apart from that VDSM will also return the last persisted status
> > > information which will conatin
> > > hostID - the last host to try and optimize of fix the image
> > Any host can optimize the image? No need to be SDM?
> On anything but lvm based block domains there will not even be an SDM.
> On SDM based domains we will try as hard as we can to have as many operations executable on any host.
> > 
> > > stage - X/Y (eg. 1/10) the last persisted stage of the fix.
> > > percent_complete - -1 or 0-100, the last persisted completion
> > > percentage of the aforementioned stage. -1 means that no progress
> > > is available for that operation.
> > > last_error - This will only be filled if the operation failed
> > > because of something other then IO or a VDSM crash for obvious
> > > reasons.
> > >               It will usually be set if the task was manually
> > >               stopped
> > >
> > > The user can either be satisfied with that information or as the
> > > host specified in host ID if it is still working on that image by
> > > checking it's running tasks.
> > 
> > So we need a function to know what tasks are running on the image
> getImageStatus()
> > >
> > > checkStorageRepository(self, repositoryId, options={}):
> > > A method to go over a storage repository and scan for any existing
> > > problems. This includes degraded\broken images and deleted images
> > > that have no yet been physically deleted\merged.
> > > It returns a list of Fix objects.
> > > Fix objects come in 4 types:
> > > clean - cleans data, run them to get more space.
> > > optimize - run them to optimize a degraded image
> > > merge - Merges two images together. Doing this sometimes
> > >          makes more images ready optimizing or cleaning.
> > >          The reason it is different from optimize is that
> > >          unmerged images are considered optimized.
> > > mend - mends a broken image
> > >
> > > The user can read these types and prioritize fixes. Fixes also
> > > contain opaque FIX data and they should be sent as received to
> > > fixStorageRepository(self, repositoryId, fix, options={}):
> > >
> > > That will start a fix operation.
> > >
> > >
> > > All major operations automatically start the appropriate "Fix" to
> > > bring the created object to an optimize\degraded state (the one
> > > that is quicker) unless one of the options is
> > > AutoFix=False. This is only useful for repos that might not be able
> > > to create volumes on all hosts (SDM) but would like to have the
> > > actual IO distributed in the cluster.
> > >
> > > Other common options is the strategy option:
> > > It has currently 2 possible values
> > > space and performance - In case VDSM has 2 ways of completing the
> > > same operation it will tell it to value one over the other. For
> > > example, whether to copy all the data or just create a qcow based
> > > of a snapshot.
> > > The default is space.
> > >
> > > You might have also noticed that it is never explicitly specified
> > > where to look for existing images. This is done purposefully, VDSM
> > > will always look in all connected repositories for existing
> > > objects.
> > > For very large setups this might be problematic. To mitigate the
> > > problem you have these options:
> > > participatingRepositories=[repoId, ...] which tell VDSM to narrow
> > > the search to just these repositories
> > > and
> > > imageHints={imgId: repoId} which will force VDSM to look for those
> > > image ID just in those repositories and fail if it doesn't find
> > > them there.
> > > _______________________________________________
> > > vdsm-devel mailing list
> > > vdsm-devel at lists.fedorahosted.org
> > > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> > 
> > 
> > --
> > ---
> > ?? Shu Ming
> > Open Virtualization Engineerning; CSTL, IBM Corp.
> > Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com or
> > shuming at linux.vnet.ibm.com
> > Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> > District, Beijing 100193, PRC
> > 
> > 
> > 
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel

-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center



From agl at us.ibm.com  Mon Dec 10 18:49:31 2012
From: agl at us.ibm.com (Adam Litke)
Date: Mon, 10 Dec 2012 12:49:31 -0600
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <2088165976.15522700.1354910021174.JavaMail.root@redhat.com>
References: <50C17D43.7010509@linux.vnet.ibm.com>
	<2088165976.15522700.1354910021174.JavaMail.root@redhat.com>
Message-ID: <20121210184931.GF10131@aglitke>

On Fri, Dec 07, 2012 at 02:53:41PM -0500, Saggi Mizrahi wrote:

<snip>

> > 1) Can you provide more info on why there is a exception for 'lvm
> > based
> > block domain'. Its not coming out clearly.
> File based domains are responsible for syncing up object manipulation (creation\deletion)
> The backend is responsible for making sure it all works either by having a single writer (NFS) or having it's own locking mechanism (gluster).
> In our LVM based domains VDSM is responsible for basic object manipulation.
> The current design uses an approach where there is a single host responsible for object creation\deleteion it is the SRM\SDM\SPM\S?M.
> If we ever find a way to make it fully clustered without a big hit in performance the S?M requirement will be removed form that type of domain.

I would like to see us maintain a LOCALFS domain as well.  For this, we would
also need SRM, correct?

-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center



From smizrahi at redhat.com  Mon Dec 10 19:03:09 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 10 Dec 2012 14:03:09 -0500 (EST)
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <20121210184931.GF10131@aglitke>
Message-ID: <2033079376.16686610.1355166189638.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Adam Litke" <agl at us.ibm.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "Deepak C Shetty" <deepakcs at linux.vnet.ibm.com>, "engine-devel" <engine-devel at ovirt.org>, "VDSM Project
> Development" <vdsm-devel at lists.fedorahosted.org>
> Sent: Monday, December 10, 2012 1:49:31 PM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> On Fri, Dec 07, 2012 at 02:53:41PM -0500, Saggi Mizrahi wrote:
> 
> <snip>
> 
> > > 1) Can you provide more info on why there is a exception for 'lvm
> > > based
> > > block domain'. Its not coming out clearly.
> > File based domains are responsible for syncing up object
> > manipulation (creation\deletion)
> > The backend is responsible for making sure it all works either by
> > having a single writer (NFS) or having it's own locking mechanism
> > (gluster).
> > In our LVM based domains VDSM is responsible for basic object
> > manipulation.
> > The current design uses an approach where there is a single host
> > responsible for object creation\deleteion it is the
> > SRM\SDM\SPM\S?M.
> > If we ever find a way to make it fully clustered without a big hit
> > in performance the S?M requirement will be removed form that type
> > of domain.
> 
> I would like to see us maintain a LOCALFS domain as well.  For this,
> we would
> also need SRM, correct?
No, why?
> 
> --
> Adam Litke <agl at us.ibm.com>
> IBM Linux Technology Center
> 
> 


From smizrahi at redhat.com  Mon Dec 10 20:36:23 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 10 Dec 2012 15:36:23 -0500 (EST)
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <20121210183951.GE10131@aglitke>
Message-ID: <1874293712.16743050.1355171783487.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Adam Litke" <agl at us.ibm.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "Shu Ming" <shuming at linux.vnet.ibm.com>, "engine-devel" <engine-devel at ovirt.org>, "VDSM Project Development"
> <vdsm-devel at lists.fedorahosted.org>
> Sent: Monday, December 10, 2012 1:39:51 PM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> On Thu, Dec 06, 2012 at 11:52:01AM -0500, Saggi Mizrahi wrote:
> > 
> > 
> > ----- Original Message -----
> > > From: "Shu Ming" <shuming at linux.vnet.ibm.com>
> > > To: "Saggi Mizrahi" <smizrahi at redhat.com>
> > > Cc: "VDSM Project Development"
> > > <vdsm-devel at lists.fedorahosted.org>, "engine-devel"
> > > <engine-devel at ovirt.org>
> > > Sent: Thursday, December 6, 2012 11:02:02 AM
> > > Subject: Re: [vdsm] RFC: New Storage API
> > > 
> > > Saggi,
> > > 
> > > Thanks for sharing your thought and I get some comments below.
> > > 
> > > 
> > > Saggi Mizrahi:
> > > > I've been throwing a lot of bits out about the new storage API
> > > > and
> > > > I think it's time to talk a bit.
> > > > I will purposefully try and keep implementation details away
> > > > and
> > > > concentrate about how the API looks and how you use it.
> > > >
> > > > First major change is in terminology, there is no long a
> > > > storage
> > > > domain but a storage repository.
> > > > This change is done because so many things are already called
> > > > domain in the system and this will make things less confusing
> > > > for
> > > > new-commers with a libvirt background.
> > > >
> > > > One other changes is that repositories no longer have a UUID.
> > > > The UUID was only used in the pool members manifest and is no
> > > > longer needed.
> > > >
> > > >
> > > > connectStorageRepository(repoId, repoFormat,
> > > > connectionParameters={}):
> > > > repoId - is a transient name that will be used to refer to the
> > > > connected domain, it is not persisted and doesn't have to be
> > > > the
> > > > same across the cluster.
> > > > repoFormat - Similar to what used to be type (eg. localfs-1.0,
> > > > nfs-3.4, clvm-1.2).
> > > > connectionParameters - This is format specific and will used to
> > > > tell VDSM how to connect to the repo.
> > > 
> > > 
> > > Where does repoID come from? I think repoID doesn't exist before
> > > connectStorageRepository() return.  Isn't repoID a return value
> > > of
> > > connectStorageRepository()?
> > No, repoIDs are no longer part of the domain, they are just a
> > transient handle.
> > The user can put whatever it wants there as long as it isn't
> > already taken by another currently connected domain.
> > > 
> > > >
> > > > disconnectStorageRepository(self, repoId)
> > > >
> > > >
> > > > In the new API there are only images, some images are mutable
> > > > and
> > > > some are not.
> > > > mutable images are also called VirtualDisks
> > > > immutable images are also called Snapshots
> > > >
> > > > There are no explicit templates, you can create as many images
> > > > as
> > > > you want from any snapshot.
> > > >
> > > > There are 4 major image operations:
> > > >
> > > >
> > > > createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> > > >                    userData={}, options={}):
> > > >
> > > > targetRepoId - ID of a connected repo where the disk will be
> > > > created
> > > > size - The size of the image you wish to create
> > > > baseSnapshotId - the ID of the snapshot you want the base the
> > > > new
> > > > virtual disk on
> > > > userData - optional data that will be attached to the new VD,
> > > > could
> > > > be anything that the user desires.
> > > > options - options to modify VDSMs default behavior
> > > >
> > > > returns the id of the new VD
> > > 
> > > I think we will also need a function to check if a a VirtualDisk
> > > is
> > > based on a specific snapshot.
> > > Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
> > No, the design is that volume dependencies are an implementation
> > detail.
> > There is no reason for you to know that an image is physically a
> > snapshot of another.
> > Logical snapshots, template information, and any other information
> > can be set by the user by using the userData field available for
> > every image.
> 
> Statements like this make me start to worry about your userData
> concept.  It's a
> sign of a bad API if the user needs to invent a custom metadata
> scheme for
> itself.  This reminds me of the abomination that is the 'custom'
> property in the
> vm definition today.
In one sentence: If VDSM doesn't care about it, VDSM doesn't manage it.

userData being a "void*" is quite common and I don't understand why you would thing it's a sign of a bad API.
Further more, giving the user choice about how to represent it's own metadata and what fields it want to keep seems reasonable to me.
Especially given the fact that VDSM never reads it.

The reason we are pulling away from the current system of VDSM understanding the extra data is that it makes that data tied to VDSMs on disk format.
VDSM on disk format has to be very stable because of clusters with multiple VDSM versions.
Further more, since this is actually manager data it has to be tied to the manager backward compatibility lifetime as well.
Having it be opaque to VDSM ties it to only one, simpler, support lifetime instead of two.

I guess you are implying that it will make it problematic for multiple users to read userData left by another user because the formats might not be compatible.
The solution is that all parties interested in using VDSM storage agree on format, and common fields, and supportability, and all the other things that choosing a supporting *something* entails.
This is, however, out of the scope of VDSM. When the time comes I think how the userData blob is actually parsed and what fields it keeps should be discussed on ovirt-devel or engine-devel.

The crux of the issue is that VDSM manages only what it cares about and the user can't modify directly.
This is done because everything we expose we commit to.
If you want any information persisted like:
- Human readable name (in whatever encoding)
- Is this a template or a snapshot
- What user owns this image

You can just put it in the userData.
VDSM is not going to impose what encoding you use.
It's not going to decide if you represent your users as IDs or names or ldap queries or Public Keys.
It's not going to decide if you have explicit templates or not.
It's not going to decide if you care what is the logical image chain.
It's not going to decide anything that is out of it's scope.
No format is future proof, no selection of fields will be good for any situation.
I'd much rather it be someone else's problem when any of them need to be changed.
They have currently been VDSMs problem and it has been hell to maintain.

> 
> > > > createSnapshot(targetRepoId, baseVirtualDiskId,
> > > >                 userData={}, options={}):
> > > > targetRepoId - The ID of a connected repo where the new
> > > > sanpshot
> > > > will be created and the original image exists as well.
> > > > size - The size of the image you wish to create
> > > > baseVirtualDisk - the ID of a mutable image (Virtual Disk) you
> > > > want
> > > > to snapshot
> > > > userData - optional data that will be attached to the new
> > > > Snapshot,
> > > > could be anything that the user desires.
> > > > options - options to modify VDSMs default behavior
> > > >
> > > > returns the id of the new Snapshot
> > > >
> > > > copyImage(targetRepoId, imageId, baseImageId=None, userData={},
> > > > options={})
> > > > targetRepoId - The ID of a connected repo where the new image
> > > > will
> > > > be created
> > > > imageId - The image you wish to copy
> > > > baseImageId - if specified, the new image will contain only the
> > > > diff between image and Id.
> > > >                If None the new image will contain all the bits
> > > >                of
> > > >                image Id. This can be used to copy partial parts
> > > >                of
> > > >                images for export.
> > > > userData - optional data that will be attached to the new
> > > > image,
> > > > could be anything that the user desires.
> > > > options - options to modify VDSMs default behavior
> > > 
> > > Does this function mean that we can copy the image from one
> > > repository
> > > to another repository? Does it cover the semantics of storage
> > > migration,
> > > storage backup, storage incremental backup?
> > Yes, the main purpose is copying to another repo. and you can even
> > do incremental backups.
> > Also the following flow
> > 1. Run a VM using imageA
> > 2. write to disk
> > 3. Stop VM
> > 4. copy imageA to repoB
> > 5. Run a VM using imageA again
> > 6. Write to disk
> > 7. Stop VM
> > 8. Copy imageA again basing it of imageA_copy1 on repoB creating a
> > diff on repo diff without snapshotting the original image.
> > 
> > > 
> > > >
> > > > return the Id of the new image. In case of copying an immutable
> > > > image the ID will be identical to the original image as they
> > > > contain the same data. However the user should not assume that
> > > > and
> > > > always use the value returned from the method.
> > > >
> > > > removeImage(repositoryId, imageId, options={}):
> > > > repositoryId - The ID of a connected repo where the image to
> > > > delete
> > > > resides
> > > > imageId - The id of the image you wish to delete.
> > > >
> > > >
> > > > ----
> > > > getImageStatus(repositoryId, imageId)
> > > > repositoryId - The ID of a connected repo where the image to
> > > > check
> > > > resides
> > > > imageId - The id of the image you wish to check.
> > > >
> > > > All operations return once the operations has been committed to
> > > > disk NOT when the operation actually completes.
> > > > This is done so that:
> > > > - operation come to a stable state as quickly as possible.
> > > > - In case where there is an SDM, only small portion of the
> > > > operation actually needs to be performed on the SDM host.
> > > > - No matter how many times the operation fails and on how many
> > > > hosts, you can always resume the operation and choose when to
> > > > do
> > > > it.
> > > > - You can stop an operation at any time and remove the
> > > > resulting
> > > > object making a distinction between "stop because the host is
> > > > overloaded" to "I don't want that image"
> > > >
> > > > This means that after calling any operation that creates a new
> > > > image the user must then call getImageStatus() to check what is
> > > > the status of the image.
> > > > The status of the image can be either optimized, degraded, or
> > > > broken.
> > > > "Optimized" means that the image is available and you can run
> > > > VMs
> > > > of it.
> > > > "Degraded" means that the image is available and will run VMs
> > > > but
> > > > it might be a better way VDSM can represent the underlying
> > > > data.
> > > 
> > > What does the "represent" mean here?
> > Anything, but mostly image formate RAW\QCOW2 when performance
> > strategy has been selected.
> > > > "Broken" means that the image can't be used at the moment,
> > > > probably
> > > > because not all the data has been set up on the volume.
> > > >
> > > > Apart from that VDSM will also return the last persisted status
> > > > information which will conatin
> > > > hostID - the last host to try and optimize of fix the image
> > > Any host can optimize the image? No need to be SDM?
> > On anything but lvm based block domains there will not even be an
> > SDM.
> > On SDM based domains we will try as hard as we can to have as many
> > operations executable on any host.
> > > 
> > > > stage - X/Y (eg. 1/10) the last persisted stage of the fix.
> > > > percent_complete - -1 or 0-100, the last persisted completion
> > > > percentage of the aforementioned stage. -1 means that no
> > > > progress
> > > > is available for that operation.
> > > > last_error - This will only be filled if the operation failed
> > > > because of something other then IO or a VDSM crash for obvious
> > > > reasons.
> > > >               It will usually be set if the task was manually
> > > >               stopped
> > > >
> > > > The user can either be satisfied with that information or as
> > > > the
> > > > host specified in host ID if it is still working on that image
> > > > by
> > > > checking it's running tasks.
> > > 
> > > So we need a function to know what tasks are running on the image
> > getImageStatus()
> > > >
> > > > checkStorageRepository(self, repositoryId, options={}):
> > > > A method to go over a storage repository and scan for any
> > > > existing
> > > > problems. This includes degraded\broken images and deleted
> > > > images
> > > > that have no yet been physically deleted\merged.
> > > > It returns a list of Fix objects.
> > > > Fix objects come in 4 types:
> > > > clean - cleans data, run them to get more space.
> > > > optimize - run them to optimize a degraded image
> > > > merge - Merges two images together. Doing this sometimes
> > > >          makes more images ready optimizing or cleaning.
> > > >          The reason it is different from optimize is that
> > > >          unmerged images are considered optimized.
> > > > mend - mends a broken image
> > > >
> > > > The user can read these types and prioritize fixes. Fixes also
> > > > contain opaque FIX data and they should be sent as received to
> > > > fixStorageRepository(self, repositoryId, fix, options={}):
> > > >
> > > > That will start a fix operation.
> > > >
> > > >
> > > > All major operations automatically start the appropriate "Fix"
> > > > to
> > > > bring the created object to an optimize\degraded state (the one
> > > > that is quicker) unless one of the options is
> > > > AutoFix=False. This is only useful for repos that might not be
> > > > able
> > > > to create volumes on all hosts (SDM) but would like to have the
> > > > actual IO distributed in the cluster.
> > > >
> > > > Other common options is the strategy option:
> > > > It has currently 2 possible values
> > > > space and performance - In case VDSM has 2 ways of completing
> > > > the
> > > > same operation it will tell it to value one over the other. For
> > > > example, whether to copy all the data or just create a qcow
> > > > based
> > > > of a snapshot.
> > > > The default is space.
> > > >
> > > > You might have also noticed that it is never explicitly
> > > > specified
> > > > where to look for existing images. This is done purposefully,
> > > > VDSM
> > > > will always look in all connected repositories for existing
> > > > objects.
> > > > For very large setups this might be problematic. To mitigate
> > > > the
> > > > problem you have these options:
> > > > participatingRepositories=[repoId, ...] which tell VDSM to
> > > > narrow
> > > > the search to just these repositories
> > > > and
> > > > imageHints={imgId: repoId} which will force VDSM to look for
> > > > those
> > > > image ID just in those repositories and fail if it doesn't find
> > > > them there.
> > > > _______________________________________________
> > > > vdsm-devel mailing list
> > > > vdsm-devel at lists.fedorahosted.org
> > > > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> > > 
> > > 
> > > --
> > > ---
> > > ?? Shu Ming
> > > Open Virtualization Engineerning; CSTL, IBM Corp.
> > > Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com
> > > or
> > > shuming at linux.vnet.ibm.com
> > > Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> > > District, Beijing 100193, PRC
> > > 
> > > 
> > > 
> > _______________________________________________
> > vdsm-devel mailing list
> > vdsm-devel at lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 
> --
> Adam Litke <agl at us.ibm.com>
> IBM Linux Technology Center
> 
>


From acathrow at redhat.com  Mon Dec 10 20:49:32 2012
From: acathrow at redhat.com (Andrew Cathrow)
Date: Mon, 10 Dec 2012 15:49:32 -0500 (EST)
Subject: [Engine-devel] Attaching a console should be controllable
In-Reply-To: <50C62A9B.6020208@redhat.com>
Message-ID: <576664928.46844358.1355172572157.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Roy Golan" <rgolan at redhat.com>
> To: engine-devel at ovirt.org
> Sent: Monday, December 10, 2012 1:31:55 PM
> Subject: Re: [Engine-devel] Attaching a console should be controllable
> 
> On 12/10/2012 02:28 PM, Frantisek Kobzik wrote:
> > Hi guys,
> >
> > Just a simple question. I'm writing an engine counterpart for this
> > vdsm feature:
> > https://bugzilla.redhat.com/show_bug.cgi?id=878459
> > http://gerrit.ovirt.org/#/c/9359
> >
> > Long story short, it adds the possibility of controlling adding the
> > virtio serial console to a VM.
> >
> > I'm going to add a new column "console_enabled" to vm_static and a

I'd make it serial_console since console can mean different things (eg vnc console, spice, etc)


> > new field "bool consoleEnabled" to appropriate entities. I'm just
> > worried about the naming of the column and the field - could I use
> > these names? Isn't it confusing? I just want to avoid unneeded
> > refactoring when the patch is done.
> both fits the convention - db columns underscored and java field
> camelCased.
> > Any feedback is welcome!
> >
> > Thanks,
> > Frank
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> 
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From agl at us.ibm.com  Mon Dec 10 21:17:13 2012
From: agl at us.ibm.com (Adam Litke)
Date: Mon, 10 Dec 2012 15:17:13 -0600
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <2033079376.16686610.1355166189638.JavaMail.root@redhat.com>
References: <20121210184931.GF10131@aglitke>
	<2033079376.16686610.1355166189638.JavaMail.root@redhat.com>
Message-ID: <20121210194615.GG10131@aglitke>

On Mon, Dec 10, 2012 at 02:03:09PM -0500, Saggi Mizrahi wrote:
> 
> 
> ----- Original Message -----
> > From: "Adam Litke" <agl at us.ibm.com>
> > To: "Saggi Mizrahi" <smizrahi at redhat.com>
> > Cc: "Deepak C Shetty" <deepakcs at linux.vnet.ibm.com>, "engine-devel" <engine-devel at ovirt.org>, "VDSM Project
> > Development" <vdsm-devel at lists.fedorahosted.org>
> > Sent: Monday, December 10, 2012 1:49:31 PM
> > Subject: Re: [vdsm] RFC: New Storage API
> > 
> > On Fri, Dec 07, 2012 at 02:53:41PM -0500, Saggi Mizrahi wrote:
> > 
> > <snip>
> > 
> > > > 1) Can you provide more info on why there is a exception for 'lvm
> > > > based
> > > > block domain'. Its not coming out clearly.
> > > File based domains are responsible for syncing up object
> > > manipulation (creation\deletion)
> > > The backend is responsible for making sure it all works either by
> > > having a single writer (NFS) or having it's own locking mechanism
> > > (gluster).
> > > In our LVM based domains VDSM is responsible for basic object
> > > manipulation.
> > > The current design uses an approach where there is a single host
> > > responsible for object creation\deleteion it is the
> > > SRM\SDM\SPM\S?M.
> > > If we ever find a way to make it fully clustered without a big hit
> > > in performance the S?M requirement will be removed form that type
> > > of domain.
> > 
> > I would like to see us maintain a LOCALFS domain as well.  For this,
> > we would
> > also need SRM, correct?
> No, why?

Sorry, nevermind.  I was thinking of a scenario with multiple clients talking to
a single vdsm and making sure they don't stomp on one another.  This is
probably not something we are going to care about though.

-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center



From agl at us.ibm.com  Mon Dec 10 21:47:46 2012
From: agl at us.ibm.com (Adam Litke)
Date: Mon, 10 Dec 2012 15:47:46 -0600
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <1874293712.16743050.1355171783487.JavaMail.root@redhat.com>
References: <20121210183951.GE10131@aglitke>
	<1874293712.16743050.1355171783487.JavaMail.root@redhat.com>
Message-ID: <20121210214746.GH10131@aglitke>

On Mon, Dec 10, 2012 at 03:36:23PM -0500, Saggi Mizrahi wrote:
> > Statements like this make me start to worry about your userData
> > concept.  It's a
> > sign of a bad API if the user needs to invent a custom metadata
> > scheme for
> > itself.  This reminds me of the abomination that is the 'custom'
> > property in the
> > vm definition today.
> In one sentence: If VDSM doesn't care about it, VDSM doesn't manage it.
> 
> userData being a "void*" is quite common and I don't understand why you would thing it's a sign of a bad API.
> Further more, giving the user choice about how to represent it's own metadata and what fields it want to keep seems reasonable to me.
> Especially given the fact that VDSM never reads it.
> 
> The reason we are pulling away from the current system of VDSM understanding the extra data is that it makes that data tied to VDSMs on disk format.
> VDSM on disk format has to be very stable because of clusters with multiple VDSM versions.
> Further more, since this is actually manager data it has to be tied to the manager backward compatibility lifetime as well.
> Having it be opaque to VDSM ties it to only one, simpler, support lifetime instead of two.
> 
> I guess you are implying that it will make it problematic for multiple users to read userData left by another user because the formats might not be compatible.
> The solution is that all parties interested in using VDSM storage agree on format, and common fields, and supportability, and all the other things that choosing a supporting *something* entails.
> This is, however, out of the scope of VDSM. When the time comes I think how the userData blob is actually parsed and what fields it keeps should be discussed on ovirt-devel or engine-devel.
> 
> The crux of the issue is that VDSM manages only what it cares about and the user can't modify directly.
> This is done because everything we expose we commit to.
> If you want any information persisted like:
> - Human readable name (in whatever encoding)
> - Is this a template or a snapshot
> - What user owns this image
> 
> You can just put it in the userData.
> VDSM is not going to impose what encoding you use.
> It's not going to decide if you represent your users as IDs or names or ldap queries or Public Keys.
> It's not going to decide if you have explicit templates or not.
> It's not going to decide if you care what is the logical image chain.
> It's not going to decide anything that is out of it's scope.
> No format is future proof, no selection of fields will be good for any situation.
> I'd much rather it be someone else's problem when any of them need to be changed.
> They have currently been VDSMs problem and it has been hell to maintain.

In general, I actually agree with most of this.  What I want to avoid is pushing
things that should actually be a part of the API into this userData blob.  We do
want to keep the API as simple as possible to give vdsm flexibility.  If, over
time, we find that users are always using userData to work around something
missing in the API, this could be a really good sign that the API needs
extension.

-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center



From smizrahi at redhat.com  Mon Dec 10 22:38:40 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 10 Dec 2012 17:38:40 -0500 (EST)
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <20121210214746.GH10131@aglitke>
Message-ID: <493555509.16798632.1355179120348.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Adam Litke" <agl at us.ibm.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "Shu Ming" <shuming at linux.vnet.ibm.com>, "engine-devel" <engine-devel at ovirt.org>, "VDSM Project Development"
> <vdsm-devel at lists.fedorahosted.org>
> Sent: Monday, December 10, 2012 4:47:46 PM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> On Mon, Dec 10, 2012 at 03:36:23PM -0500, Saggi Mizrahi wrote:
> > > Statements like this make me start to worry about your userData
> > > concept.  It's a
> > > sign of a bad API if the user needs to invent a custom metadata
> > > scheme for
> > > itself.  This reminds me of the abomination that is the 'custom'
> > > property in the
> > > vm definition today.
> > In one sentence: If VDSM doesn't care about it, VDSM doesn't manage
> > it.
> > 
> > userData being a "void*" is quite common and I don't understand why
> > you would thing it's a sign of a bad API.
> > Further more, giving the user choice about how to represent it's
> > own metadata and what fields it want to keep seems reasonable to
> > me.
> > Especially given the fact that VDSM never reads it.
> > 
> > The reason we are pulling away from the current system of VDSM
> > understanding the extra data is that it makes that data tied to
> > VDSMs on disk format.
> > VDSM on disk format has to be very stable because of clusters with
> > multiple VDSM versions.
> > Further more, since this is actually manager data it has to be tied
> > to the manager backward compatibility lifetime as well.
> > Having it be opaque to VDSM ties it to only one, simpler, support
> > lifetime instead of two.
> > 
> > I guess you are implying that it will make it problematic for
> > multiple users to read userData left by another user because the
> > formats might not be compatible.
> > The solution is that all parties interested in using VDSM storage
> > agree on format, and common fields, and supportability, and all
> > the other things that choosing a supporting *something* entails.
> > This is, however, out of the scope of VDSM. When the time comes I
> > think how the userData blob is actually parsed and what fields it
> > keeps should be discussed on ovirt-devel or engine-devel.
> > 
> > The crux of the issue is that VDSM manages only what it cares about
> > and the user can't modify directly.
> > This is done because everything we expose we commit to.
> > If you want any information persisted like:
> > - Human readable name (in whatever encoding)
> > - Is this a template or a snapshot
> > - What user owns this image
> > 
> > You can just put it in the userData.
> > VDSM is not going to impose what encoding you use.
> > It's not going to decide if you represent your users as IDs or
> > names or ldap queries or Public Keys.
> > It's not going to decide if you have explicit templates or not.
> > It's not going to decide if you care what is the logical image
> > chain.
> > It's not going to decide anything that is out of it's scope.
> > No format is future proof, no selection of fields will be good for
> > any situation.
> > I'd much rather it be someone else's problem when any of them need
> > to be changed.
> > They have currently been VDSMs problem and it has been hell to
> > maintain.
> 
> In general, I actually agree with most of this.  What I want to avoid
> is pushing
> things that should actually be a part of the API into this userData
> blob.  We do
> want to keep the API as simple as possible to give vdsm flexibility.
>  If, over
> time, we find that users are always using userData to work around
> something
> missing in the API, this could be a really good sign that the API
> needs
> extension.
I was actually contemplating about this for quite a while.
If while you create an image the reply is lost or, VDSM is unable to know if the operation was committed or not, the user will have no way of knowing what thew new image ID is.
To solve this it is recommended that the manager puts some sort of task related information in the user data.
If the operation ever finishes in an an ambiguous state the user just reads the userData from any images it doesn't know or is unsure about their state.

This is a flow that every client will have to have.
So why not just add that to the API?
Because I don't want to impose how this "information" gets generated, what is the content of that data or how unique it has to be.
Since VDSM doesn't use it for anything, I don't feel like I need to figure this out.
I am all for simplicity, but simplicity is kind of an abstract concept. Having it be a blob is in some aspects the simplest thing you can do.
Just saying that "I have a field, put whatever in it" is simple to convey but does requires more work on the user's side to figure out what to do with it.

All that being said, I do think that the format, fields and how to use them should be defined so different users can communicate and synchronize.
It's also important that you don't reinvent the wheel for every flow in every client.
I'm just saying that it's not in the scope of VDSM.
It should be done as a standard that all users of VDSM agree too conform to.
It's the same way that a file-system doesn't check that every *.PDF file is a proper PDF file. PDF clients agree on how to read\save files and users promise to mark them with that suffix.
Think of what happened if the FS did check that kind of thing. PDF features will be bound to the kernel release cycle so it doesn't refuse to save\load a file just because your PDF client is newer or older than the kernel.
You will have to have code to validate PDF files in the kernel. That means more code which, in turn, means more bugs.
> 
> --
> Adam Litke <agl at us.ibm.com>
> IBM Linux Technology Center
> 
> 


From shuming at linux.vnet.ibm.com  Tue Dec 11 03:33:23 2012
From: shuming at linux.vnet.ibm.com (Shu Ming)
Date: Tue, 11 Dec 2012 11:33:23 +0800
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <1874293712.16743050.1355171783487.JavaMail.root@redhat.com>
References: <1874293712.16743050.1355171783487.JavaMail.root@redhat.com>
Message-ID: <50C6A983.9090405@linux.vnet.ibm.com>

2012-12-11 4:36, Saggi Mizrahi:
>
> ----- Original Message -----
>> From: "Adam Litke" <agl at us.ibm.com>
>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
>> Cc: "Shu Ming" <shuming at linux.vnet.ibm.com>, "engine-devel" <engine-devel at ovirt.org>, "VDSM Project Development"
>> <vdsm-devel at lists.fedorahosted.org>
>> Sent: Monday, December 10, 2012 1:39:51 PM
>> Subject: Re: [vdsm] RFC: New Storage API
>>
>> On Thu, Dec 06, 2012 at 11:52:01AM -0500, Saggi Mizrahi wrote:
>>>
>>> ----- Original Message -----
>>>> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
>>>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
>>>> Cc: "VDSM Project Development"
>>>> <vdsm-devel at lists.fedorahosted.org>, "engine-devel"
>>>> <engine-devel at ovirt.org>
>>>> Sent: Thursday, December 6, 2012 11:02:02 AM
>>>> Subject: Re: [vdsm] RFC: New Storage API
>>>>
>>>> Saggi,
>>>>
>>>> Thanks for sharing your thought and I get some comments below.
>>>>
>>>>
>>>> Saggi Mizrahi:
>>>>> I've been throwing a lot of bits out about the new storage API
>>>>> and
>>>>> I think it's time to talk a bit.
>>>>> I will purposefully try and keep implementation details away
>>>>> and
>>>>> concentrate about how the API looks and how you use it.
>>>>>
>>>>> First major change is in terminology, there is no long a
>>>>> storage
>>>>> domain but a storage repository.
>>>>> This change is done because so many things are already called
>>>>> domain in the system and this will make things less confusing
>>>>> for
>>>>> new-commers with a libvirt background.
>>>>>
>>>>> One other changes is that repositories no longer have a UUID.
>>>>> The UUID was only used in the pool members manifest and is no
>>>>> longer needed.
>>>>>
>>>>>
>>>>> connectStorageRepository(repoId, repoFormat,
>>>>> connectionParameters={}):
>>>>> repoId - is a transient name that will be used to refer to the
>>>>> connected domain, it is not persisted and doesn't have to be
>>>>> the
>>>>> same across the cluster.
>>>>> repoFormat - Similar to what used to be type (eg. localfs-1.0,
>>>>> nfs-3.4, clvm-1.2).
>>>>> connectionParameters - This is format specific and will used to
>>>>> tell VDSM how to connect to the repo.
>>>>
>>>> Where does repoID come from? I think repoID doesn't exist before
>>>> connectStorageRepository() return.  Isn't repoID a return value
>>>> of
>>>> connectStorageRepository()?
>>> No, repoIDs are no longer part of the domain, they are just a
>>> transient handle.
>>> The user can put whatever it wants there as long as it isn't
>>> already taken by another currently connected domain.
>>>>> disconnectStorageRepository(self, repoId)
>>>>>
>>>>>
>>>>> In the new API there are only images, some images are mutable
>>>>> and
>>>>> some are not.
>>>>> mutable images are also called VirtualDisks
>>>>> immutable images are also called Snapshots
>>>>>
>>>>> There are no explicit templates, you can create as many images
>>>>> as
>>>>> you want from any snapshot.
>>>>>
>>>>> There are 4 major image operations:
>>>>>
>>>>>
>>>>> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
>>>>>                     userData={}, options={}):
>>>>>
>>>>> targetRepoId - ID of a connected repo where the disk will be
>>>>> created
>>>>> size - The size of the image you wish to create
>>>>> baseSnapshotId - the ID of the snapshot you want the base the
>>>>> new
>>>>> virtual disk on
>>>>> userData - optional data that will be attached to the new VD,
>>>>> could
>>>>> be anything that the user desires.
>>>>> options - options to modify VDSMs default behavior
>>>>>
>>>>> returns the id of the new VD
>>>> I think we will also need a function to check if a a VirtualDisk
>>>> is
>>>> based on a specific snapshot.
>>>> Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
>>> No, the design is that volume dependencies are an implementation
>>> detail.
>>> There is no reason for you to know that an image is physically a
>>> snapshot of another.
>>> Logical snapshots, template information, and any other information
>>> can be set by the user by using the userData field available for
>>> every image.
>> Statements like this make me start to worry about your userData
>> concept.  It's a
>> sign of a bad API if the user needs to invent a custom metadata
>> scheme for
>> itself.  This reminds me of the abomination that is the 'custom'
>> property in the
>> vm definition today.
> In one sentence: If VDSM doesn't care about it, VDSM doesn't manage it.
>
> userData being a "void*" is quite common and I don't understand why you would thing it's a sign of a bad API.
> Further more, giving the user choice about how to represent it's own metadata and what fields it want to keep seems reasonable to me.
> Especially given the fact that VDSM never reads it.
>
> The reason we are pulling away from the current system of VDSM understanding the extra data is that it makes that data tied to VDSMs on disk format.
> VDSM on disk format has to be very stable because of clusters with multiple VDSM versions.
> Further more, since this is actually manager data it has to be tied to the manager backward compatibility lifetime as well.
> Having it be opaque to VDSM ties it to only one, simpler, support lifetime instead of two.

Making userData being opaque gives flexibilities to the management 
applications.  To me, opaque userDaa can have two types at least. The 
first is the userData for runtime only.  The second is the userData 
expected to be persisted into the metadata disk.  For the first type, 
the management applications can store their own data structures like 
temporary task states, VDSM query caches &etc. After the VDSM host is 
fenced, the userData goes away. For the second type, different 
management applications can have different formats of userData and 
persisting it to the VDSM metadata disk will corrupt each other in very 
high possibility.  You may argue that, all the management applications 
should reach a agreement before they use it as the second type.  But in 
practice, it is pretty hard to maintain.


>
> I guess you are implying that it will make it problematic for multiple users to read userData left by another user because the formats might not be compatible.
> The solution is that all parties interested in using VDSM storage agree on format, and common fields, and supportability, and all the other things that choosing a supporting *something* entails.
> This is, however, out of the scope of VDSM. When the time comes I think how the userData blob is actually parsed and what fields it keeps should be discussed on ovirt-devel or engine-devel.
>
> The crux of the issue is that VDSM manages only what it cares about and the user can't modify directly.
> This is done because everything we expose we commit to.
> If you want any information persisted like:
> - Human readable name (in whatever encoding)
> - Is this a template or a snapshot
> - What user owns this image
>
> You can just put it in the userData.
> VDSM is not going to impose what encoding you use.
> It's not going to decide if you represent your users as IDs or names or ldap queries or Public Keys.
> It's not going to decide if you have explicit templates or not.
> It's not going to decide if you care what is the logical image chain.
> It's not going to decide anything that is out of it's scope.
> No format is future proof, no selection of fields will be good for any situation.
> I'd much rather it be someone else's problem when any of them need to be changed.
> They have currently been VDSMs problem and it has been hell to maintain.
>
>>>>> createSnapshot(targetRepoId, baseVirtualDiskId,
>>>>>                  userData={}, options={}):
>>>>> targetRepoId - The ID of a connected repo where the new
>>>>> sanpshot
>>>>> will be created and the original image exists as well.
>>>>> size - The size of the image you wish to create
>>>>> baseVirtualDisk - the ID of a mutable image (Virtual Disk) you
>>>>> want
>>>>> to snapshot
>>>>> userData - optional data that will be attached to the new
>>>>> Snapshot,
>>>>> could be anything that the user desires.
>>>>> options - options to modify VDSMs default behavior
>>>>>
>>>>> returns the id of the new Snapshot
>>>>>
>>>>> copyImage(targetRepoId, imageId, baseImageId=None, userData={},
>>>>> options={})
>>>>> targetRepoId - The ID of a connected repo where the new image
>>>>> will
>>>>> be created
>>>>> imageId - The image you wish to copy
>>>>> baseImageId - if specified, the new image will contain only the
>>>>> diff between image and Id.
>>>>>                 If None the new image will contain all the bits
>>>>>                 of
>>>>>                 image Id. This can be used to copy partial parts
>>>>>                 of
>>>>>                 images for export.
>>>>> userData - optional data that will be attached to the new
>>>>> image,
>>>>> could be anything that the user desires.
>>>>> options - options to modify VDSMs default behavior
>>>> Does this function mean that we can copy the image from one
>>>> repository
>>>> to another repository? Does it cover the semantics of storage
>>>> migration,
>>>> storage backup, storage incremental backup?
>>> Yes, the main purpose is copying to another repo. and you can even
>>> do incremental backups.
>>> Also the following flow
>>> 1. Run a VM using imageA
>>> 2. write to disk
>>> 3. Stop VM
>>> 4. copy imageA to repoB
>>> 5. Run a VM using imageA again
>>> 6. Write to disk
>>> 7. Stop VM
>>> 8. Copy imageA again basing it of imageA_copy1 on repoB creating a
>>> diff on repo diff without snapshotting the original image.
>>>
>>>>> return the Id of the new image. In case of copying an immutable
>>>>> image the ID will be identical to the original image as they
>>>>> contain the same data. However the user should not assume that
>>>>> and
>>>>> always use the value returned from the method.
>>>>>
>>>>> removeImage(repositoryId, imageId, options={}):
>>>>> repositoryId - The ID of a connected repo where the image to
>>>>> delete
>>>>> resides
>>>>> imageId - The id of the image you wish to delete.
>>>>>
>>>>>
>>>>> ----
>>>>> getImageStatus(repositoryId, imageId)
>>>>> repositoryId - The ID of a connected repo where the image to
>>>>> check
>>>>> resides
>>>>> imageId - The id of the image you wish to check.
>>>>>
>>>>> All operations return once the operations has been committed to
>>>>> disk NOT when the operation actually completes.
>>>>> This is done so that:
>>>>> - operation come to a stable state as quickly as possible.
>>>>> - In case where there is an SDM, only small portion of the
>>>>> operation actually needs to be performed on the SDM host.
>>>>> - No matter how many times the operation fails and on how many
>>>>> hosts, you can always resume the operation and choose when to
>>>>> do
>>>>> it.
>>>>> - You can stop an operation at any time and remove the
>>>>> resulting
>>>>> object making a distinction between "stop because the host is
>>>>> overloaded" to "I don't want that image"
>>>>>
>>>>> This means that after calling any operation that creates a new
>>>>> image the user must then call getImageStatus() to check what is
>>>>> the status of the image.
>>>>> The status of the image can be either optimized, degraded, or
>>>>> broken.
>>>>> "Optimized" means that the image is available and you can run
>>>>> VMs
>>>>> of it.
>>>>> "Degraded" means that the image is available and will run VMs
>>>>> but
>>>>> it might be a better way VDSM can represent the underlying
>>>>> data.
>>>> What does the "represent" mean here?
>>> Anything, but mostly image formate RAW\QCOW2 when performance
>>> strategy has been selected.
>>>>> "Broken" means that the image can't be used at the moment,
>>>>> probably
>>>>> because not all the data has been set up on the volume.
>>>>>
>>>>> Apart from that VDSM will also return the last persisted status
>>>>> information which will conatin
>>>>> hostID - the last host to try and optimize of fix the image
>>>> Any host can optimize the image? No need to be SDM?
>>> On anything but lvm based block domains there will not even be an
>>> SDM.
>>> On SDM based domains we will try as hard as we can to have as many
>>> operations executable on any host.
>>>>> stage - X/Y (eg. 1/10) the last persisted stage of the fix.
>>>>> percent_complete - -1 or 0-100, the last persisted completion
>>>>> percentage of the aforementioned stage. -1 means that no
>>>>> progress
>>>>> is available for that operation.
>>>>> last_error - This will only be filled if the operation failed
>>>>> because of something other then IO or a VDSM crash for obvious
>>>>> reasons.
>>>>>                It will usually be set if the task was manually
>>>>>                stopped
>>>>>
>>>>> The user can either be satisfied with that information or as
>>>>> the
>>>>> host specified in host ID if it is still working on that image
>>>>> by
>>>>> checking it's running tasks.
>>>> So we need a function to know what tasks are running on the image
>>> getImageStatus()
>>>>> checkStorageRepository(self, repositoryId, options={}):
>>>>> A method to go over a storage repository and scan for any
>>>>> existing
>>>>> problems. This includes degraded\broken images and deleted
>>>>> images
>>>>> that have no yet been physically deleted\merged.
>>>>> It returns a list of Fix objects.
>>>>> Fix objects come in 4 types:
>>>>> clean - cleans data, run them to get more space.
>>>>> optimize - run them to optimize a degraded image
>>>>> merge - Merges two images together. Doing this sometimes
>>>>>           makes more images ready optimizing or cleaning.
>>>>>           The reason it is different from optimize is that
>>>>>           unmerged images are considered optimized.
>>>>> mend - mends a broken image
>>>>>
>>>>> The user can read these types and prioritize fixes. Fixes also
>>>>> contain opaque FIX data and they should be sent as received to
>>>>> fixStorageRepository(self, repositoryId, fix, options={}):
>>>>>
>>>>> That will start a fix operation.
>>>>>
>>>>>
>>>>> All major operations automatically start the appropriate "Fix"
>>>>> to
>>>>> bring the created object to an optimize\degraded state (the one
>>>>> that is quicker) unless one of the options is
>>>>> AutoFix=False. This is only useful for repos that might not be
>>>>> able
>>>>> to create volumes on all hosts (SDM) but would like to have the
>>>>> actual IO distributed in the cluster.
>>>>>
>>>>> Other common options is the strategy option:
>>>>> It has currently 2 possible values
>>>>> space and performance - In case VDSM has 2 ways of completing
>>>>> the
>>>>> same operation it will tell it to value one over the other. For
>>>>> example, whether to copy all the data or just create a qcow
>>>>> based
>>>>> of a snapshot.
>>>>> The default is space.
>>>>>
>>>>> You might have also noticed that it is never explicitly
>>>>> specified
>>>>> where to look for existing images. This is done purposefully,
>>>>> VDSM
>>>>> will always look in all connected repositories for existing
>>>>> objects.
>>>>> For very large setups this might be problematic. To mitigate
>>>>> the
>>>>> problem you have these options:
>>>>> participatingRepositories=[repoId, ...] which tell VDSM to
>>>>> narrow
>>>>> the search to just these repositories
>>>>> and
>>>>> imageHints={imgId: repoId} which will force VDSM to look for
>>>>> those
>>>>> image ID just in those repositories and fail if it doesn't find
>>>>> them there.
>>>>> _______________________________________________
>>>>> vdsm-devel mailing list
>>>>> vdsm-devel at lists.fedorahosted.org
>>>>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>>>>
>>>> --
>>>> ---
>>>> ?? Shu Ming
>>>> Open Virtualization Engineerning; CSTL, IBM Corp.
>>>> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com
>>>> or
>>>> shuming at linux.vnet.ibm.com
>>>> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
>>>> District, Beijing 100193, PRC
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> vdsm-devel mailing list
>>> vdsm-devel at lists.fedorahosted.org
>>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>> --
>> Adam Litke <agl at us.ibm.com>
>> IBM Linux Technology Center
>>
>>


-- 
---
?? Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com or shuming at linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC




From amureini at redhat.com  Tue Dec 11 09:02:29 2012
From: amureini at redhat.com (Allon Mureinik)
Date: Tue, 11 Dec 2012 04:02:29 -0500 (EST)
Subject: [Engine-devel] Inherited Permissions from System
In-Reply-To: <50C5FA3D.70004@redhat.com>
Message-ID: <1811698640.13846827.1355216549247.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Ravi Nori" <rnori at redhat.com>
> To: engine-devel at ovirt.org
> Sent: Monday, December 10, 2012 5:05:33 PM
> Subject: [Engine-devel] Inherited Permissions from System
> 
> I am currently working on a bug to show the inherited permissions on
> an
> object.
> 
> So if the user has permissions assigned on a Cluster, the user should
> have permissions on the VMs in the cluster.
> 
> My question is if the user has permissions on the System, should the
> permissions be inherited to all the entities?
In a word: yes.

> 
> Link to the bug
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=871365
> 
> Thanks
> 
> Ravi
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From gang.wei at intel.com  Tue Dec 11 11:35:59 2012
From: gang.wei at intel.com (Wei, Gang)
Date: Tue, 11 Dec 2012 11:35:59 +0000
Subject: [Engine-devel] Trusted Compute Pools
In-Reply-To: <50B89105.4010006@redhat.com>
References: <D0B11485C64D4B47B66902F8A4E901BE3FCDA2@SHSMSX101.ccr.corp.intel.com>
	<50B89105.4010006@redhat.com>
Message-ID: <D0B11485C64D4B47B66902F8A4E901BE43D51B@SHSMSX101.ccr.corp.intel.com>

Itamar Heim wrote on 2012-11-30:
> for a POC patch i suggest: 
> 1. use vm custom level property as simpler for first phase then changing
the db scheme.
> 2. use bare minimum engine config support for service
location/authentication.

A dummy question: where is the engine config file?

Jimmy

> 3. assume no plugins and just add the code to the relevant part in RunVM
which filters the
> relevant hosts:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 8586 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121211/07c98cb6/attachment.p7s>

From ovedo at redhat.com  Tue Dec 11 12:30:14 2012
From: ovedo at redhat.com (Oved Ourfalli)
Date: Tue, 11 Dec 2012 07:30:14 -0500 (EST)
Subject: [Engine-devel] Trusted Compute Pools
In-Reply-To: <D0B11485C64D4B47B66902F8A4E901BE43D51B@SHSMSX101.ccr.corp.intel.com>
Message-ID: <6421923.60015738.1355229014088.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Gang Wei" <gang.wei at intel.com>
> To: "Itamar Heim" <iheim at redhat.com>
> Cc: engine-devel at ovirt.org, "James Rankin" <jrankin at redhat.com>, "Oved Ourfalli" <oourfali at redhat.com>, "Barak
> Azulay" <bazulay at redhat.com>, "Michal Skrivanek" <mskrivan at redhat.com>, "Buddy Cao" <buddy.cao at intel.com>, "Jeff
> Ruby" <jeff.ruby at intel.com>, "Gang Wei" <gang.wei at intel.com>
> Sent: Tuesday, December 11, 2012 1:35:59 PM
> Subject: RE: [Engine-devel] Trusted Compute Pools
> 
> Itamar Heim wrote on 2012-11-30:
> > for a POC patch i suggest:
> > 1. use vm custom level property as simpler for first phase then
> > changing
> the db scheme.
> > 2. use bare minimum engine config support for service
> location/authentication.
> 
> A dummy question: where is the engine config file?
> 
The engine configuration is in the database, in the vdc_options table.
You can edit it manually.

In an installed environment we also have the engine-config utility, which allows to modify existing configuration entries.

Oved
> Jimmy
> 
> > 3. assume no plugins and just add the code to the relevant part in
> > RunVM
> which filters the
> > relevant hosts:
> 


From lhornyak at redhat.com  Tue Dec 11 13:26:26 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Tue, 11 Dec 2012 08:26:26 -0500 (EST)
Subject: [Engine-devel] compatibility version
In-Reply-To: <641686052.8051465.1355232086789.JavaMail.root@redhat.com>
Message-ID: <48862853.8052872.1355232386030.JavaMail.root@redhat.com>

Hi,

When I install an ovirt engine from scratch, the Default DC is set to Compatibility version 3.1, but when I create a new DC, it's Compatibility version is set to 3.2 by default. Is this intentional? Why?

Laszlo


From lpeer at redhat.com  Tue Dec 11 13:35:13 2012
From: lpeer at redhat.com (Livnat Peer)
Date: Tue, 11 Dec 2012 15:35:13 +0200
Subject: [Engine-devel] compatibility version
In-Reply-To: <48862853.8052872.1355232386030.JavaMail.root@redhat.com>
References: <48862853.8052872.1355232386030.JavaMail.root@redhat.com>
Message-ID: <50C73691.6060103@redhat.com>

On 11/12/12 15:26, Laszlo Hornyak wrote:
> Hi,
> 
> When I install an ovirt engine from scratch, the Default DC is set to Compatibility version 3.1, but when I create a new DC, it's Compatibility version is set to 3.2 by default. Is this intentional? Why?
> 

Looks like a bug.
I think that when we added 3.2 support we did not change the pre-defined
DC to match it (BTW - I guess it should be fixed for default cluster as
well).


> Laszlo
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 



From lhornyak at redhat.com  Tue Dec 11 14:09:48 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Tue, 11 Dec 2012 09:09:48 -0500 (EST)
Subject: [Engine-devel] compatibility version
In-Reply-To: <50C73691.6060103@redhat.com>
Message-ID: <373056378.8080554.1355234988371.JavaMail.root@redhat.com>


----- Original Message -----
> From: "Livnat Peer" <lpeer at redhat.com>
> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Tuesday, December 11, 2012 2:35:13 PM
> Subject: Re: [Engine-devel] compatibility version
> 
> On 11/12/12 15:26, Laszlo Hornyak wrote:
> > Hi,
> > 
> > When I install an ovirt engine from scratch, the Default DC is set
> > to Compatibility version 3.1, but when I create a new DC, it's
> > Compatibility version is set to 3.2 by default. Is this
> > intentional? Why?
> > 
> 
> Looks like a bug.

Ok, sent a patch:
http://gerrit.ovirt.org/9844

> I think that when we added 3.2 support we did not change the
> pre-defined
> DC to match it (BTW - I guess it should be fixed for default cluster
> as
> well).

yes, the same is true for cluster.

> 
> 
> > Laszlo
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > 
> 
> 


From gang.wei at intel.com  Tue Dec 11 14:33:25 2012
From: gang.wei at intel.com (Wei, Gang)
Date: Tue, 11 Dec 2012 14:33:25 +0000
Subject: [Engine-devel] Trusted Compute Pools
In-Reply-To: <6421923.60015738.1355229014088.JavaMail.root@redhat.com>
References: <D0B11485C64D4B47B66902F8A4E901BE43D51B@SHSMSX101.ccr.corp.intel.com>
	<6421923.60015738.1355229014088.JavaMail.root@redhat.com>
Message-ID: <D0B11485C64D4B47B66902F8A4E901BE43DEBC@SHSMSX101.ccr.corp.intel.com>

Oved Ourfalli wrote on 2012-12-11:
>
>> A dummy question: where is the engine config file?
>>
> The engine configuration is in the database, in the vdc_options table.
> You can edit it manually.
>
> In an installed environment we also have the engine-config utility, which 
> allows
> to modify existing configuration entries.

Thanks for the information. It seems necessary for me to setup a installed 
environment to get more familiar with oVirt configuration & usage.

Jimmy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 8586 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121211/78ff0322/attachment.p7s>

From eedri at redhat.com  Tue Dec 11 16:24:07 2012
From: eedri at redhat.com (Eyal Edri)
Date: Tue, 11 Dec 2012 11:24:07 -0500 (EST)
Subject: [Engine-devel] Request power-user status for mom project in
	jenkins
In-Reply-To: <20121211161145.GB30988@aglitke>
Message-ID: <33068073.14181482.1355243047864.JavaMail.root@redhat.com>

[adding engine-devel]

usually there is an official vote on each new power user for jenkins,
and as long as there are no objections it is approved.

since i know you're a mom maintainer and familiar with jenkins, i vote +1.

Eyal.

----- Original Message -----
> From: "Adam Litke" <agl at us.ibm.com>
> To: infra at ovirt.org
> Cc: eedri at redhat.com
> Sent: Tuesday, December 11, 2012 6:11:45 PM
> Subject: Request power-user status for mom project in jenkins
> 
> Hi all,
> 
> I am the maintainer of the mom sub-project.  Recently we upgraded the
> mom build
> process to use autotools.  Unfortunately this has broken the Jenkins
> build for
> the project.  Rather than asking here whenever I need an update to
> the build
> job, I think it makes sense for me to administer that myself.  If
> there is
> agreement about this, could someone help me to acquire the relavent
> permissions?
> 
> Thanks!
> 
> --
> Adam Litke <agl at us.ibm.com>
> IBM Linux Technology Center
> 
> 


From lhornyak at redhat.com  Tue Dec 11 17:18:52 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Tue, 11 Dec 2012 12:18:52 -0500 (EST)
Subject: [Engine-devel] another datamodel cleanup
In-Reply-To: <1147255737.8179507.1355244114901.JavaMail.root@redhat.com>
Message-ID: <1563554956.8194515.1355246332643.JavaMail.root@redhat.com>

Hi,

Another datamodel cleanup patch is waiting for review:
http://gerrit.ovirt.org/9824

Thx,
Laszlo


From danken at redhat.com  Tue Dec 11 18:58:32 2012
From: danken at redhat.com (Dan Kenigsberg)
Date: Tue, 11 Dec 2012 20:58:32 +0200
Subject: [Engine-devel] Request power-user status for mom project in
	jenkins
In-Reply-To: <33068073.14181482.1355243047864.JavaMail.root@redhat.com>
References: <20121211161145.GB30988@aglitke>
	<33068073.14181482.1355243047864.JavaMail.root@redhat.com>
Message-ID: <20121211185832.GC19473@redhat.com>

On Tue, Dec 11, 2012 at 11:24:07AM -0500, Eyal Edri wrote:
> [adding engine-devel]
> 
> usually there is an official vote on each new power user for jenkins,
> and as long as there are no objections it is approved.
> 
> since i know you're a mom maintainer and familiar with jenkins, i vote +1.

I trust Adam (who is mom's maintainer), and trust that he is capable of
becoming familiar with jenkins without causing pain to other users.

+1


From smizrahi at redhat.com  Tue Dec 11 19:02:10 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Tue, 11 Dec 2012 14:02:10 -0500 (EST)
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <50C6A983.9090405@linux.vnet.ibm.com>
Message-ID: <549567196.17234243.1355252530303.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "Adam Litke" <agl at us.ibm.com>, "engine-devel" <engine-devel at ovirt.org>, "VDSM Project Development"
> <vdsm-devel at lists.fedorahosted.org>
> Sent: Monday, December 10, 2012 10:33:23 PM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> 2012-12-11 4:36, Saggi Mizrahi:
> >
> > ----- Original Message -----
> >> From: "Adam Litke" <agl at us.ibm.com>
> >> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> >> Cc: "Shu Ming" <shuming at linux.vnet.ibm.com>, "engine-devel"
> >> <engine-devel at ovirt.org>, "VDSM Project Development"
> >> <vdsm-devel at lists.fedorahosted.org>
> >> Sent: Monday, December 10, 2012 1:39:51 PM
> >> Subject: Re: [vdsm] RFC: New Storage API
> >>
> >> On Thu, Dec 06, 2012 at 11:52:01AM -0500, Saggi Mizrahi wrote:
> >>>
> >>> ----- Original Message -----
> >>>> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
> >>>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> >>>> Cc: "VDSM Project Development"
> >>>> <vdsm-devel at lists.fedorahosted.org>, "engine-devel"
> >>>> <engine-devel at ovirt.org>
> >>>> Sent: Thursday, December 6, 2012 11:02:02 AM
> >>>> Subject: Re: [vdsm] RFC: New Storage API
> >>>>
> >>>> Saggi,
> >>>>
> >>>> Thanks for sharing your thought and I get some comments below.
> >>>>
> >>>>
> >>>> Saggi Mizrahi:
> >>>>> I've been throwing a lot of bits out about the new storage API
> >>>>> and
> >>>>> I think it's time to talk a bit.
> >>>>> I will purposefully try and keep implementation details away
> >>>>> and
> >>>>> concentrate about how the API looks and how you use it.
> >>>>>
> >>>>> First major change is in terminology, there is no long a
> >>>>> storage
> >>>>> domain but a storage repository.
> >>>>> This change is done because so many things are already called
> >>>>> domain in the system and this will make things less confusing
> >>>>> for
> >>>>> new-commers with a libvirt background.
> >>>>>
> >>>>> One other changes is that repositories no longer have a UUID.
> >>>>> The UUID was only used in the pool members manifest and is no
> >>>>> longer needed.
> >>>>>
> >>>>>
> >>>>> connectStorageRepository(repoId, repoFormat,
> >>>>> connectionParameters={}):
> >>>>> repoId - is a transient name that will be used to refer to the
> >>>>> connected domain, it is not persisted and doesn't have to be
> >>>>> the
> >>>>> same across the cluster.
> >>>>> repoFormat - Similar to what used to be type (eg. localfs-1.0,
> >>>>> nfs-3.4, clvm-1.2).
> >>>>> connectionParameters - This is format specific and will used to
> >>>>> tell VDSM how to connect to the repo.
> >>>>
> >>>> Where does repoID come from? I think repoID doesn't exist before
> >>>> connectStorageRepository() return.  Isn't repoID a return value
> >>>> of
> >>>> connectStorageRepository()?
> >>> No, repoIDs are no longer part of the domain, they are just a
> >>> transient handle.
> >>> The user can put whatever it wants there as long as it isn't
> >>> already taken by another currently connected domain.
> >>>>> disconnectStorageRepository(self, repoId)
> >>>>>
> >>>>>
> >>>>> In the new API there are only images, some images are mutable
> >>>>> and
> >>>>> some are not.
> >>>>> mutable images are also called VirtualDisks
> >>>>> immutable images are also called Snapshots
> >>>>>
> >>>>> There are no explicit templates, you can create as many images
> >>>>> as
> >>>>> you want from any snapshot.
> >>>>>
> >>>>> There are 4 major image operations:
> >>>>>
> >>>>>
> >>>>> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> >>>>>                     userData={}, options={}):
> >>>>>
> >>>>> targetRepoId - ID of a connected repo where the disk will be
> >>>>> created
> >>>>> size - The size of the image you wish to create
> >>>>> baseSnapshotId - the ID of the snapshot you want the base the
> >>>>> new
> >>>>> virtual disk on
> >>>>> userData - optional data that will be attached to the new VD,
> >>>>> could
> >>>>> be anything that the user desires.
> >>>>> options - options to modify VDSMs default behavior
> >>>>>
> >>>>> returns the id of the new VD
> >>>> I think we will also need a function to check if a a VirtualDisk
> >>>> is
> >>>> based on a specific snapshot.
> >>>> Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
> >>> No, the design is that volume dependencies are an implementation
> >>> detail.
> >>> There is no reason for you to know that an image is physically a
> >>> snapshot of another.
> >>> Logical snapshots, template information, and any other
> >>> information
> >>> can be set by the user by using the userData field available for
> >>> every image.
> >> Statements like this make me start to worry about your userData
> >> concept.  It's a
> >> sign of a bad API if the user needs to invent a custom metadata
> >> scheme for
> >> itself.  This reminds me of the abomination that is the 'custom'
> >> property in the
> >> vm definition today.
> > In one sentence: If VDSM doesn't care about it, VDSM doesn't manage
> > it.
> >
> > userData being a "void*" is quite common and I don't understand why
> > you would thing it's a sign of a bad API.
> > Further more, giving the user choice about how to represent it's
> > own metadata and what fields it want to keep seems reasonable to
> > me.
> > Especially given the fact that VDSM never reads it.
> >
> > The reason we are pulling away from the current system of VDSM
> > understanding the extra data is that it makes that data tied to
> > VDSMs on disk format.
> > VDSM on disk format has to be very stable because of clusters with
> > multiple VDSM versions.
> > Further more, since this is actually manager data it has to be tied
> > to the manager backward compatibility lifetime as well.
> > Having it be opaque to VDSM ties it to only one, simpler, support
> > lifetime instead of two.
> 
> Making userData being opaque gives flexibilities to the management
> applications.  To me, opaque userDaa can have two types at least. The
> first is the userData for runtime only.  The second is the userData
> expected to be persisted into the metadata disk.  For the first type,
> the management applications can store their own data structures like
> temporary task states, VDSM query caches &etc. After the VDSM host is
> fenced, the userData goes away. For the second type, different
> management applications can have different formats of userData and
> persisting it to the VDSM metadata disk will corrupt each other in
> very
> high possibility.  You may argue that, all the management
> applications
> should reach a agreement before they use it as the second type.  But
> in
> practice, it is pretty hard to maintain.
We are talking only about the second type.
And yes, if not all clients agree about the format and don't check that they are modifying an image with someone else's userdata you will have problems.
This is the same problem you have if any program starts randomly messing with any other programs persistent data.
Users will have to agree on a format anyway if the want to use the same storage domains because they'll be sharing the same data anyway.
The thing is that if you are making anything other then a manager you should develop against the engine API.
If you are writing your own manager and are also sharing the host with another manager you should really not mess with other manager's repos.
That being said, agreeing on a set of fields and format to be able to have some sort of interoperability between managers is a good idea IMO.
> 
> 
> >
> > I guess you are implying that it will make it problematic for
> > multiple users to read userData left by another user because the
> > formats might not be compatible.
> > The solution is that all parties interested in using VDSM storage
> > agree on format, and common fields, and supportability, and all
> > the other things that choosing a supporting *something* entails.
> > This is, however, out of the scope of VDSM. When the time comes I
> > think how the userData blob is actually parsed and what fields it
> > keeps should be discussed on ovirt-devel or engine-devel.
> >
> > The crux of the issue is that VDSM manages only what it cares about
> > and the user can't modify directly.
> > This is done because everything we expose we commit to.
> > If you want any information persisted like:
> > - Human readable name (in whatever encoding)
> > - Is this a template or a snapshot
> > - What user owns this image
> >
> > You can just put it in the userData.
> > VDSM is not going to impose what encoding you use.
> > It's not going to decide if you represent your users as IDs or
> > names or ldap queries or Public Keys.
> > It's not going to decide if you have explicit templates or not.
> > It's not going to decide if you care what is the logical image
> > chain.
> > It's not going to decide anything that is out of it's scope.
> > No format is future proof, no selection of fields will be good for
> > any situation.
> > I'd much rather it be someone else's problem when any of them need
> > to be changed.
> > They have currently been VDSMs problem and it has been hell to
> > maintain.
> >
> >>>>> createSnapshot(targetRepoId, baseVirtualDiskId,
> >>>>>                  userData={}, options={}):
> >>>>> targetRepoId - The ID of a connected repo where the new
> >>>>> sanpshot
> >>>>> will be created and the original image exists as well.
> >>>>> size - The size of the image you wish to create
> >>>>> baseVirtualDisk - the ID of a mutable image (Virtual Disk) you
> >>>>> want
> >>>>> to snapshot
> >>>>> userData - optional data that will be attached to the new
> >>>>> Snapshot,
> >>>>> could be anything that the user desires.
> >>>>> options - options to modify VDSMs default behavior
> >>>>>
> >>>>> returns the id of the new Snapshot
> >>>>>
> >>>>> copyImage(targetRepoId, imageId, baseImageId=None, userData={},
> >>>>> options={})
> >>>>> targetRepoId - The ID of a connected repo where the new image
> >>>>> will
> >>>>> be created
> >>>>> imageId - The image you wish to copy
> >>>>> baseImageId - if specified, the new image will contain only the
> >>>>> diff between image and Id.
> >>>>>                 If None the new image will contain all the bits
> >>>>>                 of
> >>>>>                 image Id. This can be used to copy partial
> >>>>>                 parts
> >>>>>                 of
> >>>>>                 images for export.
> >>>>> userData - optional data that will be attached to the new
> >>>>> image,
> >>>>> could be anything that the user desires.
> >>>>> options - options to modify VDSMs default behavior
> >>>> Does this function mean that we can copy the image from one
> >>>> repository
> >>>> to another repository? Does it cover the semantics of storage
> >>>> migration,
> >>>> storage backup, storage incremental backup?
> >>> Yes, the main purpose is copying to another repo. and you can
> >>> even
> >>> do incremental backups.
> >>> Also the following flow
> >>> 1. Run a VM using imageA
> >>> 2. write to disk
> >>> 3. Stop VM
> >>> 4. copy imageA to repoB
> >>> 5. Run a VM using imageA again
> >>> 6. Write to disk
> >>> 7. Stop VM
> >>> 8. Copy imageA again basing it of imageA_copy1 on repoB creating
> >>> a
> >>> diff on repo diff without snapshotting the original image.
> >>>
> >>>>> return the Id of the new image. In case of copying an immutable
> >>>>> image the ID will be identical to the original image as they
> >>>>> contain the same data. However the user should not assume that
> >>>>> and
> >>>>> always use the value returned from the method.
> >>>>>
> >>>>> removeImage(repositoryId, imageId, options={}):
> >>>>> repositoryId - The ID of a connected repo where the image to
> >>>>> delete
> >>>>> resides
> >>>>> imageId - The id of the image you wish to delete.
> >>>>>
> >>>>>
> >>>>> ----
> >>>>> getImageStatus(repositoryId, imageId)
> >>>>> repositoryId - The ID of a connected repo where the image to
> >>>>> check
> >>>>> resides
> >>>>> imageId - The id of the image you wish to check.
> >>>>>
> >>>>> All operations return once the operations has been committed to
> >>>>> disk NOT when the operation actually completes.
> >>>>> This is done so that:
> >>>>> - operation come to a stable state as quickly as possible.
> >>>>> - In case where there is an SDM, only small portion of the
> >>>>> operation actually needs to be performed on the SDM host.
> >>>>> - No matter how many times the operation fails and on how many
> >>>>> hosts, you can always resume the operation and choose when to
> >>>>> do
> >>>>> it.
> >>>>> - You can stop an operation at any time and remove the
> >>>>> resulting
> >>>>> object making a distinction between "stop because the host is
> >>>>> overloaded" to "I don't want that image"
> >>>>>
> >>>>> This means that after calling any operation that creates a new
> >>>>> image the user must then call getImageStatus() to check what is
> >>>>> the status of the image.
> >>>>> The status of the image can be either optimized, degraded, or
> >>>>> broken.
> >>>>> "Optimized" means that the image is available and you can run
> >>>>> VMs
> >>>>> of it.
> >>>>> "Degraded" means that the image is available and will run VMs
> >>>>> but
> >>>>> it might be a better way VDSM can represent the underlying
> >>>>> data.
> >>>> What does the "represent" mean here?
> >>> Anything, but mostly image formate RAW\QCOW2 when performance
> >>> strategy has been selected.
> >>>>> "Broken" means that the image can't be used at the moment,
> >>>>> probably
> >>>>> because not all the data has been set up on the volume.
> >>>>>
> >>>>> Apart from that VDSM will also return the last persisted status
> >>>>> information which will conatin
> >>>>> hostID - the last host to try and optimize of fix the image
> >>>> Any host can optimize the image? No need to be SDM?
> >>> On anything but lvm based block domains there will not even be an
> >>> SDM.
> >>> On SDM based domains we will try as hard as we can to have as
> >>> many
> >>> operations executable on any host.
> >>>>> stage - X/Y (eg. 1/10) the last persisted stage of the fix.
> >>>>> percent_complete - -1 or 0-100, the last persisted completion
> >>>>> percentage of the aforementioned stage. -1 means that no
> >>>>> progress
> >>>>> is available for that operation.
> >>>>> last_error - This will only be filled if the operation failed
> >>>>> because of something other then IO or a VDSM crash for obvious
> >>>>> reasons.
> >>>>>                It will usually be set if the task was manually
> >>>>>                stopped
> >>>>>
> >>>>> The user can either be satisfied with that information or as
> >>>>> the
> >>>>> host specified in host ID if it is still working on that image
> >>>>> by
> >>>>> checking it's running tasks.
> >>>> So we need a function to know what tasks are running on the
> >>>> image
> >>> getImageStatus()
> >>>>> checkStorageRepository(self, repositoryId, options={}):
> >>>>> A method to go over a storage repository and scan for any
> >>>>> existing
> >>>>> problems. This includes degraded\broken images and deleted
> >>>>> images
> >>>>> that have no yet been physically deleted\merged.
> >>>>> It returns a list of Fix objects.
> >>>>> Fix objects come in 4 types:
> >>>>> clean - cleans data, run them to get more space.
> >>>>> optimize - run them to optimize a degraded image
> >>>>> merge - Merges two images together. Doing this sometimes
> >>>>>           makes more images ready optimizing or cleaning.
> >>>>>           The reason it is different from optimize is that
> >>>>>           unmerged images are considered optimized.
> >>>>> mend - mends a broken image
> >>>>>
> >>>>> The user can read these types and prioritize fixes. Fixes also
> >>>>> contain opaque FIX data and they should be sent as received to
> >>>>> fixStorageRepository(self, repositoryId, fix, options={}):
> >>>>>
> >>>>> That will start a fix operation.
> >>>>>
> >>>>>
> >>>>> All major operations automatically start the appropriate "Fix"
> >>>>> to
> >>>>> bring the created object to an optimize\degraded state (the one
> >>>>> that is quicker) unless one of the options is
> >>>>> AutoFix=False. This is only useful for repos that might not be
> >>>>> able
> >>>>> to create volumes on all hosts (SDM) but would like to have the
> >>>>> actual IO distributed in the cluster.
> >>>>>
> >>>>> Other common options is the strategy option:
> >>>>> It has currently 2 possible values
> >>>>> space and performance - In case VDSM has 2 ways of completing
> >>>>> the
> >>>>> same operation it will tell it to value one over the other. For
> >>>>> example, whether to copy all the data or just create a qcow
> >>>>> based
> >>>>> of a snapshot.
> >>>>> The default is space.
> >>>>>
> >>>>> You might have also noticed that it is never explicitly
> >>>>> specified
> >>>>> where to look for existing images. This is done purposefully,
> >>>>> VDSM
> >>>>> will always look in all connected repositories for existing
> >>>>> objects.
> >>>>> For very large setups this might be problematic. To mitigate
> >>>>> the
> >>>>> problem you have these options:
> >>>>> participatingRepositories=[repoId, ...] which tell VDSM to
> >>>>> narrow
> >>>>> the search to just these repositories
> >>>>> and
> >>>>> imageHints={imgId: repoId} which will force VDSM to look for
> >>>>> those
> >>>>> image ID just in those repositories and fail if it doesn't find
> >>>>> them there.
> >>>>> _______________________________________________
> >>>>> vdsm-devel mailing list
> >>>>> vdsm-devel at lists.fedorahosted.org
> >>>>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> >>>>
> >>>> --
> >>>> ---
> >>>> ?? Shu Ming
> >>>> Open Virtualization Engineerning; CSTL, IBM Corp.
> >>>> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com
> >>>> or
> >>>> shuming at linux.vnet.ibm.com
> >>>> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> >>>> District, Beijing 100193, PRC
> >>>>
> >>>>
> >>>>
> >>> _______________________________________________
> >>> vdsm-devel mailing list
> >>> vdsm-devel at lists.fedorahosted.org
> >>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> >> --
> >> Adam Litke <agl at us.ibm.com>
> >> IBM Linux Technology Center
> >>
> >>
> 
> 
> --
> ---
> ?? Shu Ming
> Open Virtualization Engineerning; CSTL, IBM Corp.
> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com or
> shuming at linux.vnet.ibm.com
> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> District, Beijing 100193, PRC
> 
> 
>


From mpastern at redhat.com  Wed Dec 12 10:56:35 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Wed, 12 Dec 2012 12:56:35 +0200
Subject: [Engine-devel] Failed to start service
	jboss.deployment.subunit."engine.ear"
Message-ID: <50C862E3.6060709@redhat.com>


any clue what it might be? (latest sources + db)

2012-12-12 12:51:27,658 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-7) Start time: 12/12/12 12:51 PM
2012-12-12 12:51:27,685 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-7) MSC00001: Failed to start service
jboss.deployment.subunit."engine.ear"."engine-bll.jar".component.Backend.START: org.jboss.msc.service.StartException in service
jboss.deployment.subunit."engine.ear"."engine-bll.jar".component.Backend.START: Failed to start service
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1767) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_20]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_20]
        at java.lang.Thread.run(Thread.java:636) [rt.jar:1.6.0_20]
Caused by: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance
        at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:163)
        at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85)
        at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:116)
        at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:130)
        at org.jboss.as.ee.component.ComponentStartService.start(ComponentStartService.java:44)
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
        ... 3 more
Caused by: javax.ejb.EJBException: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [select * from  get_all_commands()]; nested
exception is org.postgresql.util.PSQLException: ERROR: function get_all_commands() does not exist
  Hint: No function matches the given name and argument types. You might need to add explicit type casts.
  Position: 16


...


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From lhornyak at redhat.com  Wed Dec 12 11:02:01 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Wed, 12 Dec 2012 06:02:01 -0500 (EST)
Subject: [Engine-devel] Failed to start
	service	jboss.deployment.subunit."engine.ear"
In-Reply-To: <50C862E3.6060709@redhat.com>
Message-ID: <309505454.8535710.1355310121574.JavaMail.root@redhat.com>

Looks like your DB is destroyed, or JBoss is connecting to the wrong DB.

----- Original Message -----
> From: "Michael Pasternak" <mpastern at redhat.com>
> To: "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 12, 2012 11:56:35 AM
> Subject: [Engine-devel] Failed to start service	jboss.deployment.subunit."engine.ear"
> 
> 
> any clue what it might be? (latest sources + db)
> 
> 2012-12-12 12:51:27,658 INFO  [org.ovirt.engine.core.bll.Backend]
> (MSC service thread 1-7) Start time: 12/12/12 12:51 PM
> 2012-12-12 12:51:27,685 ERROR [org.jboss.msc.service.fail] (MSC
> service thread 1-7) MSC00001: Failed to start service
> jboss.deployment.subunit."engine.ear"."engine-bll.jar".component.Backend.START:
> org.jboss.msc.service.StartException in service
> jboss.deployment.subunit."engine.ear"."engine-bll.jar".component.Backend.START:
> Failed to start service
>         at
>         org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1767)
>         [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
>         at
>         java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         [rt.jar:1.6.0_20]
>         at
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         [rt.jar:1.6.0_20]
>         at java.lang.Thread.run(Thread.java:636) [rt.jar:1.6.0_20]
> Caused by: java.lang.IllegalStateException: JBAS011048: Failed to
> construct component instance
>         at
>         org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:163)
>         at
>         org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85)
>         at
>         org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:116)
>         at
>         org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:130)
>         at
>         org.jboss.as.ee.component.ComponentStartService.start(ComponentStartService.java:44)
>         at
>         org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811)
>         [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
>         at
>         org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746)
>         [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
>         ... 3 more
> Caused by: javax.ejb.EJBException:
> org.springframework.jdbc.BadSqlGrammarException:
> PreparedStatementCallback; bad SQL grammar [select * from
>  get_all_commands()]; nested
> exception is org.postgresql.util.PSQLException: ERROR: function
> get_all_commands() does not exist
>   Hint: No function matches the given name and argument types. You
>   might need to add explicit type casts.
>   Position: 16
> 
> 
> ...
> 
> 
> --
> 
> Michael Pasternak
> RedHat, ENG-Virtualization R&D
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From vszocs at redhat.com  Wed Dec 12 11:51:46 2012
From: vszocs at redhat.com (Vojtech Szocs)
Date: Wed, 12 Dec 2012 06:51:46 -0500 (EST)
Subject: [Engine-devel] UI Plugins: PoC patch revision 7 is here
In-Reply-To: <50B1D68F.9070505@redhat.com>
Message-ID: <1223886611.17503948.1355313106332.JavaMail.root@redhat.com>

Hi, sorry for my late response,

>> Our first iteration could be:
>> a1, rewrite UiCommon (UI business logic) layer to work with API types
>> a2, write adapter (e.g. modify Frontend class) between UiCommon using API types and Generic API (GWT RPC) using internal entities
>
> does this mean converting three times?

Not sure what you mean, but here's my idea of the first iteration - what would happen on frontend side (GWT):
1. UiCommon, using API types, invokes query/command via RPC bridge (org.ovirt.engine.ui.frontend.Frontend class)
2. Frontend class, still using Generic API (GWT RPC), does the actual query/command invocation
3. Frontend class receives query/command result and translates it from BE type to API type (converter callback)

My idea was to isolate BE type usage into Frontend class, so that when we decide to use REST API, we just need to update our RPC bridge (Frontend class).

>> Our second iteration could be:
>> b1, remove adapter from step a2, drop Generic API (GWT RPC) usage
>> b2, write API type <-> JSON mapper, possibly using some existing framework
>
> do you mean talking with api in JSON?

Yes, that was my original idea. I'm not against using XML when talking with REST API either. GWT supports parsing both JSON and XML.

Vojtech


----- Original Message -----
From: "Michael Pasternak" <mpastern at redhat.com>
To: "Vojtech Szocs" <vszocs at redhat.com>
Cc: "Itamar Heim" <iheim at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
Sent: Sunday, November 25, 2012 9:27:59 AM
Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here

On 11/20/2012 12:57 PM, Vojtech Szocs wrote:
> Hi Michael,
> 
>> if we plan moving UI on top of API, you should be:
>>
>> 1. importing restapi-types project
>> 2. writing intermediate layer to translate BE entities to API's using #1
>> 3. using public entities from #2
>>
>> this way your future migration to API (instead of native BE) will be much
>> more easier.
> 
> Indeed, this is very useful for GWT RPC -> REST API transition in general, many thanks for pointing this out, Michael. As you suggest, we can use restapi-types mappers to translate between internal entities and API types.
> 
> Our first iteration could be:
> a1, rewrite UiCommon (UI business logic) layer to work with API types
> a2, write adapter (e.g. modify Frontend class) between UiCommon using API types and Generic API (GWT RPC) using internal entities

does this mean converting three times?

> 
> Our second iteration could be:
> b1, remove adapter from step a2, drop Generic API (GWT RPC) usage
> b2, write API type <-> JSON mapper, possibly using some existing framework

do you mean talking with api in JSON?

> b3, write RPC layer that talks REST API with Engine
> 
> However, in order to use REST API types on client (GWT), we need their source code. restapi-interface-definition uses Maven JAXB plugin to generate API types from XSD schema (src/main/resources/api.xsd). On client, we need restapi-definition-<version>-sources.jar that includes those generated API types (target/generated-sources/xjc).

> 
> As for UI plugins (short term), we can just use restapi-types mappers and implement step b2, (API type -> JSON mapper).
> 
> Thanks,
> Vojtech
> 
> 
> ----- Original Message -----
> From: "Michael Pasternak" <mpastern at redhat.com>
> To: "Vojtech Szocs" <vszocs at redhat.com>
> Cc: "Itamar Heim" <iheim at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Tuesday, November 20, 2012 9:48:27 AM
> Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here
> 
> On 11/20/2012 12:13 AM, Itamar Heim wrote:
>> On 11/19/2012 02:07 PM, Vojtech Szocs wrote:
>>> Hi Itamar,
>>>
>>> UI plugin infrastructure translates internal business entities into JSON-like representations and passes those representations to UI plugins. (Internal entities are NOT
>>> exposed to UI plugins directly.)
>>>
>>> Currently, all entities supported by UI plugin infrastructure (as per org.ovirt.engine.ui.webadmin.plugin.entity.EntityType) are transformed into following representation:
>>>
>>> { entityId: "[BusinessEntityGuidAsString]" }
>>>
>>> For example, a VM entity with entity ID "vm123" will translate to:
>>>
>>> { entityId: "vm123" }
>>>
>>> Translation is currently based on org.ovirt.engine.core.common.businessentities.BusinessEntity interface, like so: "BusinessEntity<? extends NGuid>" (we expect ID type
>>> parameter to be NGuid-compatible). However, I've found that there are some entities (like Pool - org.ovirt.engine.core.common.businessentities.vm_pools) that don't
>>> implement BusinessEntity interface.
>>
>> ok, so we only pass the ID for now. good.
>>
>>>
>>> Quick question to backend folks - IIRC all entities extend org.ovirt.engine.core.common.businessentities.IVdcQueryable, but not all entities implement
>>> org.ovirt.engine.core.common.businessentities.BusinessEntity interface. What is the precise relation between IVdcQueryable and BusinessEntity?
>>>
>>> As for UI plugins, currently all entities get translated to above mentioned basic JSON-like representation. You can see the relevant code in
>>> org.ovirt.engine.ui.webadmin.plugin.entity.BaseEntity.from() static method. There's a TODO that says "make this class [BaseEntity] abstract and create specific entity for
>>> each EntityType" - this means we are planning to extend the above mentioned basic JSON-like representation for different entity types.
>>>
>>> For example, for a VM entity we might do:
>>>
>>> { entityId: "[BusinessEntityGuidAsString]", osType: "[VmOsType]" }
>>
>> just make sure the entity matches the REST API entity.
>> (which probably means entityId should be changed to id?)
> 
> if we plan moving UI on top of API, you should be:
> 
> 1. importing restapi-types project
> 2. writing intermediate layer to translate BE entities to API's using #1
> 3. using public entities from #2
> 
> this way your future migration to API (instead of native BE) will be much
> more easier.
> 
>>
>>>
>>> Vojtech
>>>
>>>
>>> ----- Original Message -----
>>> From: "Itamar Heim" <iheim at redhat.com>
>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>> Sent: Friday, November 16, 2012 5:24:52 PM
>>> Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here
>>>
>>> On 11/16/2012 06:08 PM, Vojtech Szocs wrote:
>>>>> is there a clear list of all APIs supported now?
>>>>
>>>> Not yet, unfortunately, this should be part of "for plugin developers" wiki that is planned to be written in upcoming weeks.
>>>
>>> i just wanted to review how we solved not using internal entities as
>>> part of the API
>>>
>>
>>
> 
> 


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From vszocs at redhat.com  Wed Dec 12 12:27:41 2012
From: vszocs at redhat.com (Vojtech Szocs)
Date: Wed, 12 Dec 2012 07:27:41 -0500 (EST)
Subject: [Engine-devel] UI plugins patches merged,
	moving out of proof-of-concept stage
In-Reply-To: <800242646.17508631.1355314071136.JavaMail.root@redhat.com>
Message-ID: <631816160.17519448.1355315261929.JavaMail.root@redhat.com>

Hi everyone,

I'm happy to announce that oVirt UI plugins feature just moved out of proof-of-concept stage, with relevant patches [1,2] merged into ovirt-engine git repository.

I'll be working on related documentation (wiki) in near future, along with contributing some sample plugins into samples-uiplugins [3] git repository.

I'd like to thank everyone involved in pushing this feature forward. I hope we'll see some interesting plugins soon!

Regards,
Vojtech

[1] http://gerrit.ovirt.org/#/c/8120/
[2] http://gerrit.ovirt.org/#/c/9250/
[3] git://gerrit.ovirt.org/samples-uiplugins


From mpastern at redhat.com  Wed Dec 12 12:33:56 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Wed, 12 Dec 2012 14:33:56 +0200
Subject: [Engine-devel] UI Plugins: PoC patch revision 7 is here
In-Reply-To: <1223886611.17503948.1355313106332.JavaMail.root@redhat.com>
References: <1223886611.17503948.1355313106332.JavaMail.root@redhat.com>
Message-ID: <50C879B4.7070409@redhat.com>

On 12/12/2012 01:51 PM, Vojtech Szocs wrote:
> Hi, sorry for my late response,
> 
>>> Our first iteration could be:
>>> a1, rewrite UiCommon (UI business logic) layer to work with API types
>>> a2, write adapter (e.g. modify Frontend class) between UiCommon using API types and Generic API (GWT RPC) using internal entities
>>
>> does this mean converting three times?
> 
> Not sure what you mean, but here's my idea of the first iteration - what would happen on frontend side (GWT):
> 1. UiCommon, using API types, invokes query/command via RPC bridge (org.ovirt.engine.ui.frontend.Frontend class)
> 2. Frontend class, still using Generic API (GWT RPC), does the actual query/command invocation
> 3. Frontend class receives query/command result and translates it from BE type to API type (converter callback)
> 
> My idea was to isolate BE type usage into Frontend class, so that when we decide to use REST API, we just need to update our RPC bridge (Frontend class).
> 
>>> Our second iteration could be:
>>> b1, remove adapter from step a2, drop Generic API (GWT RPC) usage
>>> b2, write API type <-> JSON mapper, possibly using some existing framework
>>
>> do you mean talking with api in JSON?
> 
> Yes, that was my original idea. I'm not against using XML when talking with REST API either. GWT supports parsing both JSON and XML.

infra. is there, but officially we not supporting json yet,
(resteasy-json-provider has issues with jaxb, - we waiting for fix)

> 
> Vojtech
> 
> 
> ----- Original Message -----
> From: "Michael Pasternak" <mpastern at redhat.com>
> To: "Vojtech Szocs" <vszocs at redhat.com>
> Cc: "Itamar Heim" <iheim at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Sunday, November 25, 2012 9:27:59 AM
> Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here
> 
> On 11/20/2012 12:57 PM, Vojtech Szocs wrote:
>> Hi Michael,
>>
>>> if we plan moving UI on top of API, you should be:
>>>
>>> 1. importing restapi-types project
>>> 2. writing intermediate layer to translate BE entities to API's using #1
>>> 3. using public entities from #2
>>>
>>> this way your future migration to API (instead of native BE) will be much
>>> more easier.
>>
>> Indeed, this is very useful for GWT RPC -> REST API transition in general, many thanks for pointing this out, Michael. As you suggest, we can use restapi-types mappers to translate between internal entities and API types.
>>
>> Our first iteration could be:
>> a1, rewrite UiCommon (UI business logic) layer to work with API types
>> a2, write adapter (e.g. modify Frontend class) between UiCommon using API types and Generic API (GWT RPC) using internal entities
> 
> does this mean converting three times?
> 
>>
>> Our second iteration could be:
>> b1, remove adapter from step a2, drop Generic API (GWT RPC) usage
>> b2, write API type <-> JSON mapper, possibly using some existing framework
> 
> do you mean talking with api in JSON?
> 
>> b3, write RPC layer that talks REST API with Engine
>>
>> However, in order to use REST API types on client (GWT), we need their source code. restapi-interface-definition uses Maven JAXB plugin to generate API types from XSD schema (src/main/resources/api.xsd). On client, we need restapi-definition-<version>-sources.jar that includes those generated API types (target/generated-sources/xjc).
> 
>>
>> As for UI plugins (short term), we can just use restapi-types mappers and implement step b2, (API type -> JSON mapper).
>>
>> Thanks,
>> Vojtech
>>
>>
>> ----- Original Message -----
>> From: "Michael Pasternak" <mpastern at redhat.com>
>> To: "Vojtech Szocs" <vszocs at redhat.com>
>> Cc: "Itamar Heim" <iheim at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
>> Sent: Tuesday, November 20, 2012 9:48:27 AM
>> Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here
>>
>> On 11/20/2012 12:13 AM, Itamar Heim wrote:
>>> On 11/19/2012 02:07 PM, Vojtech Szocs wrote:
>>>> Hi Itamar,
>>>>
>>>> UI plugin infrastructure translates internal business entities into JSON-like representations and passes those representations to UI plugins. (Internal entities are NOT
>>>> exposed to UI plugins directly.)
>>>>
>>>> Currently, all entities supported by UI plugin infrastructure (as per org.ovirt.engine.ui.webadmin.plugin.entity.EntityType) are transformed into following representation:
>>>>
>>>> { entityId: "[BusinessEntityGuidAsString]" }
>>>>
>>>> For example, a VM entity with entity ID "vm123" will translate to:
>>>>
>>>> { entityId: "vm123" }
>>>>
>>>> Translation is currently based on org.ovirt.engine.core.common.businessentities.BusinessEntity interface, like so: "BusinessEntity<? extends NGuid>" (we expect ID type
>>>> parameter to be NGuid-compatible). However, I've found that there are some entities (like Pool - org.ovirt.engine.core.common.businessentities.vm_pools) that don't
>>>> implement BusinessEntity interface.
>>>
>>> ok, so we only pass the ID for now. good.
>>>
>>>>
>>>> Quick question to backend folks - IIRC all entities extend org.ovirt.engine.core.common.businessentities.IVdcQueryable, but not all entities implement
>>>> org.ovirt.engine.core.common.businessentities.BusinessEntity interface. What is the precise relation between IVdcQueryable and BusinessEntity?
>>>>
>>>> As for UI plugins, currently all entities get translated to above mentioned basic JSON-like representation. You can see the relevant code in
>>>> org.ovirt.engine.ui.webadmin.plugin.entity.BaseEntity.from() static method. There's a TODO that says "make this class [BaseEntity] abstract and create specific entity for
>>>> each EntityType" - this means we are planning to extend the above mentioned basic JSON-like representation for different entity types.
>>>>
>>>> For example, for a VM entity we might do:
>>>>
>>>> { entityId: "[BusinessEntityGuidAsString]", osType: "[VmOsType]" }
>>>
>>> just make sure the entity matches the REST API entity.
>>> (which probably means entityId should be changed to id?)
>>
>> if we plan moving UI on top of API, you should be:
>>
>> 1. importing restapi-types project
>> 2. writing intermediate layer to translate BE entities to API's using #1
>> 3. using public entities from #2
>>
>> this way your future migration to API (instead of native BE) will be much
>> more easier.
>>
>>>
>>>>
>>>> Vojtech
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "Itamar Heim" <iheim at redhat.com>
>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>> Sent: Friday, November 16, 2012 5:24:52 PM
>>>> Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here
>>>>
>>>> On 11/16/2012 06:08 PM, Vojtech Szocs wrote:
>>>>>> is there a clear list of all APIs supported now?
>>>>>
>>>>> Not yet, unfortunately, this should be part of "for plugin developers" wiki that is planned to be written in upcoming weeks.
>>>>
>>>> i just wanted to review how we solved not using internal entities as
>>>> part of the API
>>>>
>>>
>>>
>>
>>
> 
> 


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From yzaslavs at redhat.com  Wed Dec 12 12:38:55 2012
From: yzaslavs at redhat.com (Yair Zaslavsky)
Date: Wed, 12 Dec 2012 07:38:55 -0500 (EST)
Subject: [Engine-devel] upgrade script
Message-ID: <1640142833.46547673.1355315935689.JavaMail.root@redhat.com>

Hi all,
Please pay attention - when you send patches with upgrade script the prefix should be

03_02_XXXX_

and not

03_01_XXXX_


For example - this script was originally 03_01_XXXX once, and it's in correct

backend/manager/dbscripts/upgrade/03_02_0020_add_libvirt_version_col.sql


Yair

	 


From mpastern at redhat.com  Wed Dec 12 12:58:02 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Wed, 12 Dec 2012 14:58:02 +0200
Subject: [Engine-devel] [SOLVED] Re: Failed to start
	service	jboss.deployment.subunit."engine.ear"
In-Reply-To: <309505454.8535710.1355310121574.JavaMail.root@redhat.com>
References: <309505454.8535710.1355310121574.JavaMail.root@redhat.com>
Message-ID: <50C87F5A.6080303@redhat.com>


turns out there were few issues with dbscripts, commits [1] fixed that.

[1] 30d68953b89b5b0a78564075acb9f69f25c4ffc1, 4fe8562e24008c415dda2e046596257586127cf2

On 12/12/2012 01:02 PM, Laszlo Hornyak wrote:
> Looks like your DB is destroyed, or JBoss is connecting to the wrong DB.
> 
> ----- Original Message -----
>> From: "Michael Pasternak" <mpastern at redhat.com>
>> To: "engine-devel" <engine-devel at ovirt.org>
>> Sent: Wednesday, December 12, 2012 11:56:35 AM
>> Subject: [Engine-devel] Failed to start service	jboss.deployment.subunit."engine.ear"
>>
>>
>> any clue what it might be? (latest sources + db)
>>
>> 2012-12-12 12:51:27,658 INFO  [org.ovirt.engine.core.bll.Backend]
>> (MSC service thread 1-7) Start time: 12/12/12 12:51 PM
>> 2012-12-12 12:51:27,685 ERROR [org.jboss.msc.service.fail] (MSC
>> service thread 1-7) MSC00001: Failed to start service
>> jboss.deployment.subunit."engine.ear"."engine-bll.jar".component.Backend.START:
>> org.jboss.msc.service.StartException in service
>> jboss.deployment.subunit."engine.ear"."engine-bll.jar".component.Backend.START:
>> Failed to start service
>>         at
>>         org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1767)
>>         [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
>>         at
>>         java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>         [rt.jar:1.6.0_20]
>>         at
>>         java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>         [rt.jar:1.6.0_20]
>>         at java.lang.Thread.run(Thread.java:636) [rt.jar:1.6.0_20]
>> Caused by: java.lang.IllegalStateException: JBAS011048: Failed to
>> construct component instance
>>         at
>>         org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:163)
>>         at
>>         org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85)
>>         at
>>         org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:116)
>>         at
>>         org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:130)
>>         at
>>         org.jboss.as.ee.component.ComponentStartService.start(ComponentStartService.java:44)
>>         at
>>         org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811)
>>         [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
>>         at
>>         org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746)
>>         [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
>>         ... 3 more
>> Caused by: javax.ejb.EJBException:
>> org.springframework.jdbc.BadSqlGrammarException:
>> PreparedStatementCallback; bad SQL grammar [select * from
>>  get_all_commands()]; nested
>> exception is org.postgresql.util.PSQLException: ERROR: function
>> get_all_commands() does not exist
>>   Hint: No function matches the given name and argument types. You
>>   might need to add explicit type casts.
>>   Position: 16
>>
>>
>> ...
>>
>>
>> --
>>
>> Michael Pasternak
>> RedHat, ENG-Virtualization R&D
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>>


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From vszocs at redhat.com  Wed Dec 12 12:57:11 2012
From: vszocs at redhat.com (Vojtech Szocs)
Date: Wed, 12 Dec 2012 07:57:11 -0500 (EST)
Subject: [Engine-devel] UI plugins - next steps
In-Reply-To: <16148881.17522063.1355315888388.JavaMail.root@redhat.com>
Message-ID: <413127613.17526084.1355317031405.JavaMail.root@redhat.com>

Hi guys,

here's my list of tasks that didn't make it into UI plugins yet. These tasks should be revisited and implemented in future patches. The list isn't sorted in any way, so feel free to comment or highlight specific tasks to raise their priority, or add your own tasks/ideas.

1. Pass proper (restapi-definition) entities to UI plugins, instead of simple "{entityId:<guidAsString>}" objects
   - using restapi-types mappers to map backend business entities to restapi-definition entities (Java POJOs generated from api.xsd via JAXB)
   - exporting restapi-definition entities for use with JavaScript (UI plugins), e.g. using gwt-exporter [1] (alternatively, we could use GWT deferred binding and do this ourselves)

2. showDialog API function should integrate with WebAdmin dialog infrastructure
   - shouldn't be too hard, need to create custom dialog PresenterWidget/View and bind it as non-singleton (just like other popups in oVirt GUI)

3. Extend existing WebAdmin dialogs - prototype Doron's proposal to extend Edit Cluster Policy dialog with custom options that would be passed to backend

4. Introduce form-based tabs - custom tab that would display key/value pairs organized in columns, just like in standard "General" sub tabs

5. Introduce table-based tabs - custom tab that would display grid (table) just like in standard main tabs, with API to add columns, set data, etc.

6. Improve api.addMainTabActionButton to allow different button vs. context menu item representations (only button, only menu item, both)

7. Add api.currentUserName and api.currentUserId API functions (these would return same data as with UserLogin event parameters)

8. Minor things: consider renaming UiInit to PluginInit (one-time plugin initialization, called only once, can do non-UI init stuff there as well), consider renaming api.ready to api.initialize (plugin indicates that infrastructure can proceed with initializing itself)

Regards,
Vojtech

[1] http://code.google.com/p/gwt-exporter/wiki/GettingStarted


From amureini at redhat.com  Wed Dec 12 13:08:10 2012
From: amureini at redhat.com (Allon Mureinik)
Date: Wed, 12 Dec 2012 08:08:10 -0500 (EST)
Subject: [Engine-devel] UI plugins patches merged,
 moving out of proof-of-concept stage
In-Reply-To: <631816160.17519448.1355315261929.JavaMail.root@redhat.com>
Message-ID: <1997012895.14709828.1355317690626.JavaMail.root@redhat.com>

Congrats!

----- Original Message -----
> From: "Vojtech Szocs" <vszocs at redhat.com>
> To: "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 12, 2012 2:27:41 PM
> Subject: [Engine-devel] UI plugins patches merged,	moving out of proof-of-concept stage
> 
> Hi everyone,
> 
> I'm happy to announce that oVirt UI plugins feature just moved out of
> proof-of-concept stage, with relevant patches [1,2] merged into
> ovirt-engine git repository.
> 
> I'll be working on related documentation (wiki) in near future, along
> with contributing some sample plugins into samples-uiplugins [3] git
> repository.
> 
> I'd like to thank everyone involved in pushing this feature forward.
> I hope we'll see some interesting plugins soon!
> 
> Regards,
> Vojtech
> 
> [1] http://gerrit.ovirt.org/#/c/8120/
> [2] http://gerrit.ovirt.org/#/c/9250/
> [3] git://gerrit.ovirt.org/samples-uiplugins
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From vszocs at redhat.com  Wed Dec 12 13:10:53 2012
From: vszocs at redhat.com (Vojtech Szocs)
Date: Wed, 12 Dec 2012 08:10:53 -0500 (EST)
Subject: [Engine-devel] UI Plugins: PoC patch revision 7 is here
In-Reply-To: <50C879B4.7070409@redhat.com>
Message-ID: <1005426847.17530385.1355317853426.JavaMail.root@redhat.com>

Thanks Michael, in that case we can just use XML when talking with REST API.

Also, one idea worth implementing in future is to have some sort of debug console on GUI (I've seen this idea implemented in other applications). When activated, users could see REST API XML communication (request/response) between WebAdmin and Engine, which could be useful for someone interested in writing his own REST API client.

Vojtech


----- Original Message -----
From: "Michael Pasternak" <mpastern at redhat.com>
To: "Vojtech Szocs" <vszocs at redhat.com>
Cc: "Itamar Heim" <iheim at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
Sent: Wednesday, December 12, 2012 1:33:56 PM
Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here

On 12/12/2012 01:51 PM, Vojtech Szocs wrote:
> Hi, sorry for my late response,
> 
>>> Our first iteration could be:
>>> a1, rewrite UiCommon (UI business logic) layer to work with API types
>>> a2, write adapter (e.g. modify Frontend class) between UiCommon using API types and Generic API (GWT RPC) using internal entities
>>
>> does this mean converting three times?
> 
> Not sure what you mean, but here's my idea of the first iteration - what would happen on frontend side (GWT):
> 1. UiCommon, using API types, invokes query/command via RPC bridge (org.ovirt.engine.ui.frontend.Frontend class)
> 2. Frontend class, still using Generic API (GWT RPC), does the actual query/command invocation
> 3. Frontend class receives query/command result and translates it from BE type to API type (converter callback)
> 
> My idea was to isolate BE type usage into Frontend class, so that when we decide to use REST API, we just need to update our RPC bridge (Frontend class).
> 
>>> Our second iteration could be:
>>> b1, remove adapter from step a2, drop Generic API (GWT RPC) usage
>>> b2, write API type <-> JSON mapper, possibly using some existing framework
>>
>> do you mean talking with api in JSON?
> 
> Yes, that was my original idea. I'm not against using XML when talking with REST API either. GWT supports parsing both JSON and XML.

infra. is there, but officially we not supporting json yet,
(resteasy-json-provider has issues with jaxb, - we waiting for fix)

> 
> Vojtech
> 
> 
> ----- Original Message -----
> From: "Michael Pasternak" <mpastern at redhat.com>
> To: "Vojtech Szocs" <vszocs at redhat.com>
> Cc: "Itamar Heim" <iheim at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Sunday, November 25, 2012 9:27:59 AM
> Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here
> 
> On 11/20/2012 12:57 PM, Vojtech Szocs wrote:
>> Hi Michael,
>>
>>> if we plan moving UI on top of API, you should be:
>>>
>>> 1. importing restapi-types project
>>> 2. writing intermediate layer to translate BE entities to API's using #1
>>> 3. using public entities from #2
>>>
>>> this way your future migration to API (instead of native BE) will be much
>>> more easier.
>>
>> Indeed, this is very useful for GWT RPC -> REST API transition in general, many thanks for pointing this out, Michael. As you suggest, we can use restapi-types mappers to translate between internal entities and API types.
>>
>> Our first iteration could be:
>> a1, rewrite UiCommon (UI business logic) layer to work with API types
>> a2, write adapter (e.g. modify Frontend class) between UiCommon using API types and Generic API (GWT RPC) using internal entities
> 
> does this mean converting three times?
> 
>>
>> Our second iteration could be:
>> b1, remove adapter from step a2, drop Generic API (GWT RPC) usage
>> b2, write API type <-> JSON mapper, possibly using some existing framework
> 
> do you mean talking with api in JSON?
> 
>> b3, write RPC layer that talks REST API with Engine
>>
>> However, in order to use REST API types on client (GWT), we need their source code. restapi-interface-definition uses Maven JAXB plugin to generate API types from XSD schema (src/main/resources/api.xsd). On client, we need restapi-definition-<version>-sources.jar that includes those generated API types (target/generated-sources/xjc).
> 
>>
>> As for UI plugins (short term), we can just use restapi-types mappers and implement step b2, (API type -> JSON mapper).
>>
>> Thanks,
>> Vojtech
>>
>>
>> ----- Original Message -----
>> From: "Michael Pasternak" <mpastern at redhat.com>
>> To: "Vojtech Szocs" <vszocs at redhat.com>
>> Cc: "Itamar Heim" <iheim at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
>> Sent: Tuesday, November 20, 2012 9:48:27 AM
>> Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here
>>
>> On 11/20/2012 12:13 AM, Itamar Heim wrote:
>>> On 11/19/2012 02:07 PM, Vojtech Szocs wrote:
>>>> Hi Itamar,
>>>>
>>>> UI plugin infrastructure translates internal business entities into JSON-like representations and passes those representations to UI plugins. (Internal entities are NOT
>>>> exposed to UI plugins directly.)
>>>>
>>>> Currently, all entities supported by UI plugin infrastructure (as per org.ovirt.engine.ui.webadmin.plugin.entity.EntityType) are transformed into following representation:
>>>>
>>>> { entityId: "[BusinessEntityGuidAsString]" }
>>>>
>>>> For example, a VM entity with entity ID "vm123" will translate to:
>>>>
>>>> { entityId: "vm123" }
>>>>
>>>> Translation is currently based on org.ovirt.engine.core.common.businessentities.BusinessEntity interface, like so: "BusinessEntity<? extends NGuid>" (we expect ID type
>>>> parameter to be NGuid-compatible). However, I've found that there are some entities (like Pool - org.ovirt.engine.core.common.businessentities.vm_pools) that don't
>>>> implement BusinessEntity interface.
>>>
>>> ok, so we only pass the ID for now. good.
>>>
>>>>
>>>> Quick question to backend folks - IIRC all entities extend org.ovirt.engine.core.common.businessentities.IVdcQueryable, but not all entities implement
>>>> org.ovirt.engine.core.common.businessentities.BusinessEntity interface. What is the precise relation between IVdcQueryable and BusinessEntity?
>>>>
>>>> As for UI plugins, currently all entities get translated to above mentioned basic JSON-like representation. You can see the relevant code in
>>>> org.ovirt.engine.ui.webadmin.plugin.entity.BaseEntity.from() static method. There's a TODO that says "make this class [BaseEntity] abstract and create specific entity for
>>>> each EntityType" - this means we are planning to extend the above mentioned basic JSON-like representation for different entity types.
>>>>
>>>> For example, for a VM entity we might do:
>>>>
>>>> { entityId: "[BusinessEntityGuidAsString]", osType: "[VmOsType]" }
>>>
>>> just make sure the entity matches the REST API entity.
>>> (which probably means entityId should be changed to id?)
>>
>> if we plan moving UI on top of API, you should be:
>>
>> 1. importing restapi-types project
>> 2. writing intermediate layer to translate BE entities to API's using #1
>> 3. using public entities from #2
>>
>> this way your future migration to API (instead of native BE) will be much
>> more easier.
>>
>>>
>>>>
>>>> Vojtech
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "Itamar Heim" <iheim at redhat.com>
>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>> Sent: Friday, November 16, 2012 5:24:52 PM
>>>> Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here
>>>>
>>>> On 11/16/2012 06:08 PM, Vojtech Szocs wrote:
>>>>>> is there a clear list of all APIs supported now?
>>>>>
>>>>> Not yet, unfortunately, this should be part of "for plugin developers" wiki that is planned to be written in upcoming weeks.
>>>>
>>>> i just wanted to review how we solved not using internal entities as
>>>> part of the API
>>>>
>>>
>>>
>>
>>
> 
> 


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From mpastern at redhat.com  Wed Dec 12 13:18:04 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Wed, 12 Dec 2012 15:18:04 +0200
Subject: [Engine-devel] UI Plugins: PoC patch revision 7 is here
In-Reply-To: <1005426847.17530385.1355317853426.JavaMail.root@redhat.com>
References: <1005426847.17530385.1355317853426.JavaMail.root@redhat.com>
Message-ID: <50C8840C.4000200@redhat.com>

On 12/12/2012 03:10 PM, Vojtech Szocs wrote:
> Thanks Michael, in that case we can just use XML when talking with REST API.
> 
> Also, one idea worth implementing in future is to have some sort of debug console on GUI (I've seen this idea implemented in other applications). When activated, users could see REST API XML communication (request/response) between WebAdmin and Engine, which could be useful for someone interested in writing his own REST API client.

right, i always do that in all clients i've wrote (including SDKs & CLI).

> 
> Vojtech
> 
> 
> ----- Original Message -----
> From: "Michael Pasternak" <mpastern at redhat.com>
> To: "Vojtech Szocs" <vszocs at redhat.com>
> Cc: "Itamar Heim" <iheim at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 12, 2012 1:33:56 PM
> Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here
> 
> On 12/12/2012 01:51 PM, Vojtech Szocs wrote:
>> Hi, sorry for my late response,
>>
>>>> Our first iteration could be:
>>>> a1, rewrite UiCommon (UI business logic) layer to work with API types
>>>> a2, write adapter (e.g. modify Frontend class) between UiCommon using API types and Generic API (GWT RPC) using internal entities
>>>
>>> does this mean converting three times?
>>
>> Not sure what you mean, but here's my idea of the first iteration - what would happen on frontend side (GWT):
>> 1. UiCommon, using API types, invokes query/command via RPC bridge (org.ovirt.engine.ui.frontend.Frontend class)
>> 2. Frontend class, still using Generic API (GWT RPC), does the actual query/command invocation
>> 3. Frontend class receives query/command result and translates it from BE type to API type (converter callback)
>>
>> My idea was to isolate BE type usage into Frontend class, so that when we decide to use REST API, we just need to update our RPC bridge (Frontend class).
>>
>>>> Our second iteration could be:
>>>> b1, remove adapter from step a2, drop Generic API (GWT RPC) usage
>>>> b2, write API type <-> JSON mapper, possibly using some existing framework
>>>
>>> do you mean talking with api in JSON?
>>
>> Yes, that was my original idea. I'm not against using XML when talking with REST API either. GWT supports parsing both JSON and XML.
> 
> infra. is there, but officially we not supporting json yet,
> (resteasy-json-provider has issues with jaxb, - we waiting for fix)
> 
>>
>> Vojtech
>>
>>
>> ----- Original Message -----
>> From: "Michael Pasternak" <mpastern at redhat.com>
>> To: "Vojtech Szocs" <vszocs at redhat.com>
>> Cc: "Itamar Heim" <iheim at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
>> Sent: Sunday, November 25, 2012 9:27:59 AM
>> Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here
>>
>> On 11/20/2012 12:57 PM, Vojtech Szocs wrote:
>>> Hi Michael,
>>>
>>>> if we plan moving UI on top of API, you should be:
>>>>
>>>> 1. importing restapi-types project
>>>> 2. writing intermediate layer to translate BE entities to API's using #1
>>>> 3. using public entities from #2
>>>>
>>>> this way your future migration to API (instead of native BE) will be much
>>>> more easier.
>>>
>>> Indeed, this is very useful for GWT RPC -> REST API transition in general, many thanks for pointing this out, Michael. As you suggest, we can use restapi-types mappers to translate between internal entities and API types.
>>>
>>> Our first iteration could be:
>>> a1, rewrite UiCommon (UI business logic) layer to work with API types
>>> a2, write adapter (e.g. modify Frontend class) between UiCommon using API types and Generic API (GWT RPC) using internal entities
>>
>> does this mean converting three times?
>>
>>>
>>> Our second iteration could be:
>>> b1, remove adapter from step a2, drop Generic API (GWT RPC) usage
>>> b2, write API type <-> JSON mapper, possibly using some existing framework
>>
>> do you mean talking with api in JSON?
>>
>>> b3, write RPC layer that talks REST API with Engine
>>>
>>> However, in order to use REST API types on client (GWT), we need their source code. restapi-interface-definition uses Maven JAXB plugin to generate API types from XSD schema (src/main/resources/api.xsd). On client, we need restapi-definition-<version>-sources.jar that includes those generated API types (target/generated-sources/xjc).
>>
>>>
>>> As for UI plugins (short term), we can just use restapi-types mappers and implement step b2, (API type -> JSON mapper).
>>>
>>> Thanks,
>>> Vojtech
>>>
>>>
>>> ----- Original Message -----
>>> From: "Michael Pasternak" <mpastern at redhat.com>
>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>> Cc: "Itamar Heim" <iheim at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
>>> Sent: Tuesday, November 20, 2012 9:48:27 AM
>>> Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here
>>>
>>> On 11/20/2012 12:13 AM, Itamar Heim wrote:
>>>> On 11/19/2012 02:07 PM, Vojtech Szocs wrote:
>>>>> Hi Itamar,
>>>>>
>>>>> UI plugin infrastructure translates internal business entities into JSON-like representations and passes those representations to UI plugins. (Internal entities are NOT
>>>>> exposed to UI plugins directly.)
>>>>>
>>>>> Currently, all entities supported by UI plugin infrastructure (as per org.ovirt.engine.ui.webadmin.plugin.entity.EntityType) are transformed into following representation:
>>>>>
>>>>> { entityId: "[BusinessEntityGuidAsString]" }
>>>>>
>>>>> For example, a VM entity with entity ID "vm123" will translate to:
>>>>>
>>>>> { entityId: "vm123" }
>>>>>
>>>>> Translation is currently based on org.ovirt.engine.core.common.businessentities.BusinessEntity interface, like so: "BusinessEntity<? extends NGuid>" (we expect ID type
>>>>> parameter to be NGuid-compatible). However, I've found that there are some entities (like Pool - org.ovirt.engine.core.common.businessentities.vm_pools) that don't
>>>>> implement BusinessEntity interface.
>>>>
>>>> ok, so we only pass the ID for now. good.
>>>>
>>>>>
>>>>> Quick question to backend folks - IIRC all entities extend org.ovirt.engine.core.common.businessentities.IVdcQueryable, but not all entities implement
>>>>> org.ovirt.engine.core.common.businessentities.BusinessEntity interface. What is the precise relation between IVdcQueryable and BusinessEntity?
>>>>>
>>>>> As for UI plugins, currently all entities get translated to above mentioned basic JSON-like representation. You can see the relevant code in
>>>>> org.ovirt.engine.ui.webadmin.plugin.entity.BaseEntity.from() static method. There's a TODO that says "make this class [BaseEntity] abstract and create specific entity for
>>>>> each EntityType" - this means we are planning to extend the above mentioned basic JSON-like representation for different entity types.
>>>>>
>>>>> For example, for a VM entity we might do:
>>>>>
>>>>> { entityId: "[BusinessEntityGuidAsString]", osType: "[VmOsType]" }
>>>>
>>>> just make sure the entity matches the REST API entity.
>>>> (which probably means entityId should be changed to id?)
>>>
>>> if we plan moving UI on top of API, you should be:
>>>
>>> 1. importing restapi-types project
>>> 2. writing intermediate layer to translate BE entities to API's using #1
>>> 3. using public entities from #2
>>>
>>> this way your future migration to API (instead of native BE) will be much
>>> more easier.
>>>
>>>>
>>>>>
>>>>> Vojtech
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Itamar Heim" <iheim at redhat.com>
>>>>> To: "Vojtech Szocs" <vszocs at redhat.com>
>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>>> Sent: Friday, November 16, 2012 5:24:52 PM
>>>>> Subject: Re: [Engine-devel] UI Plugins: PoC patch revision 7 is here
>>>>>
>>>>> On 11/16/2012 06:08 PM, Vojtech Szocs wrote:
>>>>>>> is there a clear list of all APIs supported now?
>>>>>>
>>>>>> Not yet, unfortunately, this should be part of "for plugin developers" wiki that is planned to be written in upcoming weeks.
>>>>>
>>>>> i just wanted to review how we solved not using internal entities as
>>>>> part of the API
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
> 
> 


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From ovedo at redhat.com  Wed Dec 12 13:34:25 2012
From: ovedo at redhat.com (Oved Ourfalli)
Date: Wed, 12 Dec 2012 08:34:25 -0500 (EST)
Subject: [Engine-devel] UI plugins - next steps
In-Reply-To: <413127613.17526084.1355317031405.JavaMail.root@redhat.com>
Message-ID: <616374438.60989790.1355319265885.JavaMail.root@redhat.com>

Nice work!

See some comments inline.


----- Original Message -----
> From: "Vojtech Szocs" <vszocs at redhat.com>
> To: "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 12, 2012 2:57:11 PM
> Subject: [Engine-devel] UI plugins - next steps
> 
> Hi guys,
> 
> here's my list of tasks that didn't make it into UI plugins yet.
> These tasks should be revisited and implemented in future patches.
> The list isn't sorted in any way, so feel free to comment or
> highlight specific tasks to raise their priority, or add your own
> tasks/ideas.
> 
> 1. Pass proper (restapi-definition) entities to UI plugins, instead
> of simple "{entityId:<guidAsString>}" objects
>    - using restapi-types mappers to map backend business entities to
>    restapi-definition entities (Java POJOs generated from api.xsd
>    via JAXB)
>    - exporting restapi-definition entities for use with JavaScript
>    (UI plugins), e.g. using gwt-exporter [1] (alternatively, we
>    could use GWT deferred binding and do this ourselves)
> 
> 2. showDialog API function should integrate with WebAdmin dialog
> infrastructure
>    - shouldn't be too hard, need to create custom dialog
>    PresenterWidget/View and bind it as non-singleton (just like
>    other popups in oVirt GUI)
> 
> 3. Extend existing WebAdmin dialogs - prototype Doron's proposal to
> extend Edit Cluster Policy dialog with custom options that would be
> passed to backend
> 
> 4. Introduce form-based tabs - custom tab that would display
> key/value pairs organized in columns, just like in standard
> "General" sub tabs
> 
> 5. Introduce table-based tabs - custom tab that would display grid
> (table) just like in standard main tabs, with API to add columns,
> set data, etc.
> 
> 6. Improve api.addMainTabActionButton to allow different button vs.
> context menu item representations (only button, only menu item,
> both)
> 
> 7. Add api.currentUserName and api.currentUserId API functions (these
> would return same data as with UserLogin event parameters)
> 
Pushing this now.
http://gerrit.ovirt.org/#/c/9998/

> 8. Minor things: consider renaming UiInit to PluginInit (one-time
> plugin initialization, called only once, can do non-UI init stuff
> there as well), consider renaming api.ready to api.initialize
> (plugin indicates that infrastructure can proceed with initializing
> itself)
> 

I think we should also consider adding an option to add a button in a more general scope, for things that aren't entity-specific, but system-specific.
Maybe next to the "configure" button, in the upper right corner of the screen?

> Regards,
> Vojtech
> 
> [1] http://code.google.com/p/gwt-exporter/wiki/GettingStarted
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From George.Costea at netapp.com  Wed Dec 12 13:58:42 2012
From: George.Costea at netapp.com (Costea, George)
Date: Wed, 12 Dec 2012 13:58:42 +0000
Subject: [Engine-devel] UI plugins patches merged,
 moving out of proof-of-concept stage
In-Reply-To: <631816160.17519448.1355315261929.JavaMail.root@redhat.com>
References: <800242646.17508631.1355314071136.JavaMail.root@redhat.com>
	<631816160.17519448.1355315261929.JavaMail.root@redhat.com>
Message-ID: <6C8AC8C50E170C4E9B44D47B39B24A4812532F78@SACEXCMBX04-PRD.hq.netapp.com>

Great work Vojtech.  Thank you.

-----Original Message-----
From: engine-devel-bounces at ovirt.org [mailto:engine-devel-bounces at ovirt.org] On Behalf Of Vojtech Szocs
Sent: Wednesday, December 12, 2012 7:28 AM
To: engine-devel
Subject: [Engine-devel] UI plugins patches merged, moving out of proof-of-concept stage

Hi everyone,

I'm happy to announce that oVirt UI plugins feature just moved out of proof-of-concept stage, with relevant patches [1,2] merged into ovirt-engine git repository.

I'll be working on related documentation (wiki) in near future, along with contributing some sample plugins into samples-uiplugins [3] git repository.

I'd like to thank everyone involved in pushing this feature forward. I hope we'll see some interesting plugins soon!

Regards,
Vojtech

[1] http://gerrit.ovirt.org/#/c/8120/
[2] http://gerrit.ovirt.org/#/c/9250/
[3] git://gerrit.ovirt.org/samples-uiplugins
_______________________________________________
Engine-devel mailing list
Engine-devel at ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


From mburns at redhat.com  Wed Dec 12 15:35:55 2012
From: mburns at redhat.com (Mike Burns)
Date: Wed, 12 Dec 2012 10:35:55 -0500
Subject: [Engine-devel] Request power-user status for mom project in
	jenkins
In-Reply-To: <20121211185832.GC19473@redhat.com>
References: <20121211161145.GB30988@aglitke>
	<33068073.14181482.1355243047864.JavaMail.root@redhat.com>
	<20121211185832.GC19473@redhat.com>
Message-ID: <1355326555.8794.20.camel@beelzebub.mburnsfire.net>

On Tue, 2012-12-11 at 20:58 +0200, Dan Kenigsberg wrote:
> On Tue, Dec 11, 2012 at 11:24:07AM -0500, Eyal Edri wrote:
> > [adding engine-devel]
> > 
> > usually there is an official vote on each new power user for jenkins,
> > and as long as there are no objections it is approved.
> > 
> > since i know you're a mom maintainer and familiar with jenkins, i vote +1.
> 
> I trust Adam (who is mom's maintainer), and trust that he is capable of
> becoming familiar with jenkins without causing pain to other users.
> 
> +1

+1

> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra




From jhernand at redhat.com  Wed Dec 12 16:06:30 2012
From: jhernand at redhat.com (Juan Hernandez)
Date: Wed, 12 Dec 2012 17:06:30 +0100
Subject: [Engine-devel] ThreadPoolUtil with 500 threads and no queue?
Message-ID: <50C8AB86.4060304@redhat.com>

Hello all,

What is the reasoning behind the decision to have a pool with a maximum
of 500 threads and no job queue (see ThreadPoolUtil.java)? Wouldn't it
make more sense to have a much smaller thread pool and a potentially
large queue of jobs?

Regards,
Juan Hernandez
-- 
Direcci?n Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
3?D, 28016 Madrid, Spain
Inscrita en el Reg. Mercantil de Madrid ? C.I.F. B82657941 - Red Hat S.L.


From mkublin at redhat.com  Wed Dec 12 16:21:38 2012
From: mkublin at redhat.com (Michael Kublin)
Date: Wed, 12 Dec 2012 11:21:38 -0500 (EST)
Subject: [Engine-devel] ThreadPoolUtil with 500 threads and no queue?
In-Reply-To: <50C8AB86.4060304@redhat.com>
Message-ID: <298170225.290893.1355329298903.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Juan Hernandez" <jhernand at redhat.com>
> To: engine-devel at ovirt.org
> Sent: Wednesday, December 12, 2012 6:06:30 PM
> Subject: [Engine-devel] ThreadPoolUtil with 500 threads and no queue?
> 
> Hello all,
> 
> What is the reasoning behind the decision to have a pool with a
> maximum
> of 500 threads and no job queue (see ThreadPoolUtil.java)? Wouldn't
> it
> make more sense to have a much smaller thread pool and a potentially
> large queue of jobs?
> 
> Regards,
> Juan Hernandez

There are three general strategies for queuing:

 1) Direct handoffs. A good default choice for a work queue is a SynchronousQueue that hands off tasks to threads without otherwise holding them. Here, an attempt to queue a task will fail if no threads are immediately available to run it, so a new thread will be constructed. This policy avoids lockups when handling sets of requests that might have internal dependencies. Direct handoffs generally require unbounded maximumPoolSizes to avoid rejection of new submitted tasks. This in turn admits the possibility of unbounded thread growth when commands continue to arrive on average faster than they can be processed.
  2) Unbounded queues. Using an unbounded queue (for example a LinkedBlockingQueue without a predefined capacity) will cause new tasks to wait in the queue when all corePoolSize threads are busy. Thus, no more than corePoolSize threads will ever be created. (And the value of the maximumPoolSize therefore doesn't have any effect.) This may be appropriate when each task is completely independent of others, so tasks cannot affect each others execution; for example, in a web page server. While this style of queuing can be useful in smoothing out transient bursts of requests, it admits the possibility of unbounded work queue growth when commands continue to arrive on average faster than they can be processed.
  3) Bounded queues. A bounded queue (for example, an ArrayBlockingQueue) helps prevent resource exhaustion when used with finite maximumPoolSizes, but can be more difficult to tune and control. Queue sizes and maximum pool sizes may be traded off for each other: Using large queues and small pools minimizes CPU usage, OS resources, and context-switching overhead, but can lead to artificially low throughput. If tasks frequently block (for example if they are I/O bound), a system may be able to schedule time for more threads than you otherwise allow. Use of small queues generally requires larger pool sizes, which keeps CPUs busier but may encounter unacceptable scheduling overhead, which also decreases throughput.

Why not? we are using 1).
Actually 500 threads should be enough for very big applications


From jhernand at redhat.com  Wed Dec 12 16:31:44 2012
From: jhernand at redhat.com (Juan Hernandez)
Date: Wed, 12 Dec 2012 17:31:44 +0100
Subject: [Engine-devel] ThreadPoolUtil with 500 threads and no queue?
In-Reply-To: <298170225.290893.1355329298903.JavaMail.root@redhat.com>
References: <298170225.290893.1355329298903.JavaMail.root@redhat.com>
Message-ID: <50C8B170.3020405@redhat.com>

On 12/12/2012 05:21 PM, Michael Kublin wrote:
> 
> 
> ----- Original Message -----
>> From: "Juan Hernandez" <jhernand at redhat.com>
>> To: engine-devel at ovirt.org
>> Sent: Wednesday, December 12, 2012 6:06:30 PM
>> Subject: [Engine-devel] ThreadPoolUtil with 500 threads and no queue?
>>
>> Hello all,
>>
>> What is the reasoning behind the decision to have a pool with a
>> maximum
>> of 500 threads and no job queue (see ThreadPoolUtil.java)? Wouldn't
>> it
>> make more sense to have a much smaller thread pool and a potentially
>> large queue of jobs?
>>
>> Regards,
>> Juan Hernandez
> 
> There are three general strategies for queuing:
> 
>  1) Direct handoffs. A good default choice for a work queue is a SynchronousQueue that hands off tasks to threads without otherwise holding them. Here, an attempt to queue a task will fail if no threads are immediately available to run it, so a new thread will be constructed. This policy avoids lockups when handling sets of requests that might have internal dependencies. Direct handoffs generally require unbounded maximumPoolSizes to avoid rejection of new submitted tasks. This in turn admits the possibility of unbounded thread growth when commands continue to arrive on average faster than they can be processed.
>   2) Unbounded queues. Using an unbounded queue (for example a LinkedBlockingQueue without a predefined capacity) will cause new tasks to wait in the queue when all corePoolSize threads are busy. Thus, no more than corePoolSize threads will ever be created. (And the value of the maximumPoolSize therefore doesn't have any effect.) This may be appropriate when each task is completely independent of others, so tasks cannot affect each others execution; for example, in a web page server. While this style of queuing can be useful in smoothing out transient bursts of requests, it admits the possibility of unbounded work queue growth when commands continue to arrive on average faster than they can be processed.
>   3) Bounded queues. A bounded queue (for example, an ArrayBlockingQueue) helps prevent resource exhaustion when used with finite maximumPoolSizes, but can be more difficult to tune and control. Queue sizes and maximum pool sizes may be traded off for each other: Using large queues and small pools minimizes CPU usage, OS resources, and context-switching overhead, but can lead to artificially low throughput. If tasks frequently block (for example if they are I/O bound), a system may be able to schedule time for more threads than you otherwise allow. Use of small queues generally requires larger pool sizes, which keeps CPUs busier but may encounter unacceptable scheduling overhead, which also decreases throughput.
> 
> Why not? we are using 1).
> Actually 500 threads should be enough for very big applications

I think that 500 are maybe too much, even for a very big application,
the reasons you explain very well in 3). A resource that we can very
easily overload is the database. If those 500 threads happen to need
database connections we have a problem.

I think we should use a bounded queue and have both the number of
threads and the size of the queue configurable. Does that make sense?

-- 
Direcci?n Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
3?D, 28016 Madrid, Spain
Inscrita en el Reg. Mercantil de Madrid ? C.I.F. B82657941 - Red Hat S.L.


From alonbl at redhat.com  Wed Dec 12 16:53:30 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Wed, 12 Dec 2012 11:53:30 -0500 (EST)
Subject: [Engine-devel] Adding Authentication mechanism to oVirt
In-Reply-To: <50C6D9FE.40008@univ-montp2.fr>
Message-ID: <1490596332.4736653.1355331210001.JavaMail.root@redhat.com>

Hello Thierry,

If I understand correctly you wish to help in modifying the engine to support non GSSAPI authentication methods.

Following is a quick design goals for this implementation.

I will be glad to improve this.
Alon

---

Implementation should support the following transports:

1. LDAP (plain).
2. LDAP over TLS.
3. LDAP with StartTLS.

Implementation should support the following authentication methods:

1. Simple.
2. Digest-MD5 (plain and strong).

I believe the GSSAPI can be dropped, I see no advantage of using it.

A sample of low level implementation for transport and authentication is attached.

When adding a domain the following facts should be provided:

 1. Search user name.
 2. Search user password.
 3. Transport type (ldap, ldaps, ldap+startTLS)
 4. Authentication (simple, Digest-MD5)
 5. Sever selection policy (failover, round-robin, random).
 6. Server address type (explicit, DNS record)
 7. Server address set.
 8. Optional base DN.
 9. Optional root certificate.
10. Optional certificate chain.
11. Search page size.
10. Query timeout.
etc...

Within product there are two separate components that perform LDAP authentication:

1. User password validation.
2. User permission fetch.

These two components needs to work in share-nothing mode, meaning that each should communicate with directory independently with the other.

USER PASSWORD VALIDATION

Input: user
Input: domain
Input: password
Output: DN of user
Output: success/failure
Credentials used: user/password provided.
Notes: LDAP session should not be cached.
Logic: Perform LDAP bind.

USER PERMISSION FETCH

Input: DN of user (passed by user password validation)
Input: domain (passed by user password validation)
Output: A set of permissions
Credentials used: search user and password configured within system.
Notes: LDAP context can be cached. 
Logic: Perform LDAP searches, this is most of current logic.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LDAPSearch.java
Type: text/x-java
Size: 7540 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121212/51d91c61/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: UnsecureSSLSocketFactory.java
Type: text/x-java
Size: 2292 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121212/51d91c61/attachment-0001.bin>

From ykaul at redhat.com  Wed Dec 12 17:18:18 2012
From: ykaul at redhat.com (Yaniv Kaul)
Date: Wed, 12 Dec 2012 19:18:18 +0200
Subject: [Engine-devel] Adding Authentication mechanism to oVirt
In-Reply-To: <1490596332.4736653.1355331210001.JavaMail.root@redhat.com>
References: <1490596332.4736653.1355331210001.JavaMail.root@redhat.com>
Message-ID: <50C8BC5A.1050705@redhat.com>

On 12/12/2012 06:53 PM, Alon Bar-Lev wrote:
> Hello Thierry,
>
> If I understand correctly you wish to help in modifying the engine to support non GSSAPI authentication methods.
>
> Following is a quick design goals for this implementation.
>
> I will be glad to improve this.
> Alon
>
> ---
>
> Implementation should support the following transports:
>
> 1. LDAP (plain).
> 2. LDAP over TLS.
> 3. LDAP with StartTLS.
>
> Implementation should support the following authentication methods:
>
> 1. Simple.
> 2. Digest-MD5 (plain and strong).
>
> I believe the GSSAPI can be dropped, I see no advantage of using it.

Except that Active Directory usually does not work (especially not out 
of the box) with certificates.
Y.

>
> A sample of low level implementation for transport and authentication is attached.
>
> When adding a domain the following facts should be provided:
>
>   1. Search user name.
>   2. Search user password.
>   3. Transport type (ldap, ldaps, ldap+startTLS)
>   4. Authentication (simple, Digest-MD5)
>   5. Sever selection policy (failover, round-robin, random).
>   6. Server address type (explicit, DNS record)
>   7. Server address set.
>   8. Optional base DN.
>   9. Optional root certificate.
> 10. Optional certificate chain.
> 11. Search page size.
> 10. Query timeout.
> etc...
>
> Within product there are two separate components that perform LDAP authentication:
>
> 1. User password validation.
> 2. User permission fetch.
>
> These two components needs to work in share-nothing mode, meaning that each should communicate with directory independently with the other.
>
> USER PASSWORD VALIDATION
>
> Input: user
> Input: domain
> Input: password
> Output: DN of user
> Output: success/failure
> Credentials used: user/password provided.
> Notes: LDAP session should not be cached.
> Logic: Perform LDAP bind.
>
> USER PERMISSION FETCH
>
> Input: DN of user (passed by user password validation)
> Input: domain (passed by user password validation)
> Output: A set of permissions
> Credentials used: search user and password configured within system.
> Notes: LDAP context can be cached.
> Logic: Perform LDAP searches, this is most of current logic.



From alonbl at redhat.com  Wed Dec 12 17:25:03 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Wed, 12 Dec 2012 12:25:03 -0500 (EST)
Subject: [Engine-devel] Adding Authentication mechanism to oVirt
In-Reply-To: <50C8BC5A.1050705@redhat.com>
Message-ID: <1061921830.4742907.1355333103405.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Yaniv Kaul" <ykaul at redhat.com>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "Thierry Kauffmann" <thierry.kauffmann at univ-montp2.fr>, "Itamar Heim" <iheim at redhat.com>, "engine-devel"
> <engine-devel at ovirt.org>
> Sent: Wednesday, December 12, 2012 7:18:18 PM
> Subject: Re: Adding Authentication mechanism to oVirt
> 
> On 12/12/2012 06:53 PM, Alon Bar-Lev wrote:
> > Hello Thierry,
> >
> > If I understand correctly you wish to help in modifying the engine
> > to support non GSSAPI authentication methods.
> >
> > Following is a quick design goals for this implementation.
> >
> > I will be glad to improve this.
> > Alon
> >
> > ---
> >
> > Implementation should support the following transports:
> >
> > 1. LDAP (plain).
> > 2. LDAP over TLS.
> > 3. LDAP with StartTLS.
> >
> > Implementation should support the following authentication methods:
> >
> > 1. Simple.
> > 2. Digest-MD5 (plain and strong).
> >
> > I believe the GSSAPI can be dropped, I see no advantage of using
> > it.
> 
> Except that Active Directory usually does not work (especially not
> out
> of the box) with certificates.
> Y.

Active Directory support SASL with Diget-MD5, no need to support GSSAPI.

> 
> >
> > A sample of low level implementation for transport and
> > authentication is attached.
> >
> > When adding a domain the following facts should be provided:
> >
> >   1. Search user name.
> >   2. Search user password.
> >   3. Transport type (ldap, ldaps, ldap+startTLS)
> >   4. Authentication (simple, Digest-MD5)
> >   5. Sever selection policy (failover, round-robin, random).
> >   6. Server address type (explicit, DNS record)
> >   7. Server address set.
> >   8. Optional base DN.
> >   9. Optional root certificate.
> > 10. Optional certificate chain.
> > 11. Search page size.
> > 10. Query timeout.
> > etc...
> >
> > Within product there are two separate components that perform LDAP
> > authentication:
> >
> > 1. User password validation.
> > 2. User permission fetch.
> >
> > These two components needs to work in share-nothing mode, meaning
> > that each should communicate with directory independently with the
> > other.
> >
> > USER PASSWORD VALIDATION
> >
> > Input: user
> > Input: domain
> > Input: password
> > Output: DN of user
> > Output: success/failure
> > Credentials used: user/password provided.
> > Notes: LDAP session should not be cached.
> > Logic: Perform LDAP bind.
> >
> > USER PERMISSION FETCH
> >
> > Input: DN of user (passed by user password validation)
> > Input: domain (passed by user password validation)
> > Output: A set of permissions
> > Credentials used: search user and password configured within
> > system.
> > Notes: LDAP context can be cached.
> > Logic: Perform LDAP searches, this is most of current logic.
> 
> 


From mkublin at redhat.com  Wed Dec 12 17:37:26 2012
From: mkublin at redhat.com (Michael Kublin)
Date: Wed, 12 Dec 2012 12:37:26 -0500 (EST)
Subject: [Engine-devel] ThreadPoolUtil with 500 threads and no queue?
In-Reply-To: <50C8B170.3020405@redhat.com>
Message-ID: <1273316839.316941.1355333846389.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Juan Hernandez" <jhernand at redhat.com>
> To: "Michael Kublin" <mkublin at redhat.com>
> Cc: engine-devel at ovirt.org
> Sent: Wednesday, December 12, 2012 6:31:44 PM
> Subject: Re: [Engine-devel] ThreadPoolUtil with 500 threads and no queue?
> 
> On 12/12/2012 05:21 PM, Michael Kublin wrote:
> > 
> > 
> > ----- Original Message -----
> >> From: "Juan Hernandez" <jhernand at redhat.com>
> >> To: engine-devel at ovirt.org
> >> Sent: Wednesday, December 12, 2012 6:06:30 PM
> >> Subject: [Engine-devel] ThreadPoolUtil with 500 threads and no
> >> queue?
> >>
> >> Hello all,
> >>
> >> What is the reasoning behind the decision to have a pool with a
> >> maximum
> >> of 500 threads and no job queue (see ThreadPoolUtil.java)?
> >> Wouldn't
> >> it
> >> make more sense to have a much smaller thread pool and a
> >> potentially
> >> large queue of jobs?
> >>
> >> Regards,
> >> Juan Hernandez
> > 
> > There are three general strategies for queuing:
> > 
> >  1) Direct handoffs. A good default choice for a work queue is a
> >  SynchronousQueue that hands off tasks to threads without
> >  otherwise holding them. Here, an attempt to queue a task will
> >  fail if no threads are immediately available to run it, so a new
> >  thread will be constructed. This policy avoids lockups when
> >  handling sets of requests that might have internal dependencies.
> >  Direct handoffs generally require unbounded maximumPoolSizes to
> >  avoid rejection of new submitted tasks. This in turn admits the
> >  possibility of unbounded thread growth when commands continue to
> >  arrive on average faster than they can be processed.
> >   2) Unbounded queues. Using an unbounded queue (for example a
> >   LinkedBlockingQueue without a predefined capacity) will cause
> >   new tasks to wait in the queue when all corePoolSize threads are
> >   busy. Thus, no more than corePoolSize threads will ever be
> >   created. (And the value of the maximumPoolSize therefore doesn't
> >   have any effect.) This may be appropriate when each task is
> >   completely independent of others, so tasks cannot affect each
> >   others execution; for example, in a web page server. While this
> >   style of queuing can be useful in smoothing out transient bursts
> >   of requests, it admits the possibility of unbounded work queue
> >   growth when commands continue to arrive on average faster than
> >   they can be processed.
> >   3) Bounded queues. A bounded queue (for example, an
> >   ArrayBlockingQueue) helps prevent resource exhaustion when used
> >   with finite maximumPoolSizes, but can be more difficult to tune
> >   and control. Queue sizes and maximum pool sizes may be traded
> >   off for each other: Using large queues and small pools minimizes
> >   CPU usage, OS resources, and context-switching overhead, but can
> >   lead to artificially low throughput. If tasks frequently block
> >   (for example if they are I/O bound), a system may be able to
> >   schedule time for more threads than you otherwise allow. Use of
> >   small queues generally requires larger pool sizes, which keeps
> >   CPUs busier but may encounter unacceptable scheduling overhead,
> >   which also decreases throughput.
> > 
> > Why not? we are using 1).
> > Actually 500 threads should be enough for very big applications
> 
> I think that 500 are maybe too much, even for a very big application,
> the reasons you explain very well in 3). A resource that we can very
> easily overload is the database. If those 500 threads happen to need
> database connections we have a problem.
> 
> I think we should use a bounded queue and have both the number of
> threads and the size of the queue configurable. Does that make sense?
> 
Actually I think that it is more complicated.
Today most of the threads are used to perform xml rpc calls and in order to paralyse some operations 
(for example connect all host to some storage domain), most of the action from users are not
opening a new thread in order to retrieve some information from DB.
I think today most of the load on DB comes from scheduled monitoring jobs,
which are running every couple seconds and performing enormous number of queries to DB.
I think if we want to make some kind of queue it should be done at the business logic level
and if we decided that we can not run some action because of load (user action or internal action) we can reject it,
I think that thread pool is internal mechanism which can not solve our design problems, we can tune it but it is not
main cause of problem.
 


From iheim at redhat.com  Wed Dec 12 23:27:15 2012
From: iheim at redhat.com (Itamar Heim)
Date: Thu, 13 Dec 2012 01:27:15 +0200
Subject: [Engine-devel] Adding Authentication mechanism to oVirt
In-Reply-To: <1490596332.4736653.1355331210001.JavaMail.root@redhat.com>
References: <1490596332.4736653.1355331210001.JavaMail.root@redhat.com>
Message-ID: <50C912D3.8080109@redhat.com>

On 12/12/2012 06:53 PM, Alon Bar-Lev wrote:
> I believe the GSSAPI can be dropped, I see no advantage of using it.

how would upgrade/backward compatibility look like?



From alonbl at redhat.com  Wed Dec 12 23:32:18 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Wed, 12 Dec 2012 18:32:18 -0500 (EST)
Subject: [Engine-devel] Adding Authentication mechanism to oVirt
In-Reply-To: <50C912D3.8080109@redhat.com>
Message-ID: <848751896.4812544.1355355138933.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "Thierry Kauffmann" <thierry.kauffmann at univ-montp2.fr>, "Yaniv Kaul" <ykaul at redhat.com>, "engine-devel"
> <engine-devel at ovirt.org>
> Sent: Thursday, December 13, 2012 1:27:15 AM
> Subject: Re: Adding Authentication mechanism to oVirt
> 
> On 12/12/2012 06:53 PM, Alon Bar-Lev wrote:
> > I believe the GSSAPI can be dropped, I see no advantage of using
> > it.
> 
> how would upgrade/backward compatibility look like?
> 
> 

We have the user/password, so there should be no problem we can test if authentication succeeds in methods based on some order... but we cross this when we reach the point.
Asking the user for transport/authentication scheme during upgrade is something that can also be done.

We can keep the GSSAPI around of course... Not something I would like to do.

Alon


From iheim at redhat.com  Wed Dec 12 23:39:38 2012
From: iheim at redhat.com (Itamar Heim)
Date: Thu, 13 Dec 2012 01:39:38 +0200
Subject: [Engine-devel] UI plugins - next steps
In-Reply-To: <413127613.17526084.1355317031405.JavaMail.root@redhat.com>
References: <413127613.17526084.1355317031405.JavaMail.root@redhat.com>
Message-ID: <50C915BA.6090607@redhat.com>

On 12/12/2012 02:57 PM, Vojtech Szocs wrote:
> 1. Pass proper (restapi-definition) entities to UI plugins, instead of simple "{entityId:<guidAsString>}" objects
>     - using restapi-types mappers to map backend business entities to restapi-definition entities (Java POJOs generated from api.xsd via JAXB)
>     - exporting restapi-definition entities for use with JavaScript (UI plugins), e.g. using gwt-exporter [1] (alternatively, we could use GWT deferred binding and do this ourselves)

I think its time we revisit the implications/scope of moving the UI to 
work on top of the REST API or just the REST entities.
until then, i think just adding specific fields that make sense/are 
asked for for these entities should be good enough.
if we see the "moving UI on top of REST API" is still too far, then 
doing the type mapping for just the plugins would make sense.



From iheim at redhat.com  Wed Dec 12 23:43:19 2012
From: iheim at redhat.com (Itamar Heim)
Date: Thu, 13 Dec 2012 01:43:19 +0200
Subject: [Engine-devel] Adding Authentication mechanism to oVirt
In-Reply-To: <1490596332.4736653.1355331210001.JavaMail.root@redhat.com>
References: <1490596332.4736653.1355331210001.JavaMail.root@redhat.com>
Message-ID: <50C91697.5040501@redhat.com>

On 12/12/2012 06:53 PM, Alon Bar-Lev wrote:
> Hello Thierry,
>
> If I understand correctly you wish to help in modifying the engine to support non GSSAPI authentication methods.
>
> Following is a quick design goals for this implementation.
>
> I will be glad to improve this.

thanks for writing this part.
what about making the LDAP part plugable, or at least 
scheme-configurable to cover openLDAP, or any other source (atlassian 
crowd), or well, anything?

thanks,
    Itamar

> Alon
>
> ---
>
> Implementation should support the following transports:
>
> 1. LDAP (plain).
> 2. LDAP over TLS.
> 3. LDAP with StartTLS.
>
> Implementation should support the following authentication methods:
>
> 1. Simple.
> 2. Digest-MD5 (plain and strong).
>
> I believe the GSSAPI can be dropped, I see no advantage of using it.
>
> A sample of low level implementation for transport and authentication is attached.
>
> When adding a domain the following facts should be provided:
>
>   1. Search user name.
>   2. Search user password.
>   3. Transport type (ldap, ldaps, ldap+startTLS)
>   4. Authentication (simple, Digest-MD5)
>   5. Sever selection policy (failover, round-robin, random).
>   6. Server address type (explicit, DNS record)
>   7. Server address set.
>   8. Optional base DN.
>   9. Optional root certificate.
> 10. Optional certificate chain.
> 11. Search page size.
> 10. Query timeout.
> etc...
>
> Within product there are two separate components that perform LDAP authentication:
>
> 1. User password validation.
> 2. User permission fetch.
>
> These two components needs to work in share-nothing mode, meaning that each should communicate with directory independently with the other.
>
> USER PASSWORD VALIDATION
>
> Input: user
> Input: domain
> Input: password
> Output: DN of user
> Output: success/failure
> Credentials used: user/password provided.
> Notes: LDAP session should not be cached.
> Logic: Perform LDAP bind.
>
> USER PERMISSION FETCH
>
> Input: DN of user (passed by user password validation)
> Input: domain (passed by user password validation)
> Output: A set of permissions
> Credentials used: search user and password configured within system.
> Notes: LDAP context can be cached.
> Logic: Perform LDAP searches, this is most of current logic.
>




From alonbl at redhat.com  Wed Dec 12 23:47:27 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Wed, 12 Dec 2012 18:47:27 -0500 (EST)
Subject: [Engine-devel] Adding Authentication mechanism to oVirt
In-Reply-To: <50C91697.5040501@redhat.com>
Message-ID: <1478454618.4813202.1355356047110.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "Thierry Kauffmann" <thierry.kauffmann at univ-montp2.fr>, "Yaniv Kaul" <ykaul at redhat.com>, "engine-devel"
> <engine-devel at ovirt.org>
> Sent: Thursday, December 13, 2012 1:43:19 AM
> Subject: Re: Adding Authentication mechanism to oVirt
> 
> On 12/12/2012 06:53 PM, Alon Bar-Lev wrote:
> > Hello Thierry,
> >
> > If I understand correctly you wish to help in modifying the engine
> > to support non GSSAPI authentication methods.
> >
> > Following is a quick design goals for this implementation.
> >
> > I will be glad to improve this.
> 
> thanks for writing this part.
> what about making the LDAP part plugable, or at least
> scheme-configurable to cover openLDAP, or any other source (atlassian
> crowd), or well, anything?

I thought our primary goal is to reduce the noise around the kerberos.
Doing a complete re-write (aka modular) will consume far more resources and time.
It also touches many part of the implementation instead of only the LDAP context and authentication.

> 
> thanks,
>     Itamar
> 
> > Alon
> >
> > ---
> >
> > Implementation should support the following transports:
> >
> > 1. LDAP (plain).
> > 2. LDAP over TLS.
> > 3. LDAP with StartTLS.
> >
> > Implementation should support the following authentication methods:
> >
> > 1. Simple.
> > 2. Digest-MD5 (plain and strong).
> >
> > I believe the GSSAPI can be dropped, I see no advantage of using
> > it.
> >
> > A sample of low level implementation for transport and
> > authentication is attached.
> >
> > When adding a domain the following facts should be provided:
> >
> >   1. Search user name.
> >   2. Search user password.
> >   3. Transport type (ldap, ldaps, ldap+startTLS)
> >   4. Authentication (simple, Digest-MD5)
> >   5. Sever selection policy (failover, round-robin, random).
> >   6. Server address type (explicit, DNS record)
> >   7. Server address set.
> >   8. Optional base DN.
> >   9. Optional root certificate.
> > 10. Optional certificate chain.
> > 11. Search page size.
> > 10. Query timeout.
> > etc...
> >
> > Within product there are two separate components that perform LDAP
> > authentication:
> >
> > 1. User password validation.
> > 2. User permission fetch.
> >
> > These two components needs to work in share-nothing mode, meaning
> > that each should communicate with directory independently with the
> > other.
> >
> > USER PASSWORD VALIDATION
> >
> > Input: user
> > Input: domain
> > Input: password
> > Output: DN of user
> > Output: success/failure
> > Credentials used: user/password provided.
> > Notes: LDAP session should not be cached.
> > Logic: Perform LDAP bind.
> >
> > USER PERMISSION FETCH
> >
> > Input: DN of user (passed by user password validation)
> > Input: domain (passed by user password validation)
> > Output: A set of permissions
> > Credentials used: search user and password configured within
> > system.
> > Notes: LDAP context can be cached.
> > Logic: Perform LDAP searches, this is most of current logic.
> >
> 
> 
> 


From thierry.kauffmann at univ-montp2.fr  Thu Dec 13 07:14:19 2012
From: thierry.kauffmann at univ-montp2.fr (Thierry Kauffmann)
Date: Thu, 13 Dec 2012 08:14:19 +0100
Subject: [Engine-devel] Adding Authentication mechanism to oVirt
In-Reply-To: <1490596332.4736653.1355331210001.JavaMail.root@redhat.com>
References: <1490596332.4736653.1355331210001.JavaMail.root@redhat.com>
Message-ID: <50C9804B.2060608@univ-montp2.fr>

Le 12/12/2012 17:53, Alon Bar-Lev a ?crit :
> Hello Thierry,
>
> If I understand correctly you wish to help in modifying the engine to support non GSSAPI authentication methods.
Hi Alon,

you're right.
My main (selfish) goal is to adapt oVirt authentication schemes to our
requirements : that is "LDAP SIMPLE auth over startTLS."

> Following is a quick design goals for this implementation.
>
> I will be glad to improve this.
> Alon
Thanks a lot for this description.

Tell me how I could help : I am more a system administrator than a
developer. I kwon a bit of java (at least I can read and understand
oVirt's code that I downloaded recently).

Thierry

> ---
>
> Implementation should support the following transports:
>
> 1. LDAP (plain).
> 2. LDAP over TLS.
> 3. LDAP with StartTLS.
>
> Implementation should support the following authentication methods:
>
> 1. Simple.
> 2. Digest-MD5 (plain and strong).
>
> I believe the GSSAPI can be dropped, I see no advantage of using it.
>
> A sample of low level implementation for transport and authentication is attached.
>
> When adding a domain the following facts should be provided:
>
>  1. Search user name.
>  2. Search user password.
>  3. Transport type (ldap, ldaps, ldap+startTLS)
>  4. Authentication (simple, Digest-MD5)
>  5. Sever selection policy (failover, round-robin, random).
>  6. Server address type (explicit, DNS record)
>  7. Server address set.
>  8. Optional base DN.
>  9. Optional root certificate.
> 10. Optional certificate chain.
> 11. Search page size.
> 10. Query timeout.
> etc...
>
> Within product there are two separate components that perform LDAP authentication:
>
> 1. User password validation.
> 2. User permission fetch.
>
> These two components needs to work in share-nothing mode, meaning that each should communicate with directory independently with the other.
>
> USER PASSWORD VALIDATION
>
> Input: user
> Input: domain
> Input: password
> Output: DN of user
> Output: success/failure
> Credentials used: user/password provided.
> Notes: LDAP session should not be cached.
> Logic: Perform LDAP bind.
>
> USER PERMISSION FETCH
>
> Input: DN of user (passed by user password validation)
> Input: domain (passed by user password validation)
> Output: A set of permissions
> Credentials used: search user and password configured within system.
> Notes: LDAP context can be cached. 
> Logic: Perform LDAP searches, this is most of current logic.


-- 
signature-TK Thierry Kauffmann
Chef du Service Informatique // Facult? des Sciences // Universit? de
Montpellier 2

SIF - Service Informatique de la Facult? des Sciences
<http://sif.info-ufr.univ-montp2.fr/> 	UM2 - Universit? de Montpellier 2
<http://www.univ-montp2.fr/> 	Service informatique de la Facult? des
Sciences (SIF)
Universit? de Montpellier 2
CC437 // Place Eug?ne Bataillon // 34095 Montpellier Cedex 5

T?l : 04 67 14 31 58
email : thierry.kauffmann at univ-montp2.fr
<mailto:thierry.kauffmann at univ-montp2.fr>
web : http://sif.info-ufr.univ-montp2.fr/ 
http://www.fdsweb.univ-montp2.fr/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121213/e1e33b52/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sif.png
Type: image/png
Size: 11755 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121213/e1e33b52/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: um2.png
Type: image/png
Size: 29129 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121213/e1e33b52/attachment-0001.png>

From lhornyak at redhat.com  Thu Dec 13 08:33:04 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Thu, 13 Dec 2012 03:33:04 -0500 (EST)
Subject: [Engine-devel] patches waiting for review
In-Reply-To: <742546088.9369105.1355387502687.JavaMail.root@redhat.com>
Message-ID: <748794841.9369198.1355387584449.JavaMail.root@redhat.com>

Hi,

I have abandoned my patches that did not get +2 in a month, so now I have a reduced list of patches waiting for +2:
http://gerrit.ovirt.org/#/dashboard/1000027

Your feedback is welcome!

Thx,
Laszlo


From masayag at redhat.com  Thu Dec 13 10:25:07 2012
From: masayag at redhat.com (Moti Asayag)
Date: Thu, 13 Dec 2012 05:25:07 -0500 (EST)
Subject: [Engine-devel] Error on starting webadmin
In-Reply-To: <1732420003.15235128.1355393475360.JavaMail.root@redhat.com>
Message-ID: <506639344.15247293.1355394307247.JavaMail.root@redhat.com>

Hi,

I get the following error when trying to connect to webadmin:

2012-12-13 11:57:06,689 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[WebAdminDynamicHosting]] (http--0.0.0.0-8700-1) Servlet.service() for servlet WebAdminDynamicHosting threw exception: java.lang.IllegalArgumentException: The property "ENGINE_USR" doesn't have a value.

It caused by the PluginDataManager instantiation which requires several variables to be defined.

In order to by pass those errors do:
1. create a file /usr/share/ovirt-engine/conf/engine.conf.defaults
2. Add the following properties to the file:
ENGINE_USR=username
ENGINE_ETC=/etc/ovirt-engine
3. restart Jboss

Regards,
Moti


From alonbl at redhat.com  Thu Dec 13 10:42:09 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Thu, 13 Dec 2012 05:42:09 -0500 (EST)
Subject: [Engine-devel] Adding Authentication mechanism to oVirt
In-Reply-To: <50C9804B.2060608@univ-montp2.fr>
Message-ID: <730250965.4898662.1355395329517.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Thierry Kauffmann" <thierry.kauffmann at univ-montp2.fr>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "Yaniv Kaul" <ykaul at redhat.com>, "Itamar Heim" <iheim at redhat.com>, "engine-devel" <engine-devel at ovirt.org>
> Sent: Thursday, December 13, 2012 9:14:19 AM
> Subject: Re: Adding Authentication mechanism to oVirt
> 
> 
> Le 12/12/2012 17:53, Alon Bar-Lev a ?crit :
> 
> 
>> Hello Thierry,
> 
>> If I understand correctly you wish to help in modifying the engine to
>> support non GSSAPI authentication methods. Hi Alon,
> 
> you're right.
> My main (selfish) goal is to adapt oVirt authentication schemes to
> our requirements : that is "LDAP SIMPLE auth over startTLS."
>
>> Following is a quick design goals for this implementation.
> 
> I will be glad to improve this.
> Alon Thanks a lot for this description.
> 
> Tell me how I could help : I am more a system administrator than a
> developer. I kwon a bit of java (at least I can read and understand
> oVirt's code that I downloaded recently).
> 
> Thierry

Well, nobody is working on this one, so helping is actually implementing that... I believe it will be difficult to do so without some java and/or infrastructure background... but it is a great challenge!

Alon


From rgolan at redhat.com  Thu Dec 13 10:42:33 2012
From: rgolan at redhat.com (Roy Golan)
Date: Thu, 13 Dec 2012 12:42:33 +0200
Subject: [Engine-devel] Error on starting webadmin
In-Reply-To: <506639344.15247293.1355394307247.JavaMail.root@redhat.com>
References: <506639344.15247293.1355394307247.JavaMail.root@redhat.com>
Message-ID: <50C9B119.20705@redhat.com>

On 12/13/2012 12:25 PM, Moti Asayag wrote:
> Hi,
>
> I get the following error when trying to connect to webadmin:
>
> 2012-12-13 11:57:06,689 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[WebAdminDynamicHosting]] (http--0.0.0.0-8700-1) Servlet.service() for servlet WebAdminDynamicHosting threw exception: java.lang.IllegalArgumentException: The property "ENGINE_USR" doesn't have a value.
>
> It caused by the PluginDataManager instantiation which requires several variables to be defined.
>
> In order to by pass those errors do:
> 1. create a file /usr/share/ovirt-engine/conf/engine.conf.defaults
> 2. Add the following properties to the file:
> ENGINE_USR=username
> ENGINE_ETC=/etc/ovirt-engine
> 3. restart Jboss
thanks its working. I think we need to add that to "setup" target of the 
ear so a developer could get up & running
> Regards,
> Moti
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel



From derez at redhat.com  Thu Dec 13 10:49:13 2012
From: derez at redhat.com (Daniel Erez)
Date: Thu, 13 Dec 2012 05:49:13 -0500 (EST)
Subject: [Engine-devel] Error on starting webadmin
In-Reply-To: <506639344.15247293.1355394307247.JavaMail.root@redhat.com>
Message-ID: <733602608.558541.1355395753122.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Moti Asayag" <masayag at redhat.com>
> To: engine-devel at ovirt.org
> Sent: Thursday, December 13, 2012 12:25:07 PM
> Subject: [Engine-devel] Error on starting webadmin
> 
> Hi,
> 
> I get the following error when trying to connect to webadmin:
> 
> 2012-12-13 11:57:06,689 ERROR
> [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[WebAdminDynamicHosting]]
> (http--0.0.0.0-8700-1) Servlet.service() for servlet
> WebAdminDynamicHosting threw exception:
> java.lang.IllegalArgumentException: The property "ENGINE_USR"
> doesn't have a value.
> 
> It caused by the PluginDataManager instantiation which requires
> several variables to be defined.
> 
> In order to by pass those errors do:
> 1. create a file /usr/share/ovirt-engine/conf/engine.conf.defaults
> 2. Add the following properties to the file:
> ENGINE_USR=username
> ENGINE_ETC=/etc/ovirt-engine
> 3. restart Jboss

You can use the 'engine.conf.defaults' that resides at 'backend/manager/conf'.

> 
> Regards,
> Moti
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From rgolan at redhat.com  Thu Dec 13 11:21:51 2012
From: rgolan at redhat.com (Roy Golan)
Date: Thu, 13 Dec 2012 13:21:51 +0200
Subject: [Engine-devel] Error on starting webadmin
In-Reply-To: <50C9B119.20705@redhat.com>
References: <506639344.15247293.1355394307247.JavaMail.root@redhat.com>
	<50C9B119.20705@redhat.com>
Message-ID: <50C9BA4F.9050305@redhat.com>

On 12/13/2012 12:42 PM, Roy Golan wrote:
> On 12/13/2012 12:25 PM, Moti Asayag wrote:
>> Hi,
>>
>> I get the following error when trying to connect to webadmin:
>>
>> 2012-12-13 11:57:06,689 ERROR 
>> [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[WebAdminDynamicHosting]] 
>> (http--0.0.0.0-8700-1) Servlet.service() for servlet 
>> WebAdminDynamicHosting threw exception: 
>> java.lang.IllegalArgumentException: The property "ENGINE_USR" doesn't 
>> have a value.
>>
>> It caused by the PluginDataManager instantiation which requires 
>> several variables to be defined.
>>
>> In order to by pass those errors do:
>> 1. create a file /usr/share/ovirt-engine/conf/engine.conf.defaults
>> 2. Add the following properties to the file:
>> ENGINE_USR=username
>> ENGINE_ETC=/etc/ovirt-engine
>> 3. restart Jboss
> thanks its working. I think we need to add that to "setup" target of 
> the ear so a developer could get up & running
http://gerrit.ovirt.org/#/c/10026/
>> Regards,
>> Moti
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel



From vszocs at redhat.com  Thu Dec 13 13:49:39 2012
From: vszocs at redhat.com (Vojtech Szocs)
Date: Thu, 13 Dec 2012 08:49:39 -0500 (EST)
Subject: [Engine-devel] Error on starting webadmin
In-Reply-To: <733602608.558541.1355395753122.JavaMail.root@redhat.com>
Message-ID: <151530555.18075758.1355406579606.JavaMail.root@redhat.com>

Hi guys,

as Moti mentioned, this issue is related to UI plugins patches that have been merged recently.

However, the root cause of this issue is LocalConfig class (machine-specific Engine configuration) throwing exception in following cases:
- cannot load default properties (ENGINE_DEFAULTS property if defined, otherwise /usr/share/ovirt-engine/conf/engine.conf.defaults)
- cannot load custom properties (ENGINE_VARS property if defined, otherwise /etc/sysconfig/ovirt-engine)
- requesting missing property value (LocalConfig.getProperty method)

(Note that UI plugins use two Engine properties: ENGINE_USR and ENGINE_ETC.)

As Juan suggests in his comment in http://gerrit.ovirt.org/#/c/10026/, we could include this in "building Engine" wiki. For example:
- have ENGINE_DEFAULTS property point to OVIRT_HOME/backend/manager/conf/engine.conf.defaults (alternatively, users can copy this file to /usr/share/ovirt-engine/conf)
- create custom Engine config file, e.g. ~/myengine.conf, and have ENGINE_VARS property point to this file

Vojtech


----- Original Message -----
From: "Daniel Erez" <derez at redhat.com>
To: "Moti Asayag" <masayag at redhat.com>
Cc: engine-devel at ovirt.org
Sent: Thursday, December 13, 2012 11:49:13 AM
Subject: Re: [Engine-devel] Error on starting webadmin



----- Original Message -----
> From: "Moti Asayag" <masayag at redhat.com>
> To: engine-devel at ovirt.org
> Sent: Thursday, December 13, 2012 12:25:07 PM
> Subject: [Engine-devel] Error on starting webadmin
> 
> Hi,
> 
> I get the following error when trying to connect to webadmin:
> 
> 2012-12-13 11:57:06,689 ERROR
> [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[WebAdminDynamicHosting]]
> (http--0.0.0.0-8700-1) Servlet.service() for servlet
> WebAdminDynamicHosting threw exception:
> java.lang.IllegalArgumentException: The property "ENGINE_USR"
> doesn't have a value.
> 
> It caused by the PluginDataManager instantiation which requires
> several variables to be defined.
> 
> In order to by pass those errors do:
> 1. create a file /usr/share/ovirt-engine/conf/engine.conf.defaults
> 2. Add the following properties to the file:
> ENGINE_USR=username
> ENGINE_ETC=/etc/ovirt-engine
> 3. restart Jboss

You can use the 'engine.conf.defaults' that resides at 'backend/manager/conf'.

> 
> Regards,
> Moti
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 
_______________________________________________
Engine-devel mailing list
Engine-devel at ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


From vszocs at redhat.com  Thu Dec 13 14:32:02 2012
From: vszocs at redhat.com (Vojtech Szocs)
Date: Thu, 13 Dec 2012 09:32:02 -0500 (EST)
Subject: [Engine-devel] Error on starting webadmin
In-Reply-To: <151530555.18075758.1355406579606.JavaMail.root@redhat.com>
Message-ID: <1514512035.18102730.1355409122132.JavaMail.root@redhat.com>

Hi,

I've updated http://www.ovirt.org/Building_Engine_Draft - added section "Engine local configuration" (feel free to edit or modify as necessary)

Vojtech


----- Original Message -----
From: "Vojtech Szocs" <vszocs at redhat.com>
To: engine-devel at ovirt.org
Sent: Thursday, December 13, 2012 2:49:39 PM
Subject: Re: [Engine-devel] Error on starting webadmin

Hi guys,

as Moti mentioned, this issue is related to UI plugins patches that have been merged recently.

However, the root cause of this issue is LocalConfig class (machine-specific Engine configuration) throwing exception in following cases:
- cannot load default properties (ENGINE_DEFAULTS property if defined, otherwise /usr/share/ovirt-engine/conf/engine.conf.defaults)
- cannot load custom properties (ENGINE_VARS property if defined, otherwise /etc/sysconfig/ovirt-engine)
- requesting missing property value (LocalConfig.getProperty method)

(Note that UI plugins use two Engine properties: ENGINE_USR and ENGINE_ETC.)

As Juan suggests in his comment in http://gerrit.ovirt.org/#/c/10026/, we could include this in "building Engine" wiki. For example:
- have ENGINE_DEFAULTS property point to OVIRT_HOME/backend/manager/conf/engine.conf.defaults (alternatively, users can copy this file to /usr/share/ovirt-engine/conf)
- create custom Engine config file, e.g. ~/myengine.conf, and have ENGINE_VARS property point to this file

Vojtech


----- Original Message -----
From: "Daniel Erez" <derez at redhat.com>
To: "Moti Asayag" <masayag at redhat.com>
Cc: engine-devel at ovirt.org
Sent: Thursday, December 13, 2012 11:49:13 AM
Subject: Re: [Engine-devel] Error on starting webadmin



----- Original Message -----
> From: "Moti Asayag" <masayag at redhat.com>
> To: engine-devel at ovirt.org
> Sent: Thursday, December 13, 2012 12:25:07 PM
> Subject: [Engine-devel] Error on starting webadmin
> 
> Hi,
> 
> I get the following error when trying to connect to webadmin:
> 
> 2012-12-13 11:57:06,689 ERROR
> [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[WebAdminDynamicHosting]]
> (http--0.0.0.0-8700-1) Servlet.service() for servlet
> WebAdminDynamicHosting threw exception:
> java.lang.IllegalArgumentException: The property "ENGINE_USR"
> doesn't have a value.
> 
> It caused by the PluginDataManager instantiation which requires
> several variables to be defined.
> 
> In order to by pass those errors do:
> 1. create a file /usr/share/ovirt-engine/conf/engine.conf.defaults
> 2. Add the following properties to the file:
> ENGINE_USR=username
> ENGINE_ETC=/etc/ovirt-engine
> 3. restart Jboss

You can use the 'engine.conf.defaults' that resides at 'backend/manager/conf'.

> 
> Regards,
> Moti
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 
_______________________________________________
Engine-devel mailing list
Engine-devel at ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________
Engine-devel mailing list
Engine-devel at ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


From rgolan at redhat.com  Thu Dec 13 14:36:40 2012
From: rgolan at redhat.com (Roy Golan)
Date: Thu, 13 Dec 2012 16:36:40 +0200
Subject: [Engine-devel] Error on starting webadmin
In-Reply-To: <1514512035.18102730.1355409122132.JavaMail.root@redhat.com>
References: <1514512035.18102730.1355409122132.JavaMail.root@redhat.com>
Message-ID: <50C9E7F8.6090907@redhat.com>

On 12/13/2012 04:32 PM, Vojtech Szocs wrote:
> Hi,
>
> I've updated http://www.ovirt.org/Building_Engine_Draft - added section "Engine local configuration" (feel free to edit or modify as necessary)
this is making the contribution process more complex. lets think of a 
lighter way to get a developing setup.

>
> Vojtech
>
>
> ----- Original Message -----
> From: "Vojtech Szocs" <vszocs at redhat.com>
> To: engine-devel at ovirt.org
> Sent: Thursday, December 13, 2012 2:49:39 PM
> Subject: Re: [Engine-devel] Error on starting webadmin
>
> Hi guys,
>
> as Moti mentioned, this issue is related to UI plugins patches that have been merged recently.
>
> However, the root cause of this issue is LocalConfig class (machine-specific Engine configuration) throwing exception in following cases:
> - cannot load default properties (ENGINE_DEFAULTS property if defined, otherwise /usr/share/ovirt-engine/conf/engine.conf.defaults)
> - cannot load custom properties (ENGINE_VARS property if defined, otherwise /etc/sysconfig/ovirt-engine)
> - requesting missing property value (LocalConfig.getProperty method)
>
> (Note that UI plugins use two Engine properties: ENGINE_USR and ENGINE_ETC.)
>
> As Juan suggests in his comment in http://gerrit.ovirt.org/#/c/10026/, we could include this in "building Engine" wiki. For example:
> - have ENGINE_DEFAULTS property point to OVIRT_HOME/backend/manager/conf/engine.conf.defaults (alternatively, users can copy this file to /usr/share/ovirt-engine/conf)
> - create custom Engine config file, e.g. ~/myengine.conf, and have ENGINE_VARS property point to this file
>
> Vojtech
>
>
> ----- Original Message -----
> From: "Daniel Erez" <derez at redhat.com>
> To: "Moti Asayag" <masayag at redhat.com>
> Cc: engine-devel at ovirt.org
> Sent: Thursday, December 13, 2012 11:49:13 AM
> Subject: Re: [Engine-devel] Error on starting webadmin
>
>
>
> ----- Original Message -----
>> From: "Moti Asayag" <masayag at redhat.com>
>> To: engine-devel at ovirt.org
>> Sent: Thursday, December 13, 2012 12:25:07 PM
>> Subject: [Engine-devel] Error on starting webadmin
>>
>> Hi,
>>
>> I get the following error when trying to connect to webadmin:
>>
>> 2012-12-13 11:57:06,689 ERROR
>> [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[WebAdminDynamicHosting]]
>> (http--0.0.0.0-8700-1) Servlet.service() for servlet
>> WebAdminDynamicHosting threw exception:
>> java.lang.IllegalArgumentException: The property "ENGINE_USR"
>> doesn't have a value.
>>
>> It caused by the PluginDataManager instantiation which requires
>> several variables to be defined.
>>
>> In order to by pass those errors do:
>> 1. create a file /usr/share/ovirt-engine/conf/engine.conf.defaults
>> 2. Add the following properties to the file:
>> ENGINE_USR=username
>> ENGINE_ETC=/etc/ovirt-engine
>> 3. restart Jboss
> You can use the 'engine.conf.defaults' that resides at 'backend/manager/conf'.
>
>> Regards,
>> Moti
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>>
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel



From vszocs at redhat.com  Thu Dec 13 14:42:34 2012
From: vszocs at redhat.com (Vojtech Szocs)
Date: Thu, 13 Dec 2012 09:42:34 -0500 (EST)
Subject: [Engine-devel] UI plugins - next steps
In-Reply-To: <616374438.60989790.1355319265885.JavaMail.root@redhat.com>
Message-ID: <154730274.18108162.1355409754977.JavaMail.root@redhat.com>

Hi Oved,

> Pushing this now.
> http://gerrit.ovirt.org/#/c/9998/

Thanks!

> I think we should also consider adding an option to add a button in a more general scope, for things that aren't entity-specific, but system-specific.
> Maybe next to the "configure" button, in the upper right corner of the screen?

I agree, this could be useful as well. Shouldn't be too hard to implement, it's just about adding dynamic links to HeaderPresenterWidget/HeaderView in WebAdmin.

Vojtech


----- Original Message -----
From: "Oved Ourfalli" <ovedo at redhat.com>
To: "Vojtech Szocs" <vszocs at redhat.com>
Cc: "engine-devel" <engine-devel at ovirt.org>
Sent: Wednesday, December 12, 2012 2:34:25 PM
Subject: Re: [Engine-devel] UI plugins - next steps

Nice work!

See some comments inline.


----- Original Message -----
> From: "Vojtech Szocs" <vszocs at redhat.com>
> To: "engine-devel" <engine-devel at ovirt.org>
> Sent: Wednesday, December 12, 2012 2:57:11 PM
> Subject: [Engine-devel] UI plugins - next steps
> 
> Hi guys,
> 
> here's my list of tasks that didn't make it into UI plugins yet.
> These tasks should be revisited and implemented in future patches.
> The list isn't sorted in any way, so feel free to comment or
> highlight specific tasks to raise their priority, or add your own
> tasks/ideas.
> 
> 1. Pass proper (restapi-definition) entities to UI plugins, instead
> of simple "{entityId:<guidAsString>}" objects
>    - using restapi-types mappers to map backend business entities to
>    restapi-definition entities (Java POJOs generated from api.xsd
>    via JAXB)
>    - exporting restapi-definition entities for use with JavaScript
>    (UI plugins), e.g. using gwt-exporter [1] (alternatively, we
>    could use GWT deferred binding and do this ourselves)
> 
> 2. showDialog API function should integrate with WebAdmin dialog
> infrastructure
>    - shouldn't be too hard, need to create custom dialog
>    PresenterWidget/View and bind it as non-singleton (just like
>    other popups in oVirt GUI)
> 
> 3. Extend existing WebAdmin dialogs - prototype Doron's proposal to
> extend Edit Cluster Policy dialog with custom options that would be
> passed to backend
> 
> 4. Introduce form-based tabs - custom tab that would display
> key/value pairs organized in columns, just like in standard
> "General" sub tabs
> 
> 5. Introduce table-based tabs - custom tab that would display grid
> (table) just like in standard main tabs, with API to add columns,
> set data, etc.
> 
> 6. Improve api.addMainTabActionButton to allow different button vs.
> context menu item representations (only button, only menu item,
> both)
> 
> 7. Add api.currentUserName and api.currentUserId API functions (these
> would return same data as with UserLogin event parameters)
> 
Pushing this now.
http://gerrit.ovirt.org/#/c/9998/

> 8. Minor things: consider renaming UiInit to PluginInit (one-time
> plugin initialization, called only once, can do non-UI init stuff
> there as well), consider renaming api.ready to api.initialize
> (plugin indicates that infrastructure can proceed with initializing
> itself)
> 

I think we should also consider adding an option to add a button in a more general scope, for things that aren't entity-specific, but system-specific.
Maybe next to the "configure" button, in the upper right corner of the screen?

> Regards,
> Vojtech
> 
> [1] http://code.google.com/p/gwt-exporter/wiki/GettingStarted
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From vszocs at redhat.com  Thu Dec 13 14:49:51 2012
From: vszocs at redhat.com (Vojtech Szocs)
Date: Thu, 13 Dec 2012 09:49:51 -0500 (EST)
Subject: [Engine-devel] Error on starting webadmin
In-Reply-To: <50C9E7F8.6090907@redhat.com>
Message-ID: <1522414004.18111689.1355410191913.JavaMail.root@redhat.com>

> this is making the contribution process more complex. lets think of a 
> lighter way to get a developing setup.

I agree, I just wanted to have the local Engine configuration steps documented for reference.. If there's a better way to do it, I'm for it.

Vojtech


----- Original Message -----
From: "Roy Golan" <rgolan at redhat.com>
To: engine-devel at ovirt.org
Sent: Thursday, December 13, 2012 3:36:40 PM
Subject: Re: [Engine-devel] Error on starting webadmin

On 12/13/2012 04:32 PM, Vojtech Szocs wrote:
> Hi,
>
> I've updated http://www.ovirt.org/Building_Engine_Draft - added section "Engine local configuration" (feel free to edit or modify as necessary)
this is making the contribution process more complex. lets think of a 
lighter way to get a developing setup.

>
> Vojtech
>
>
> ----- Original Message -----
> From: "Vojtech Szocs" <vszocs at redhat.com>
> To: engine-devel at ovirt.org
> Sent: Thursday, December 13, 2012 2:49:39 PM
> Subject: Re: [Engine-devel] Error on starting webadmin
>
> Hi guys,
>
> as Moti mentioned, this issue is related to UI plugins patches that have been merged recently.
>
> However, the root cause of this issue is LocalConfig class (machine-specific Engine configuration) throwing exception in following cases:
> - cannot load default properties (ENGINE_DEFAULTS property if defined, otherwise /usr/share/ovirt-engine/conf/engine.conf.defaults)
> - cannot load custom properties (ENGINE_VARS property if defined, otherwise /etc/sysconfig/ovirt-engine)
> - requesting missing property value (LocalConfig.getProperty method)
>
> (Note that UI plugins use two Engine properties: ENGINE_USR and ENGINE_ETC.)
>
> As Juan suggests in his comment in http://gerrit.ovirt.org/#/c/10026/, we could include this in "building Engine" wiki. For example:
> - have ENGINE_DEFAULTS property point to OVIRT_HOME/backend/manager/conf/engine.conf.defaults (alternatively, users can copy this file to /usr/share/ovirt-engine/conf)
> - create custom Engine config file, e.g. ~/myengine.conf, and have ENGINE_VARS property point to this file
>
> Vojtech
>
>
> ----- Original Message -----
> From: "Daniel Erez" <derez at redhat.com>
> To: "Moti Asayag" <masayag at redhat.com>
> Cc: engine-devel at ovirt.org
> Sent: Thursday, December 13, 2012 11:49:13 AM
> Subject: Re: [Engine-devel] Error on starting webadmin
>
>
>
> ----- Original Message -----
>> From: "Moti Asayag" <masayag at redhat.com>
>> To: engine-devel at ovirt.org
>> Sent: Thursday, December 13, 2012 12:25:07 PM
>> Subject: [Engine-devel] Error on starting webadmin
>>
>> Hi,
>>
>> I get the following error when trying to connect to webadmin:
>>
>> 2012-12-13 11:57:06,689 ERROR
>> [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[WebAdminDynamicHosting]]
>> (http--0.0.0.0-8700-1) Servlet.service() for servlet
>> WebAdminDynamicHosting threw exception:
>> java.lang.IllegalArgumentException: The property "ENGINE_USR"
>> doesn't have a value.
>>
>> It caused by the PluginDataManager instantiation which requires
>> several variables to be defined.
>>
>> In order to by pass those errors do:
>> 1. create a file /usr/share/ovirt-engine/conf/engine.conf.defaults
>> 2. Add the following properties to the file:
>> ENGINE_USR=username
>> ENGINE_ETC=/etc/ovirt-engine
>> 3. restart Jboss
> You can use the 'engine.conf.defaults' that resides at 'backend/manager/conf'.
>
>> Regards,
>> Moti
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>>
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel

_______________________________________________
Engine-devel mailing list
Engine-devel at ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


From sanjal at redhat.com  Thu Dec 13 14:52:21 2012
From: sanjal at redhat.com (Shireesh Anjal)
Date: Thu, 13 Dec 2012 20:22:21 +0530
Subject: [Engine-devel] Error on starting webadmin
In-Reply-To: <50C9E7F8.6090907@redhat.com>
References: <1514512035.18102730.1355409122132.JavaMail.root@redhat.com>
	<50C9E7F8.6090907@redhat.com>
Message-ID: <50C9EBA5.1050704@redhat.com>

On Thursday 13 December 2012 08:06 PM, Roy Golan wrote:
> On 12/13/2012 04:32 PM, Vojtech Szocs wrote:
>> Hi,
>>
>> I've updated http://www.ovirt.org/Building_Engine_Draft - added 
>> section "Engine local configuration" (feel free to edit or modify as 
>> necessary)
> this is making the contribution process more complex. lets think of a 
> lighter way to get a developing setup.

+1

>
>>
>> Vojtech
>>
>>
>> ----- Original Message -----
>> From: "Vojtech Szocs" <vszocs at redhat.com>
>> To: engine-devel at ovirt.org
>> Sent: Thursday, December 13, 2012 2:49:39 PM
>> Subject: Re: [Engine-devel] Error on starting webadmin
>>
>> Hi guys,
>>
>> as Moti mentioned, this issue is related to UI plugins patches that 
>> have been merged recently.
>>
>> However, the root cause of this issue is LocalConfig class 
>> (machine-specific Engine configuration) throwing exception in 
>> following cases:
>> - cannot load default properties (ENGINE_DEFAULTS property if 
>> defined, otherwise /usr/share/ovirt-engine/conf/engine.conf.defaults)
>> - cannot load custom properties (ENGINE_VARS property if defined, 
>> otherwise /etc/sysconfig/ovirt-engine)
>> - requesting missing property value (LocalConfig.getProperty method)
>>
>> (Note that UI plugins use two Engine properties: ENGINE_USR and 
>> ENGINE_ETC.)
>>
>> As Juan suggests in his comment in 
>> http://gerrit.ovirt.org/#/c/10026/, we could include this in 
>> "building Engine" wiki. For example:
>> - have ENGINE_DEFAULTS property point to 
>> OVIRT_HOME/backend/manager/conf/engine.conf.defaults (alternatively, 
>> users can copy this file to /usr/share/ovirt-engine/conf)
>> - create custom Engine config file, e.g. ~/myengine.conf, and have 
>> ENGINE_VARS property point to this file
>>
>> Vojtech
>>
>>
>> ----- Original Message -----
>> From: "Daniel Erez" <derez at redhat.com>
>> To: "Moti Asayag" <masayag at redhat.com>
>> Cc: engine-devel at ovirt.org
>> Sent: Thursday, December 13, 2012 11:49:13 AM
>> Subject: Re: [Engine-devel] Error on starting webadmin
>>
>>
>>
>> ----- Original Message -----
>>> From: "Moti Asayag" <masayag at redhat.com>
>>> To: engine-devel at ovirt.org
>>> Sent: Thursday, December 13, 2012 12:25:07 PM
>>> Subject: [Engine-devel] Error on starting webadmin
>>>
>>> Hi,
>>>
>>> I get the following error when trying to connect to webadmin:
>>>
>>> 2012-12-13 11:57:06,689 ERROR
>>> [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[WebAdminDynamicHosting]] 
>>>
>>> (http--0.0.0.0-8700-1) Servlet.service() for servlet
>>> WebAdminDynamicHosting threw exception:
>>> java.lang.IllegalArgumentException: The property "ENGINE_USR"
>>> doesn't have a value.
>>>
>>> It caused by the PluginDataManager instantiation which requires
>>> several variables to be defined.
>>>
>>> In order to by pass those errors do:
>>> 1. create a file /usr/share/ovirt-engine/conf/engine.conf.defaults
>>> 2. Add the following properties to the file:
>>> ENGINE_USR=username
>>> ENGINE_ETC=/etc/ovirt-engine
>>> 3. restart Jboss
>> You can use the 'engine.conf.defaults' that resides at 
>> 'backend/manager/conf'.
>>
>>> Regards,
>>> Moti
>>> _______________________________________________
>>> Engine-devel mailing list
>>> Engine-devel at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>>>
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel



From rgolan at redhat.com  Thu Dec 13 14:55:48 2012
From: rgolan at redhat.com (Roy Golan)
Date: Thu, 13 Dec 2012 16:55:48 +0200
Subject: [Engine-devel] Error on starting webadmin
In-Reply-To: <1522414004.18111689.1355410191913.JavaMail.root@redhat.com>
References: <1522414004.18111689.1355410191913.JavaMail.root@redhat.com>
Message-ID: <50C9EC74.6030902@redhat.com>

On 12/13/2012 04:49 PM, Vojtech Szocs wrote:
>> this is making the contribution process more complex. lets think of a
>> lighter way to get a developing setup.
> I agree, I just wanted to have the local Engine configuration steps documented for reference.. If there's a better way to do it, I'm for it.
>
> Vojtech

having LocalConfig look for "engine.conf.defaults" system property 
before fallback to System.getenv("ENGINE_DEFAULTS");
  +  concatenating -Dengine.conf.defaults=$HOME/.engine.conf.defaults 
to  JAVA_OPTS  on standalone.conf
>
>
> ----- Original Message -----
> From: "Roy Golan" <rgolan at redhat.com>
> To: engine-devel at ovirt.org
> Sent: Thursday, December 13, 2012 3:36:40 PM
> Subject: Re: [Engine-devel] Error on starting webadmin
>
> On 12/13/2012 04:32 PM, Vojtech Szocs wrote:
>> Hi,
>>
>> I've updated http://www.ovirt.org/Building_Engine_Draft - added section "Engine local configuration" (feel free to edit or modify as necessary)
> this is making the contribution process more complex. lets think of a
> lighter way to get a developing setup.
>
>> Vojtech
>>
>>
>> ----- Original Message -----
>> From: "Vojtech Szocs" <vszocs at redhat.com>
>> To: engine-devel at ovirt.org
>> Sent: Thursday, December 13, 2012 2:49:39 PM
>> Subject: Re: [Engine-devel] Error on starting webadmin
>>
>> Hi guys,
>>
>> as Moti mentioned, this issue is related to UI plugins patches that have been merged recently.
>>
>> However, the root cause of this issue is LocalConfig class (machine-specific Engine configuration) throwing exception in following cases:
>> - cannot load default properties (ENGINE_DEFAULTS property if defined, otherwise /usr/share/ovirt-engine/conf/engine.conf.defaults)
>> - cannot load custom properties (ENGINE_VARS property if defined, otherwise /etc/sysconfig/ovirt-engine)
>> - requesting missing property value (LocalConfig.getProperty method)
>>
>> (Note that UI plugins use two Engine properties: ENGINE_USR and ENGINE_ETC.)
>>
>> As Juan suggests in his comment in http://gerrit.ovirt.org/#/c/10026/, we could include this in "building Engine" wiki. For example:
>> - have ENGINE_DEFAULTS property point to OVIRT_HOME/backend/manager/conf/engine.conf.defaults (alternatively, users can copy this file to /usr/share/ovirt-engine/conf)
>> - create custom Engine config file, e.g. ~/myengine.conf, and have ENGINE_VARS property point to this file
>>
>> Vojtech
>>
>>
>> ----- Original Message -----
>> From: "Daniel Erez" <derez at redhat.com>
>> To: "Moti Asayag" <masayag at redhat.com>
>> Cc: engine-devel at ovirt.org
>> Sent: Thursday, December 13, 2012 11:49:13 AM
>> Subject: Re: [Engine-devel] Error on starting webadmin
>>
>>
>>
>> ----- Original Message -----
>>> From: "Moti Asayag" <masayag at redhat.com>
>>> To: engine-devel at ovirt.org
>>> Sent: Thursday, December 13, 2012 12:25:07 PM
>>> Subject: [Engine-devel] Error on starting webadmin
>>>
>>> Hi,
>>>
>>> I get the following error when trying to connect to webadmin:
>>>
>>> 2012-12-13 11:57:06,689 ERROR
>>> [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[WebAdminDynamicHosting]]
>>> (http--0.0.0.0-8700-1) Servlet.service() for servlet
>>> WebAdminDynamicHosting threw exception:
>>> java.lang.IllegalArgumentException: The property "ENGINE_USR"
>>> doesn't have a value.
>>>
>>> It caused by the PluginDataManager instantiation which requires
>>> several variables to be defined.
>>>
>>> In order to by pass those errors do:
>>> 1. create a file /usr/share/ovirt-engine/conf/engine.conf.defaults
>>> 2. Add the following properties to the file:
>>> ENGINE_USR=username
>>> ENGINE_ETC=/etc/ovirt-engine
>>> 3. restart Jboss
>> You can use the 'engine.conf.defaults' that resides at 'backend/manager/conf'.
>>
>>> Regards,
>>> Moti
>>> _______________________________________________
>>> Engine-devel mailing list
>>> Engine-devel at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>>>
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel



From vszocs at redhat.com  Thu Dec 13 15:37:22 2012
From: vszocs at redhat.com (Vojtech Szocs)
Date: Thu, 13 Dec 2012 10:37:22 -0500 (EST)
Subject: [Engine-devel] UI plugins - next steps
In-Reply-To: <50C915BA.6090607@redhat.com>
Message-ID: <131151679.18130744.1355413042408.JavaMail.root@redhat.com>

Hi Itamar,

> I think its time we revisit the implications/scope of moving the UI to 
> work on top of the REST API or just the REST entities.
> until then, i think just adding specific fields that make sense/are 
> asked for for these entities should be good enough.

Sounds reasonable, given that we want to move UI on top of REST API anyway.

> if we see the "moving UI on top of REST API" is still too far, then 
> doing the type mapping for just the plugins would make sense.

Maybe the "moving UI on top of REST API" process could be split into two phases:

1) patching UiCommon code to work with generated REST API types, instead of internal business entities
   patching Frontend RPC bridge (still using Generic API) to map internal business entities to REST API types

2) patching Frontend RPC bridge to map action/query invocation to REST API HTTP requests
   patching Frontend RPC bridge to map REST API HTTP responses back to REST API types

Doing both 1) and 2) in one sweep might be risky..

Vojtech


----- Original Message -----
From: "Itamar Heim" <iheim at redhat.com>
To: "Vojtech Szocs" <vszocs at redhat.com>
Cc: "engine-devel" <engine-devel at ovirt.org>
Sent: Thursday, December 13, 2012 12:39:38 AM
Subject: Re: [Engine-devel] UI plugins - next steps

On 12/12/2012 02:57 PM, Vojtech Szocs wrote:
> 1. Pass proper (restapi-definition) entities to UI plugins, instead of simple "{entityId:<guidAsString>}" objects
>     - using restapi-types mappers to map backend business entities to restapi-definition entities (Java POJOs generated from api.xsd via JAXB)
>     - exporting restapi-definition entities for use with JavaScript (UI plugins), e.g. using gwt-exporter [1] (alternatively, we could use GWT deferred binding and do this ourselves)

I think its time we revisit the implications/scope of moving the UI to 
work on top of the REST API or just the REST entities.
until then, i think just adding specific fields that make sense/are 
asked for for these entities should be good enough.
if we see the "moving UI on top of REST API" is still too far, then 
doing the type mapping for just the plugins would make sense.



From iheim at redhat.com  Thu Dec 13 16:06:36 2012
From: iheim at redhat.com (Itamar Heim)
Date: Thu, 13 Dec 2012 18:06:36 +0200
Subject: [Engine-devel] UI plugins - next steps
In-Reply-To: <131151679.18130744.1355413042408.JavaMail.root@redhat.com>
References: <131151679.18130744.1355413042408.JavaMail.root@redhat.com>
Message-ID: <50C9FD0C.8080305@redhat.com>

On 12/13/2012 05:37 PM, Vojtech Szocs wrote:
> Hi Itamar,
>
>> I think its time we revisit the implications/scope of moving the UI to
>> work on top of the REST API or just the REST entities.
>> until then, i think just adding specific fields that make sense/are
>> asked for for these entities should be good enough.
>
> Sounds reasonable, given that we want to move UI on top of REST API anyway.
>
>> if we see the "moving UI on top of REST API" is still too far, then
>> doing the type mapping for just the plugins would make sense.
>
> Maybe the "moving UI on top of REST API" process could be split into two phases:
>
> 1) patching UiCommon code to work with generated REST API types, instead of internal business entities
>     patching Frontend RPC bridge (still using Generic API) to map internal business entities to REST API types
>
> 2) patching Frontend RPC bridge to map action/query invocation to REST API HTTP requests
>     patching Frontend RPC bridge to map REST API HTTP responses back to REST API types
>
> Doing both 1) and 2) in one sweep might be risky..

agree, and this would be the general idea.
though once you start changing entities in the uicommon, it's kind of 
infecting.
i suggest to start with a simple one (say, bookmark, then config, then 
tag, etc.).

you will then find some queries don't return business entities though, 
rather more complex types.
so you could also start by mapping all the quries not returning a simple 
list of entities.

>
> Vojtech
>
>
> ----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Vojtech Szocs" <vszocs at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Thursday, December 13, 2012 12:39:38 AM
> Subject: Re: [Engine-devel] UI plugins - next steps
>
> On 12/12/2012 02:57 PM, Vojtech Szocs wrote:
>> 1. Pass proper (restapi-definition) entities to UI plugins, instead of simple "{entityId:<guidAsString>}" objects
>>      - using restapi-types mappers to map backend business entities to restapi-definition entities (Java POJOs generated from api.xsd via JAXB)
>>      - exporting restapi-definition entities for use with JavaScript (UI plugins), e.g. using gwt-exporter [1] (alternatively, we could use GWT deferred binding and do this ourselves)
>
> I think its time we revisit the implications/scope of moving the UI to
> work on top of the REST API or just the REST entities.
> until then, i think just adding specific fields that make sense/are
> asked for for these entities should be good enough.
> if we see the "moving UI on top of REST API" is still too far, then
> doing the type mapping for just the plugins would make sense.
>




From jhernand at redhat.com  Fri Dec 14 11:25:57 2012
From: jhernand at redhat.com (Juan Hernandez)
Date: Fri, 14 Dec 2012 12:25:57 +0100
Subject: [Engine-devel] Error on starting webadmin
In-Reply-To: <50C9EC74.6030902@redhat.com>
References: <1522414004.18111689.1355410191913.JavaMail.root@redhat.com>
	<50C9EC74.6030902@redhat.com>
Message-ID: <50CB0CC5.2030806@redhat.com>

On 12/13/2012 03:55 PM, Roy Golan wrote:
> On 12/13/2012 04:49 PM, Vojtech Szocs wrote:
>>> this is making the contribution process more complex. lets think of a
>>> lighter way to get a developing setup.
>> I agree, I just wanted to have the local Engine configuration steps documented for reference.. If there's a better way to do it, I'm for it.
>>
>> Vojtech
> 
> having LocalConfig look for "engine.conf.defaults" system property 
> before fallback to System.getenv("ENGINE_DEFAULTS");
>   +  concatenating -Dengine.conf.defaults=$HOME/.engine.conf.defaults 
> to  JAVA_OPTS  on standalone.conf

How is the system property simpler than the environment variable?

I agree that this makes the development process a bit more complex at
the moment, but I think that the way to make it simpler is not to
continue adding things to standalone.conf. I think that we should move
towards a development environment that is closer to the production
environment, not the other way around. Ideally the developer should be
able to do something like "make install" to have the engine deployed to
a directory structure similar to what he have in the production
environment. Then you should be able to go to the bin directory inside
that structure and start the engine (and the other tools) using the same
script that we use in production environments. If we achieve this goal
then we have a simple development environment setup and also we have all
developers testing almost the same thing that will go into the
production environments. At the moment we don't have that, most times
you are testing something quite different (in terms of directory
structure, configuration, etc) to what will be installed in production
environments. I am working in that direction.

>>
>>
>> ----- Original Message -----
>> From: "Roy Golan" <rgolan at redhat.com>
>> To: engine-devel at ovirt.org
>> Sent: Thursday, December 13, 2012 3:36:40 PM
>> Subject: Re: [Engine-devel] Error on starting webadmin
>>
>> On 12/13/2012 04:32 PM, Vojtech Szocs wrote:
>>> Hi,
>>>
>>> I've updated http://www.ovirt.org/Building_Engine_Draft - added section "Engine local configuration" (feel free to edit or modify as necessary)
>> this is making the contribution process more complex. lets think of a
>> lighter way to get a developing setup.
>>
>>> Vojtech
>>>
>>>
>>> ----- Original Message -----
>>> From: "Vojtech Szocs" <vszocs at redhat.com>
>>> To: engine-devel at ovirt.org
>>> Sent: Thursday, December 13, 2012 2:49:39 PM
>>> Subject: Re: [Engine-devel] Error on starting webadmin
>>>
>>> Hi guys,
>>>
>>> as Moti mentioned, this issue is related to UI plugins patches that have been merged recently.
>>>
>>> However, the root cause of this issue is LocalConfig class (machine-specific Engine configuration) throwing exception in following cases:
>>> - cannot load default properties (ENGINE_DEFAULTS property if defined, otherwise /usr/share/ovirt-engine/conf/engine.conf.defaults)
>>> - cannot load custom properties (ENGINE_VARS property if defined, otherwise /etc/sysconfig/ovirt-engine)
>>> - requesting missing property value (LocalConfig.getProperty method)
>>>
>>> (Note that UI plugins use two Engine properties: ENGINE_USR and ENGINE_ETC.)
>>>
>>> As Juan suggests in his comment in http://gerrit.ovirt.org/#/c/10026/, we could include this in "building Engine" wiki. For example:
>>> - have ENGINE_DEFAULTS property point to OVIRT_HOME/backend/manager/conf/engine.conf.defaults (alternatively, users can copy this file to /usr/share/ovirt-engine/conf)
>>> - create custom Engine config file, e.g. ~/myengine.conf, and have ENGINE_VARS property point to this file
>>>
>>> Vojtech
>>>
>>>
>>> ----- Original Message -----
>>> From: "Daniel Erez" <derez at redhat.com>
>>> To: "Moti Asayag" <masayag at redhat.com>
>>> Cc: engine-devel at ovirt.org
>>> Sent: Thursday, December 13, 2012 11:49:13 AM
>>> Subject: Re: [Engine-devel] Error on starting webadmin
>>>
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Moti Asayag" <masayag at redhat.com>
>>>> To: engine-devel at ovirt.org
>>>> Sent: Thursday, December 13, 2012 12:25:07 PM
>>>> Subject: [Engine-devel] Error on starting webadmin
>>>>
>>>> Hi,
>>>>
>>>> I get the following error when trying to connect to webadmin:
>>>>
>>>> 2012-12-13 11:57:06,689 ERROR
>>>> [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[WebAdminDynamicHosting]]
>>>> (http--0.0.0.0-8700-1) Servlet.service() for servlet
>>>> WebAdminDynamicHosting threw exception:
>>>> java.lang.IllegalArgumentException: The property "ENGINE_USR"
>>>> doesn't have a value.
>>>>
>>>> It caused by the PluginDataManager instantiation which requires
>>>> several variables to be defined.
>>>>
>>>> In order to by pass those errors do:
>>>> 1. create a file /usr/share/ovirt-engine/conf/engine.conf.defaults
>>>> 2. Add the following properties to the file:
>>>> ENGINE_USR=username
>>>> ENGINE_ETC=/etc/ovirt-engine
>>>> 3. restart Jboss
>>> You can use the 'engine.conf.defaults' that resides at 'backend/manager/conf'.
>>>
>>>> Regards,
>>>> Moti

-- 
Direcci?n Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
3?D, 28016 Madrid, Spain
Inscrita en el Reg. Mercantil de Madrid ? C.I.F. B82657941 - Red Hat S.L.


From emesika at redhat.com  Sat Dec 15 23:01:46 2012
From: emesika at redhat.com (Eli Mesika)
Date: Sat, 15 Dec 2012 18:01:46 -0500 (EST)
Subject: [Engine-devel] Design review summary for 3.2 - adding support
 for External Events
In-Reply-To: <50BD3F54.4000508@redhat.com>
Message-ID: <794046395.25326988.1355612506861.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Eli Mesika" <emesika at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Tuesday, December 4, 2012 2:09:56 AM
> Subject: Re: [Engine-devel] Design review summary for 3.2 - adding support for External Events
> 
> On 11/29/2012 11:11 PM, Eli Mesika wrote:
> > Hi
> >
> > please note that this section
> > http://wiki.ovirt.org/wiki/Features/Design/DetailedExternalEvents#Permissions
> > was changed and approved by PM (Miki)
> >
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> >
> 
> 
> 
> > Invoke ..api/events/add API giving at least AuditLogType, Severity,
> > Origin & CustomEventId
> 
> why do you need the AuditLogType - you already know it is
> ExternalEvent/Alert? you only need to know the severity iiuc.

Sorry for the late respond (too many urgent issue :-))

I need that because we have not just and Alert but also Warning , Error & Normal External Events 

> 
> > When the Event/Alert is on a specific object, the object instance
> > id should be set.
> 
> what if it is on multiple objects, like events are today?

The statement says 'at least' so one/multiple entity IDs should be applied as well

> 
> ...
> 
> > Permissions on Entity Instances
> >
> > There will be no permission check on entity instances on which
> > External Events are injected
> 
> this means this permission is always at system level, breaking the
> current permission scheme where permission can be given on a portion
> of
> the system.

I had talk with that with miki and that was our understanding of the requirement, it can be changed if it does not match the requirements or if we have misunderstood them. 

> 
> > The reason is that in order to invoke an External Events on an
> > entity instance, the invoker should know the entity instance UUID
> > and therefore we had already checked that the invoker has the
> > right permissions on the entity instance when he gets the
> > information.
> 
> UUID are not cryptographically secure. knowing a UUID is not a way to
> decide to give a permission.
> (also, this is an admin action, and admins always see all objects in
> the
> system today)
> 
> > Also, double checking that in the AddExternalEvent command is not
> > simple, since each Entity may have several ActionGroups (Create,
> > Edit etc.) associated with it, so it is not clear which to check
> 
> it is very clear - you check for each entity the user has the new
> ActionGroup that allows to inject events (same one you added to the
> superuser and to the newly created special role).

Yea, as I said above , we didn't look in External Events as an permission per object type and that leads to a general implementation..... 

> 
> > So, in order to keep things simple, we will assume that if the
> > caller to Add External Event has the Entity UUID in hand, all we
> > have to check is that he has permission to inject External Events
> 
> as i stated above, this means you are creating an ActionGroup which
> is
> global, regardless of the entity the role with it is given on.

This can be changed if required and still be ready until the 24 DEC deadline 

> 
> ...
> 
> 
> > User Experience
> >
> > Global External Events will be displayed on the Global Events TAB
> > Entity instance External Events will be displayed on the Events TAB
> > when selecting the Entity instance
> 
> 1. not clear here they will also be displayed in the global tab?
> 2. not clear here that multiple entities can be specified for an
> event.
> 
> ...
> 
> > Installation/Upgrade
> >
> > Add additional fields to audit_log table upon upgrade
> > Add the permission(ActionGroup) to manipulate External Events to
> > other admin roles already defined upon upgrade.
> 
> but in permission it is stated only superuser and a new role will be
> given this actiongroup, and here more admin roles are mentioned (if
> so,
> which is the correct one).
> 
> Thanks,
>     Itamar
> 


From iheim at redhat.com  Sun Dec 16 06:21:16 2012
From: iheim at redhat.com (Itamar Heim)
Date: Sun, 16 Dec 2012 08:21:16 +0200
Subject: [Engine-devel] Design review summary for 3.2 - adding support
 for External Events
In-Reply-To: <794046395.25326988.1355612506861.JavaMail.root@redhat.com>
References: <794046395.25326988.1355612506861.JavaMail.root@redhat.com>
Message-ID: <50CD685C.6010009@redhat.com>

On 12/16/2012 01:01 AM, Eli Mesika wrote:
>>> Invoke ..api/events/add API giving at least AuditLogType, Severity,
>>> > >Origin & CustomEventId
>> >
>> >why do you need the AuditLogType - you already know it is
>> >ExternalEvent/Alert? you only need to know the severity iiuc.
> Sorry for the late respond (too many urgent issue :-))
>
> I need that because we have not just and Alert but also Warning , Error & Normal External Events
>

yes, but the fact they are external events is known already in this API, 
so why do you need more than the severity in the API?



From emesika at redhat.com  Sun Dec 16 07:44:52 2012
From: emesika at redhat.com (Eli Mesika)
Date: Sun, 16 Dec 2012 02:44:52 -0500 (EST)
Subject: [Engine-devel] Design review summary for 3.2 - adding support
 for External Events
In-Reply-To: <50CD685C.6010009@redhat.com>
Message-ID: <245830052.25353133.1355643892822.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Eli Mesika" <emesika at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, "Miki Kenneth" <mkenneth at redhat.com>, "Barak Azulay"
> <bazulay at redhat.com>, "Yair Zaslavsky" <yzaslavs at redhat.com>
> Sent: Sunday, December 16, 2012 8:21:16 AM
> Subject: Re: [Engine-devel] Design review summary for 3.2 - adding support for External Events
> 
> On 12/16/2012 01:01 AM, Eli Mesika wrote:
> >>> Invoke ..api/events/add API giving at least AuditLogType,
> >>> Severity,
> >>> > >Origin & CustomEventId
> >> >
> >> >why do you need the AuditLogType - you already know it is
> >> >ExternalEvent/Alert? you only need to know the severity iiuc.
> > Sorry for the late respond (too many urgent issue :-))
> >
> > I need that because we have not just and Alert but also Warning ,
> > Error & Normal External Events
> >
> 
> yes, but the fact they are external events is known already in this
> API,
> so why do you need more than the severity in the API?

You are right, updated the wiki and added also API sample for adding event code
http://www.ovirt.org/Features/Design/DetailedExternalEvents#API

> 
> 


From rgolan at redhat.com  Sun Dec 16 07:57:12 2012
From: rgolan at redhat.com (Roy Golan)
Date: Sun, 16 Dec 2012 09:57:12 +0200
Subject: [Engine-devel] Error on starting webadmin
In-Reply-To: <50CB0CC5.2030806@redhat.com>
References: <1522414004.18111689.1355410191913.JavaMail.root@redhat.com>
	<50C9EC74.6030902@redhat.com> <50CB0CC5.2030806@redhat.com>
Message-ID: <50CD7ED8.6070903@redhat.com>

On 12/14/2012 01:25 PM, Juan Hernandez wrote:
> On 12/13/2012 03:55 PM, Roy Golan wrote:
>> On 12/13/2012 04:49 PM, Vojtech Szocs wrote:
>>>> this is making the contribution process more complex. lets think of a
>>>> lighter way to get a developing setup.
>>> I agree, I just wanted to have the local Engine configuration steps documented for reference.. If there's a better way to do it, I'm for it.
>>>
>>> Vojtech
>> having LocalConfig look for "engine.conf.defaults" system property
>> before fallback to System.getenv("ENGINE_DEFAULTS");
>>    +  concatenating -Dengine.conf.defaults=$HOME/.engine.conf.defaults
>> to  JAVA_OPTS  on standalone.conf
> How is the system property simpler than the environment variable?
>
> I agree that this makes the development process a bit more complex at
> the moment, but I think that the way to make it simpler is not to
> continue adding things to standalone.conf. I think that we should move
> towards a development environment that is closer to the production
> environment, not the other way around.
till certain point. that's why we have an installation script for 
production and "make install" for a developer
- alternatively, if(!) its easy and smooth we can run the engine-setup 
with an answer file  or just a "dev" flag than we can call it from the 
-Psetup profile and
make a unified process

>   Ideally the developer should be
> able to do something like "make install" to have the engine deployed to
> a directory structure similar to what he have in the production
> environment.
+1 and this is where the dev setup should end.
>   Then you should be able to go to the bin directory inside
> that structure and start the engine (and the other tools) using the same
> script that we use in production environments. If we achieve this goal
> then we have a simple development environment setup and also we have all
> developers testing almost the same thing that will go into the
> production environments.
again agree but only if its simple (minimum steps, don't have to follow 
a long and detailed wiki) and fast.
>   At the moment we don't have that, most times
> you are testing something quite different (in terms of directory
> structure, configuration, etc) to what will be installed in production
> environments. I am working in that direction.
>
>>>
>>> ----- Original Message -----
>>> From: "Roy Golan" <rgolan at redhat.com>
>>> To: engine-devel at ovirt.org
>>> Sent: Thursday, December 13, 2012 3:36:40 PM
>>> Subject: Re: [Engine-devel] Error on starting webadmin
>>>
>>> On 12/13/2012 04:32 PM, Vojtech Szocs wrote:
>>>> Hi,
>>>>
>>>> I've updated http://www.ovirt.org/Building_Engine_Draft - added section "Engine local configuration" (feel free to edit or modify as necessary)
>>> this is making the contribution process more complex. lets think of a
>>> lighter way to get a developing setup.
>>>
>>>> Vojtech
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "Vojtech Szocs" <vszocs at redhat.com>
>>>> To: engine-devel at ovirt.org
>>>> Sent: Thursday, December 13, 2012 2:49:39 PM
>>>> Subject: Re: [Engine-devel] Error on starting webadmin
>>>>
>>>> Hi guys,
>>>>
>>>> as Moti mentioned, this issue is related to UI plugins patches that have been merged recently.
>>>>
>>>> However, the root cause of this issue is LocalConfig class (machine-specific Engine configuration) throwing exception in following cases:
>>>> - cannot load default properties (ENGINE_DEFAULTS property if defined, otherwise /usr/share/ovirt-engine/conf/engine.conf.defaults)
>>>> - cannot load custom properties (ENGINE_VARS property if defined, otherwise /etc/sysconfig/ovirt-engine)
>>>> - requesting missing property value (LocalConfig.getProperty method)
>>>>
>>>> (Note that UI plugins use two Engine properties: ENGINE_USR and ENGINE_ETC.)
>>>>
>>>> As Juan suggests in his comment in http://gerrit.ovirt.org/#/c/10026/, we could include this in "building Engine" wiki. For example:
>>>> - have ENGINE_DEFAULTS property point to OVIRT_HOME/backend/manager/conf/engine.conf.defaults (alternatively, users can copy this file to /usr/share/ovirt-engine/conf)
>>>> - create custom Engine config file, e.g. ~/myengine.conf, and have ENGINE_VARS property point to this file
>>>>
>>>> Vojtech
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "Daniel Erez" <derez at redhat.com>
>>>> To: "Moti Asayag" <masayag at redhat.com>
>>>> Cc: engine-devel at ovirt.org
>>>> Sent: Thursday, December 13, 2012 11:49:13 AM
>>>> Subject: Re: [Engine-devel] Error on starting webadmin
>>>>
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Moti Asayag" <masayag at redhat.com>
>>>>> To: engine-devel at ovirt.org
>>>>> Sent: Thursday, December 13, 2012 12:25:07 PM
>>>>> Subject: [Engine-devel] Error on starting webadmin
>>>>>
>>>>> Hi,
>>>>>
>>>>> I get the following error when trying to connect to webadmin:
>>>>>
>>>>> 2012-12-13 11:57:06,689 ERROR
>>>>> [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[WebAdminDynamicHosting]]
>>>>> (http--0.0.0.0-8700-1) Servlet.service() for servlet
>>>>> WebAdminDynamicHosting threw exception:
>>>>> java.lang.IllegalArgumentException: The property "ENGINE_USR"
>>>>> doesn't have a value.
>>>>>
>>>>> It caused by the PluginDataManager instantiation which requires
>>>>> several variables to be defined.
>>>>>
>>>>> In order to by pass those errors do:
>>>>> 1. create a file /usr/share/ovirt-engine/conf/engine.conf.defaults
>>>>> 2. Add the following properties to the file:
>>>>> ENGINE_USR=username
>>>>> ENGINE_ETC=/etc/ovirt-engine
>>>>> 3. restart Jboss
>>>> You can use the 'engine.conf.defaults' that resides at 'backend/manager/conf'.
>>>>
>>>>> Regards,
>>>>> Moti



From lpeer at redhat.com  Sun Dec 16 09:23:23 2012
From: lpeer at redhat.com (Livnat Peer)
Date: Sun, 16 Dec 2012 11:23:23 +0200
Subject: [Engine-devel] ThreadPoolUtil with 500 threads and no queue?
In-Reply-To: <50C8AB86.4060304@redhat.com>
References: <50C8AB86.4060304@redhat.com>
Message-ID: <50CD930B.1010500@redhat.com>

On 12/12/12 18:06, Juan Hernandez wrote:
> Hello all,
> 
> What is the reasoning behind the decision to have a pool with a maximum
> of 500 threads and no job queue (see ThreadPoolUtil.java)? Wouldn't it
> make more sense to have a much smaller thread pool and a potentially
> large queue of jobs?
> 

Hi Juan,
I think there is no right/wrong number, as Kublin added on this thread
there are several approaches to address this issue.

My 2 cents on this is that a change should be based on a given workload
profile. Any given solution would suits specific workload and hurt
another, as long as we are not sure what is the common workload our
users use I would make any change configurable and write a
recommendation on how to configure it.
For writing such document one should characterized few typical usages of
the system and test what is the preferred configuration for them.

Livnat

> Regards,
> Juan Hernandez
> 



From lpeer at redhat.com  Sun Dec 16 09:51:15 2012
From: lpeer at redhat.com (Livnat Peer)
Date: Sun, 16 Dec 2012 11:51:15 +0200
Subject: [Engine-devel] Creating New Data-Center and Permission on
	Network
In-Reply-To: <50C5EC5D.50503@redhat.com>
References: <50C5EC5D.50503@redhat.com>
Message-ID: <50CD9993.6000701@redhat.com>

On 10/12/12 16:06, Moti Asayag wrote:
> Hi,
> 
> When creating a new data-center, a management network is created for it.
> By default the created management network is defined as VM network.
> 
> I'd like to consult from permissions perspective, what is the preferred
> permission settings for that network.
> 
> The network is defined as management network, therefore it is designed
> be used by VMs.

I think you meant that the management network is marked by default as a
VM network and therefor is available for usage within the VMs.

> However, the admin should grant permissions on that
> networks to the target users (which one might find tedious).
> 

I'm in favor of keeping this behavior, I think that by default the
management network should not be available to users unless explicitly a
permission was added by the admin.

> We can grant permission on that network to 'everyone' with role
> 'NetworkUser', but in case the admin doesn't meant this network to be
> used, the permission should be removed.
> 
> In 'Add Logical Network' dialog I've added a new checkbox to allow
> granting 'everyone' a role for using that network ('NetworkUser').
> We can embrace same method in 'Add Data-Center' dialog.
> 

That's an interesting option I'd wait with adding this until we get some
feedback from the users on the usage of network permissions.


> Thoughts ?
> 
> Thanks,
> Moti
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 



From danken at redhat.com  Sun Dec 16 14:20:14 2012
From: danken at redhat.com (Dan Kenigsberg)
Date: Sun, 16 Dec 2012 16:20:14 +0200
Subject: [Engine-devel] [vdsm] VDSM tasks, the future
In-Reply-To: <1468813011.14266792.1354718748927.JavaMail.root@redhat.com>
References: <50BF4E13.7090700@redhat.com>
	<1468813011.14266792.1354718748927.JavaMail.root@redhat.com>
Message-ID: <20121216142014.GI9782@redhat.com>

On Wed, Dec 05, 2012 at 09:45:48AM -0500, Saggi Mizrahi wrote:
> I'm sorry but your email client messed up the formatting and I can't figure out what are you comments.
> Could you please use text only emails.

It's funny to hear this form someone that uses top-posting with long
lines.

But that's not the reason of my confusion. I know very well that the
current task framework in vdsm is an over complicated mess. I believe
that there is only one living person that understands
http://wiki.ovirt.org/File:Vdsmtasks.jpg, and it is not me. However, I
do not know what you want to achieve in the future.

You detail what tasks are not going to give me: no persistence, no
cleanup, no synchronization. Do they give me a fake progress bar
percentage?

When Engine requests to run a new VM, it calls the vmCreate verb, which
returns immediately. Engine then polls the VM until it leaves its
initial waitForLaunch state. I've daydreamed about changing this to
something more generic, using the task framework - but I do not see how
your new vision helps to achieve this.

Could you provide some guidance on your vision, with this use case in
mind?

Dan.


From bazulay at redhat.com  Sun Dec 16 16:24:08 2012
From: bazulay at redhat.com (Barak Azulay)
Date: Sun, 16 Dec 2012 11:24:08 -0500 (EST)
Subject: [Engine-devel] [vdsm] Host bios information
In-Reply-To: <1807653280.23656995.1355443624198.JavaMail.root@redhat.com>
Message-ID: <947046802.52821756.1355675048716.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Ayal Baron" <abaron at redhat.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "VDSM Project Development" <vdsm-devel at lists.fedorahosted.org>
> Sent: Friday, December 14, 2012 2:07:04 AM
> Subject: Re: [vdsm] Host bios information
> 
> 
> 
> ----- Original Message -----
> > 
> > 
> > ----- Original Message -----
> > > From: "Ayal Baron" <abaron at redhat.com>
> > > To: "Saggi Mizrahi" <smizrahi at redhat.com>
> > > Cc: "VDSM Project Development"
> > > <vdsm-devel at lists.fedorahosted.org>,
> > > "Shu Ming" <shuming at linux.vnet.ibm.com>
> > > Sent: Thursday, December 13, 2012 11:30:56 AM
> > > Subject: Re: [vdsm] Host bios information
> > > 
> > > 
> > > 
> > > ----- Original Message -----
> > > > I think that for the new current XML-RPC API it's OK to add it
> > > > to
> > > > the
> > > > getVdsCaps() verb.
> > > > For the new API I suggest moving it to it's own API. The
> > > > smaller
> > > > the
> > > > APIs the easier they are to deprecate and support.
> > > > I quite doubt the fields in getBiosInfo() will change half as
> > > > frequently as whatever getVdsCaps() returns.
> > > > I also kind of want to throw away getVdsCaps() and split it to
> > > > better
> > > > named better encapsulated methods.
> > > 
> > > Ack.  I just don't understand why not start right now?
> > > Any new patch should improve things at least a little.
> > > We know getVdsCaps() is wrong so let's put the bios info (and
> > > anything in getVdsCaps that makes sense to put with it if
> > > relevant)
> > > in a separate call.  Adding a call in engine to this new method
> > > should be a no brainer, I don't think that is a good reason for
> > > not
> > > doing things properly in vdsm, even if we're talking about the
> > > current API.
> > Well, from what I know the current overhead per call is too large
> > to
> > mandate a lot of calls.
> > At least that is what I've been told. If that is not an issue, do
> > it
> > in the XML-RPC API too.
> 
> if you call it every 2 seconds to each host then perhaps, but
> getVdsCaps is called in initvdsonup which is pretty rare (hours,
> days or more) so avoiding an extra call there seems wrong to me.


I totally agree that in the long run it is correct to add a different API on vdsm,
However for the initial 6 bios params:

1. Host Manufacturer - Manufacturer of the host's machine and bios' vendor (e.g LENOVO)
2. Host Version - For each host the manufacturer gives a unique name (e.g. Lenovo T420s)
3. Host Product Name - ID of the product - same for all similar products (e.g 4174BH4)
4. Host UUID - Unique ID for each host (e.g E03DD601-5219-11CB-BB3F-892313086897)
5. Host Family - Type of host's CPU - (e.g Core i5)
6. Host Serial Number - Unique ID for host's chassis (e.g R9M4N4G)

As Dan stated below, is o.k. for this specific phase to add them to getVdsCaps

So I suggest:
1 Adding the above 6 above params to getVdsCaps
2 Add a new API to vdsm to retrieve host bios information - that will eventually return the entire list (current will retrieve the exact 6 params and will use the same underlying implementation)
3 In the future engine will start using the new API

Start using the new API now on the engine side will require much more testing as it might have an influence on the host state machine.

Thanks
Barak Azulay

> 
> > > 
> > > > 
> > > > Also, in the json-rpc base model, calls are not only cheaper,
> > > > you
> > > > also have batch calls. This means you can send multiple
> > > > requests
> > > > as
> > > > one message and have VDSM send you the responses as one message
> > > > once
> > > > all tasks completed. This makes splitting aggregated methods to
> > > > smaller methods painless and with minimal overhead.
> > > > 
> > > > ----- Original Message -----
> > > > > From: "Shu Ming" <shuming at linux.vnet.ibm.com>
> > > > > To: "ybronhei" <ybronhei at redhat.com>
> > > > > Cc: "VDSM Project Development"
> > > > > <vdsm-devel at lists.fedorahosted.org>
> > > > > Sent: Thursday, December 13, 2012 11:04:09 AM
> > > > > Subject: Re: [vdsm] Host bios information
> > > > > 
> > > > > After a quick review of the wiki page, it was stated that
> > > > > dmidecode
> > > > > gave
> > > > > too much informations. Only five fields will be displayed in
> > > > > the
> > > > > hardware tab, "Manufactory", "Version", "Family", "UUID" and
> > > > > "serial
> > > > > number".  For "Family", it is mean the CPU core's family.
> > > > >  And
> > > > > it
> > > > > confuses me a bit with the "CPU name" and "CPU type" fields
> > > > > in
> > > > > general
> > > > > tab. I think we should chose the best one to characterizethe
> > > > > CPU
> > > > > type.
> > > > > 
> > > > > 
> > > > > ybronhei:
> > > > > > Today in the Api we display general information about the
> > > > > > host
> > > > > > that
> > > > > > vdsm export by getCapabilities Api.
> > > > > >
> > > > > > We decided to add bios information as part of the
> > > > > > information
> > > > > > that
> > > > > > is
> > > > > > displayed in UI under host's general sub-tab.
> > > > > >
> > > > > > To summaries the feature - We'll modify General tab to
> > > > > > Software
> > > > > > Information and add another tab for Hardware Information
> > > > > > which
> > > > > > will
> > > > > > include all the bios data that we'll decide to gather from
> > > > > > the
> > > > > > host
> > > > > > and display.
> > > > > >
> > > > > > Following this feature page:
> > > > > > http://www.ovirt.org/Features/Design/HostBiosInfo for more
> > > > > > details.
> > > > > > All the parameters that can be displayed are mentioned in
> > > > > > the
> > > > > > wiki.
> > > > > >
> > > > > > I would greatly appreciate your comments and questions.
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > 
> > > > > 
> > > > > --
> > > > > ---
> > > > > ?? Shu Ming
> > > > > Open Virtualization Engineerning; CSTL, IBM Corp.
> > > > > Tel: 86-10-82451626  Tieline: 9051626 E-mail:
> > > > > shuming at cn.ibm.com
> > > > > or
> > > > > shuming at linux.vnet.ibm.com
> > > > > Address: 3/F Ring Building, ZhongGuanCun Software Park,
> > > > > Haidian
> > > > > District, Beijing 100193, PRC
> > > > > 
> > > > > 
> > > > > _______________________________________________
> > > > > vdsm-devel mailing list
> > > > > vdsm-devel at lists.fedorahosted.org
> > > > > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> > > > > 
> > > > _______________________________________________
> > > > vdsm-devel mailing list
> > > > vdsm-devel at lists.fedorahosted.org
> > > > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> > > > 
> > > 
> > 
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>


From ybronhei at redhat.com  Sun Dec 16 17:04:32 2012
From: ybronhei at redhat.com (ybronhei)
Date: Sun, 16 Dec 2012 19:04:32 +0200
Subject: [Engine-devel] [vdsm] Host bios information
In-Reply-To: <947046802.52821756.1355675048716.JavaMail.root@redhat.com>
References: <947046802.52821756.1355675048716.JavaMail.root@redhat.com>
Message-ID: <50CDFF20.9010302@redhat.com>

On 12/16/2012 06:24 PM, Barak Azulay wrote:
>
>
> ----- Original Message -----
>> From: "Ayal Baron" <abaron at redhat.com>
>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
>> Cc: "VDSM Project Development" <vdsm-devel at lists.fedorahosted.org>
>> Sent: Friday, December 14, 2012 2:07:04 AM
>> Subject: Re: [vdsm] Host bios information
>>
>>
>>
>> ----- Original Message -----
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Ayal Baron" <abaron at redhat.com>
>>>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
>>>> Cc: "VDSM Project Development"
>>>> <vdsm-devel at lists.fedorahosted.org>,
>>>> "Shu Ming" <shuming at linux.vnet.ibm.com>
>>>> Sent: Thursday, December 13, 2012 11:30:56 AM
>>>> Subject: Re: [vdsm] Host bios information
>>>>
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> I think that for the new current XML-RPC API it's OK to add it
>>>>> to
>>>>> the
>>>>> getVdsCaps() verb.
>>>>> For the new API I suggest moving it to it's own API. The
>>>>> smaller
>>>>> the
>>>>> APIs the easier they are to deprecate and support.
>>>>> I quite doubt the fields in getBiosInfo() will change half as
>>>>> frequently as whatever getVdsCaps() returns.
>>>>> I also kind of want to throw away getVdsCaps() and split it to
>>>>> better
>>>>> named better encapsulated methods.
>>>>
>>>> Ack.  I just don't understand why not start right now?
>>>> Any new patch should improve things at least a little.
>>>> We know getVdsCaps() is wrong so let's put the bios info (and
>>>> anything in getVdsCaps that makes sense to put with it if
>>>> relevant)
>>>> in a separate call.  Adding a call in engine to this new method
>>>> should be a no brainer, I don't think that is a good reason for
>>>> not
>>>> doing things properly in vdsm, even if we're talking about the
>>>> current API.
>>> Well, from what I know the current overhead per call is too large
>>> to
>>> mandate a lot of calls.
>>> At least that is what I've been told. If that is not an issue, do
>>> it
>>> in the XML-RPC API too.
>>
>> if you call it every 2 seconds to each host then perhaps, but
>> getVdsCaps is called in initvdsonup which is pretty rare (hours,
>> days or more) so avoiding an extra call there seems wrong to me.
>
>
> I totally agree that in the long run it is correct to add a different API on vdsm,
> However for the initial 6 bios params:
>
> 1. Host Manufacturer - Manufacturer of the host's machine and bios' vendor (e.g LENOVO)
> 2. Host Version - For each host the manufacturer gives a unique name (e.g. Lenovo T420s)
> 3. Host Product Name - ID of the product - same for all similar products (e.g 4174BH4)
> 4. Host UUID - Unique ID for each host (e.g E03DD601-5219-11CB-BB3F-892313086897)
> 5. Host Family - Type of host's CPU - (e.g Core i5)
> 6. Host Serial Number - Unique ID for host's chassis (e.g R9M4N4G)
>
> As Dan stated below, is o.k. for this specific phase to add them to getVdsCaps
>
> So I suggest:
> 1 Adding the above 6 above params to getVdsCaps
> 2 Add a new API to vdsm to retrieve host bios information - that will eventually return the entire list (current will retrieve the exact 6 params and will use the same underlying implementation)
> 3 In the future engine will start using the new API
>
> Start using the new API now on the engine side will require much more testing as it might have an influence on the host state machine.
>
> Thanks
> Barak Azulay
>
Agree,
In vdsm side as Andrew suggested we can add new xml-rpc Api that 
internally calls by getCapabilities until we will define the flow 
changes that we need to do in the engine side to permit split calls that 
fill VDS object fields.

Before we change the engine side to perform 2 calls to fill vds data, we 
need to define the flows for retrieving the data, in which table to keep 
it, and when to refresh it.
getCapabilities API call signs for the engine that the host is 
operational. By performing another API call we will get into cases that 
we've never checked and we need to define and test.

This feature can be split to 2 levels, first we have the requirement for 
6 field, and only those fields (as i mentioned in the wiki), those  can 
be added as part of host capabilities structure, in declared structure 
that defines the bios information, until we define the new api. 
Additionally, we can add vdsm API the returns only this structure for 
future use by the engine.

Anyhow, when defining new structure for bios information we can add more 
values of dmidecode in the future if the users will request for such 
info, and it won't effect the changes we add to the UI as part of this 
feature.

IMPOV, I think we should implement that feature as described now in the 
engine side. Using new vdsm API by the engine should be part of another 
feature that arranges getCapabilities variables, defines the flows of 
requesting bios variables and in which VDS table engine should store 
this data.

The engine part patches of this feature are:
http://gerrit.ovirt.org/#/c/10114/ - Modifying db tables
http://gerrit.ovirt.org/#/c/9337/ - Adding bios data to backend's objects
http://gerrit.ovirt.org/#/c/10115/ - UI changes to add host subtab for 
Bios info

vdsm part:
http://gerrit.ovirt.org/#/c/9258/ - will be modified asap to external 
structure.

I would appreciate your comments and reviews

thanks!

>>
>>>>
>>>>>
>>>>> Also, in the json-rpc base model, calls are not only cheaper,
>>>>> you
>>>>> also have batch calls. This means you can send multiple
>>>>> requests
>>>>> as
>>>>> one message and have VDSM send you the responses as one message
>>>>> once
>>>>> all tasks completed. This makes splitting aggregated methods to
>>>>> smaller methods painless and with minimal overhead.
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
>>>>>> To: "ybronhei" <ybronhei at redhat.com>
>>>>>> Cc: "VDSM Project Development"
>>>>>> <vdsm-devel at lists.fedorahosted.org>
>>>>>> Sent: Thursday, December 13, 2012 11:04:09 AM
>>>>>> Subject: Re: [vdsm] Host bios information
>>>>>>
>>>>>> After a quick review of the wiki page, it was stated that
>>>>>> dmidecode
>>>>>> gave
>>>>>> too much informations. Only five fields will be displayed in
>>>>>> the
>>>>>> hardware tab, "Manufactory", "Version", "Family", "UUID" and
>>>>>> "serial
>>>>>> number".  For "Family", it is mean the CPU core's family.
>>>>>>   And
>>>>>> it
>>>>>> confuses me a bit with the "CPU name" and "CPU type" fields
>>>>>> in
>>>>>> general
>>>>>> tab. I think we should chose the best one to characterizethe
>>>>>> CPU
>>>>>> type.
>>>>>>
>>>>>>
>>>>>> ybronhei:
>>>>>>> Today in the Api we display general information about the
>>>>>>> host
>>>>>>> that
>>>>>>> vdsm export by getCapabilities Api.
>>>>>>>
>>>>>>> We decided to add bios information as part of the
>>>>>>> information
>>>>>>> that
>>>>>>> is
>>>>>>> displayed in UI under host's general sub-tab.
>>>>>>>
>>>>>>> To summaries the feature - We'll modify General tab to
>>>>>>> Software
>>>>>>> Information and add another tab for Hardware Information
>>>>>>> which
>>>>>>> will
>>>>>>> include all the bios data that we'll decide to gather from
>>>>>>> the
>>>>>>> host
>>>>>>> and display.
>>>>>>>
>>>>>>> Following this feature page:
>>>>>>> http://www.ovirt.org/Features/Design/HostBiosInfo for more
>>>>>>> details.
>>>>>>> All the parameters that can be displayed are mentioned in
>>>>>>> the
>>>>>>> wiki.
>>>>>>>
>>>>>>> I would greatly appreciate your comments and questions.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> ---
>>>>>> ?? Shu Ming
>>>>>> Open Virtualization Engineerning; CSTL, IBM Corp.
>>>>>> Tel: 86-10-82451626  Tieline: 9051626 E-mail:
>>>>>> shuming at cn.ibm.com
>>>>>> or
>>>>>> shuming at linux.vnet.ibm.com
>>>>>> Address: 3/F Ring Building, ZhongGuanCun Software Park,
>>>>>> Haidian
>>>>>> District, Beijing 100193, PRC
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> vdsm-devel mailing list
>>>>>> vdsm-devel at lists.fedorahosted.org
>>>>>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>>>>>>
>>>>> _______________________________________________
>>>>> vdsm-devel mailing list
>>>>> vdsm-devel at lists.fedorahosted.org
>>>>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>>>>>
>>>>
>>>
>> _______________________________________________
>> vdsm-devel mailing list
>> vdsm-devel at lists.fedorahosted.org
>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>>
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
>


-- 
Yaniv Bronhaim.
RedHat, Israel
09-7692289
054-7744187


From acathrow at redhat.com  Sun Dec 16 17:32:10 2012
From: acathrow at redhat.com (Andrew Cathrow)
Date: Sun, 16 Dec 2012 12:32:10 -0500 (EST)
Subject: [Engine-devel] [vdsm]   Host bios information
In-Reply-To: <50CDFF20.9010302@redhat.com>
Message-ID: <830405099.52832254.1355679130551.JavaMail.root@redhat.com>



----- Original Message -----
> From: "ybronhei" <ybronhei at redhat.com>
> To: "Barak Azulay" <bazulay at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, "VDSM Project Development" <vdsm-devel at lists.fedorahosted.org>
> Sent: Sunday, December 16, 2012 12:04:32 PM
> Subject: Re: [vdsm] [Engine-devel]  Host bios information
> 
> On 12/16/2012 06:24 PM, Barak Azulay wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Ayal Baron" <abaron at redhat.com>
> >> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> >> Cc: "VDSM Project Development" <vdsm-devel at lists.fedorahosted.org>
> >> Sent: Friday, December 14, 2012 2:07:04 AM
> >> Subject: Re: [vdsm] Host bios information
> >>
> >>
> >>
> >> ----- Original Message -----
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Ayal Baron" <abaron at redhat.com>
> >>>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> >>>> Cc: "VDSM Project Development"
> >>>> <vdsm-devel at lists.fedorahosted.org>,
> >>>> "Shu Ming" <shuming at linux.vnet.ibm.com>
> >>>> Sent: Thursday, December 13, 2012 11:30:56 AM
> >>>> Subject: Re: [vdsm] Host bios information
> >>>>
> >>>>
> >>>>
> >>>> ----- Original Message -----
> >>>>> I think that for the new current XML-RPC API it's OK to add it
> >>>>> to
> >>>>> the
> >>>>> getVdsCaps() verb.
> >>>>> For the new API I suggest moving it to it's own API. The
> >>>>> smaller
> >>>>> the
> >>>>> APIs the easier they are to deprecate and support.
> >>>>> I quite doubt the fields in getBiosInfo() will change half as
> >>>>> frequently as whatever getVdsCaps() returns.
> >>>>> I also kind of want to throw away getVdsCaps() and split it to
> >>>>> better
> >>>>> named better encapsulated methods.
> >>>>
> >>>> Ack.  I just don't understand why not start right now?
> >>>> Any new patch should improve things at least a little.
> >>>> We know getVdsCaps() is wrong so let's put the bios info (and
> >>>> anything in getVdsCaps that makes sense to put with it if
> >>>> relevant)
> >>>> in a separate call.  Adding a call in engine to this new method
> >>>> should be a no brainer, I don't think that is a good reason for
> >>>> not
> >>>> doing things properly in vdsm, even if we're talking about the
> >>>> current API.
> >>> Well, from what I know the current overhead per call is too large
> >>> to
> >>> mandate a lot of calls.
> >>> At least that is what I've been told. If that is not an issue, do
> >>> it
> >>> in the XML-RPC API too.
> >>
> >> if you call it every 2 seconds to each host then perhaps, but
> >> getVdsCaps is called in initvdsonup which is pretty rare (hours,
> >> days or more) so avoiding an extra call there seems wrong to me.
> >
> >
> > I totally agree that in the long run it is correct to add a
> > different API on vdsm,
> > However for the initial 6 bios params:
> >
> > 1. Host Manufacturer - Manufacturer of the host's machine and bios'
> > vendor (e.g LENOVO)
> > 2. Host Version - For each host the manufacturer gives a unique
> > name (e.g. Lenovo T420s)
> > 3. Host Product Name - ID of the product - same for all similar
> > products (e.g 4174BH4)
> > 4. Host UUID - Unique ID for each host (e.g
> > E03DD601-5219-11CB-BB3F-892313086897)
> > 5. Host Family - Type of host's CPU - (e.g Core i5)
> > 6. Host Serial Number - Unique ID for host's chassis (e.g R9M4N4G)
> >
> > As Dan stated below, is o.k. for this specific phase to add them to
> > getVdsCaps
> >
> > So I suggest:
> > 1 Adding the above 6 above params to getVdsCaps
> > 2 Add a new API to vdsm to retrieve host bios information - that
> > will eventually return the entire list (current will retrieve the
> > exact 6 params and will use the same underlying implementation)
> > 3 In the future engine will start using the new API
> >
> > Start using the new API now on the engine side will require much
> > more testing as it might have an influence on the host state
> > machine.
> >
> > Thanks
> > Barak Azulay
> >
> Agree,
> In vdsm side as Andrew suggested we can add new xml-rpc Api that
> internally calls by getCapabilities until we will define the flow
> changes that we need to do in the engine side to permit split calls
> that
> fill VDS object fields.
> 
> Before we change the engine side to perform 2 calls to fill vds data,
> we
> need to define the flows for retrieving the data, in which table to
> keep
> it, and when to refresh it.
> getCapabilities API call signs for the engine that the host is
> operational. By performing another API call we will get into cases
> that
> we've never checked and we need to define and test.
> 
> This feature can be split to 2 levels, first we have the requirement
> for
> 6 field, and only those fields (as i mentioned in the wiki), those
>  can
> be added as part of host capabilities structure, in declared
> structure
> that defines the bios information, until we define the new api.
> Additionally, we can add vdsm API the returns only this structure for
> future use by the engine.
> 
> Anyhow, when defining new structure for bios information we can add
> more
> values of dmidecode in the future if the users will request for such
> info, and it won't effect the changes we add to the UI as part of
> this
> feature.
> 
> IMPOV, I think we should implement that feature as described now in
> the
> engine side. Using new vdsm API by the engine should be part of
> another
> feature that arranges getCapabilities variables, defines the flows of
> requesting bios variables and in which VDS table engine should store
> this data.
> 
> The engine part patches of this feature are:
> http://gerrit.ovirt.org/#/c/10114/ - Modifying db tables
> http://gerrit.ovirt.org/#/c/9337/ - Adding bios data to backend's
> objects
> http://gerrit.ovirt.org/#/c/10115/ - UI changes to add host subtab
> for
> Bios info

I think we are still missing some information even for the smaller subset - eg asset tag

James can you do a little research with our oem partners and see which makes sense

> 
> vdsm part:
> http://gerrit.ovirt.org/#/c/9258/ - will be modified asap to external
> structure.
> 
> I would appreciate your comments and reviews
> 
> thanks!
> 
> >>
> >>>>
> >>>>>
> >>>>> Also, in the json-rpc base model, calls are not only cheaper,
> >>>>> you
> >>>>> also have batch calls. This means you can send multiple
> >>>>> requests
> >>>>> as
> >>>>> one message and have VDSM send you the responses as one message
> >>>>> once
> >>>>> all tasks completed. This makes splitting aggregated methods to
> >>>>> smaller methods painless and with minimal overhead.
> >>>>>
> >>>>> ----- Original Message -----
> >>>>>> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
> >>>>>> To: "ybronhei" <ybronhei at redhat.com>
> >>>>>> Cc: "VDSM Project Development"
> >>>>>> <vdsm-devel at lists.fedorahosted.org>
> >>>>>> Sent: Thursday, December 13, 2012 11:04:09 AM
> >>>>>> Subject: Re: [vdsm] Host bios information
> >>>>>>
> >>>>>> After a quick review of the wiki page, it was stated that
> >>>>>> dmidecode
> >>>>>> gave
> >>>>>> too much informations. Only five fields will be displayed in
> >>>>>> the
> >>>>>> hardware tab, "Manufactory", "Version", "Family", "UUID" and
> >>>>>> "serial
> >>>>>> number".  For "Family", it is mean the CPU core's family.
> >>>>>>   And
> >>>>>> it
> >>>>>> confuses me a bit with the "CPU name" and "CPU type" fields
> >>>>>> in
> >>>>>> general
> >>>>>> tab. I think we should chose the best one to characterizethe
> >>>>>> CPU
> >>>>>> type.
> >>>>>>
> >>>>>>
> >>>>>> ybronhei:
> >>>>>>> Today in the Api we display general information about the
> >>>>>>> host
> >>>>>>> that
> >>>>>>> vdsm export by getCapabilities Api.
> >>>>>>>
> >>>>>>> We decided to add bios information as part of the
> >>>>>>> information
> >>>>>>> that
> >>>>>>> is
> >>>>>>> displayed in UI under host's general sub-tab.
> >>>>>>>
> >>>>>>> To summaries the feature - We'll modify General tab to
> >>>>>>> Software
> >>>>>>> Information and add another tab for Hardware Information
> >>>>>>> which
> >>>>>>> will
> >>>>>>> include all the bios data that we'll decide to gather from
> >>>>>>> the
> >>>>>>> host
> >>>>>>> and display.
> >>>>>>>
> >>>>>>> Following this feature page:
> >>>>>>> http://www.ovirt.org/Features/Design/HostBiosInfo for more
> >>>>>>> details.
> >>>>>>> All the parameters that can be displayed are mentioned in
> >>>>>>> the
> >>>>>>> wiki.
> >>>>>>>
> >>>>>>> I would greatly appreciate your comments and questions.
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> ---
> >>>>>> ?? Shu Ming
> >>>>>> Open Virtualization Engineerning; CSTL, IBM Corp.
> >>>>>> Tel: 86-10-82451626  Tieline: 9051626 E-mail:
> >>>>>> shuming at cn.ibm.com
> >>>>>> or
> >>>>>> shuming at linux.vnet.ibm.com
> >>>>>> Address: 3/F Ring Building, ZhongGuanCun Software Park,
> >>>>>> Haidian
> >>>>>> District, Beijing 100193, PRC
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> vdsm-devel mailing list
> >>>>>> vdsm-devel at lists.fedorahosted.org
> >>>>>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> >>>>>>
> >>>>> _______________________________________________
> >>>>> vdsm-devel mailing list
> >>>>> vdsm-devel at lists.fedorahosted.org
> >>>>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> >>>>>
> >>>>
> >>>
> >> _______________________________________________
> >> vdsm-devel mailing list
> >> vdsm-devel at lists.fedorahosted.org
> >> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> >>
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> >
> 
> 
> --
> Yaniv Bronhaim.
> RedHat, Israel
> 09-7692289
> 054-7744187
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 


From deepakcs at linux.vnet.ibm.com  Mon Dec 17 04:40:01 2012
From: deepakcs at linux.vnet.ibm.com (Deepak C Shetty)
Date: Mon, 17 Dec 2012 10:10:01 +0530
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <2088165976.15522700.1354910021174.JavaMail.root@redhat.com>
References: <2088165976.15522700.1354910021174.JavaMail.root@redhat.com>
Message-ID: <50CEA221.6000902@linux.vnet.ibm.com>

On 12/08/2012 01:23 AM, Saggi Mizrahi wrote:
>
> ----- Original Message -----
>> From: "Deepak C Shetty" <deepakcs at linux.vnet.ibm.com>
>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
>> Cc: "Shu Ming" <shuming at linux.vnet.ibm.com>, "engine-devel" <engine-devel at ovirt.org>, "VDSM Project Development"
>> <vdsm-devel at lists.fedorahosted.org>, "Deepak C Shetty" <deepakcs at linux.vnet.ibm.com>
>> Sent: Friday, December 7, 2012 12:23:15 AM
>> Subject: Re: [vdsm] RFC: New Storage API
>>
>> On 12/06/2012 10:22 PM, Saggi Mizrahi wrote:
>>> ----- Original Message -----
>>>> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
>>>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
>>>> Cc: "VDSM Project Development"
>>>> <vdsm-devel at lists.fedorahosted.org>, "engine-devel"
>>>> <engine-devel at ovirt.org>
>>>> Sent: Thursday, December 6, 2012 11:02:02 AM
>>>> Subject: Re: [vdsm] RFC: New Storage API
>>>>
>>>> Saggi,
>>>>
>>>> Thanks for sharing your thought and I get some comments below.
>>>>
>>>>
>>>> Saggi Mizrahi:
>>>>> I've been throwing a lot of bits out about the new storage API
>>>>> and
>>>>> I think it's time to talk a bit.
>>>>> I will purposefully try and keep implementation details away and
>>>>> concentrate about how the API looks and how you use it.
>>>>>
>>>>> First major change is in terminology, there is no long a storage
>>>>> domain but a storage repository.
>>>>> This change is done because so many things are already called
>>>>> domain in the system and this will make things less confusing for
>>>>> new-commers with a libvirt background.
>>>>>
>>>>> One other changes is that repositories no longer have a UUID.
>>>>> The UUID was only used in the pool members manifest and is no
>>>>> longer needed.
>>>>>
>>>>>
>>>>> connectStorageRepository(repoId, repoFormat,
>>>>> connectionParameters={}):
>>>>> repoId - is a transient name that will be used to refer to the
>>>>> connected domain, it is not persisted and doesn't have to be the
>>>>> same across the cluster.
>>>>> repoFormat - Similar to what used to be type (eg. localfs-1.0,
>>>>> nfs-3.4, clvm-1.2).
>>>>> connectionParameters - This is format specific and will used to
>>>>> tell VDSM how to connect to the repo.
>>>> Where does repoID come from? I think repoID doesn't exist before
>>>> connectStorageRepository() return.  Isn't repoID a return value of
>>>> connectStorageRepository()?
>>> No, repoIDs are no longer part of the domain, they are just a
>>> transient handle.
>>> The user can put whatever it wants there as long as it isn't
>>> already taken by another currently connected domain.
>> So what happens when user mistakenly gives a repoID that is in use
>> before.. there should be something in the return value that specifies
>> the error and/or reason for error so that user can try with a
>> new/diff
>> repoID ?
> Asi I said, connect fails if the repoId is in use ATM.
>>>>> disconnectStorageRepository(self, repoId)
>>>>>
>>>>>
>>>>> In the new API there are only images, some images are mutable and
>>>>> some are not.
>>>>> mutable images are also called VirtualDisks
>>>>> immutable images are also called Snapshots
>>>>>
>>>>> There are no explicit templates, you can create as many images as
>>>>> you want from any snapshot.
>>>>>
>>>>> There are 4 major image operations:
>>>>>
>>>>>
>>>>> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
>>>>>                      userData={}, options={}):
>>>>>
>>>>> targetRepoId - ID of a connected repo where the disk will be
>>>>> created
>>>>> size - The size of the image you wish to create
>>>>> baseSnapshotId - the ID of the snapshot you want the base the new
>>>>> virtual disk on
>>>>> userData - optional data that will be attached to the new VD,
>>>>> could
>>>>> be anything that the user desires.
>>>>> options - options to modify VDSMs default behavior
>> IIUC, i can use options to do storage offloads ? For eg. I can create
>> a
>> LUN that represents this VD on my storage array based on the
>> 'options'
>> parameter ? Is this the intended way to use 'options' ?
> No, this has nothing to do with offloads.
> If by "offloads" you mean having other VDSM hosts to the heavy lifting then this is what the option autoFix=False and the fix mechanism is for.
> If you are talking about advanced scsi features (ie. write same) they will be used automatically whenever possible.
> In any case, how we manage LUNs (if they are even used) is an implementation detail.

I am a bit more interested in how storage array offloads ( by that I 
mean, offload VD creation, snapshot, clone etc to the storage array when 
available/possible) can be done from VDSM ?
In the past there were talks of using libSM to do that. How does that 
strategy play in this new Storage API scenario ? I agree its implmn 
detail, but how & where does that implm sit and how it would be 
triggered is not very clear to me. Looking at createVD args, it sounded 
like 'options' seems to be a trigger point for deciding whether to use 
storage offloads or not, but you spoke otherwise :) Can you provide your 
vision on how VDSM can understand the storage array capabilities & 
exploit storgae array offloads in this New Storage API context ? -- 
Thanks deepak




From gpadgett at redhat.com  Mon Dec 17 14:37:57 2012
From: gpadgett at redhat.com (Greg Padgett)
Date: Mon, 17 Dec 2012 09:37:57 -0500
Subject: [Engine-devel] CPU Overcommit Feature
Message-ID: <50CF2E45.9080302@redhat.com>

Hi,

I've been working on a feature to allow CPU Overcommitment of hosts in a 
cluster.  This first stage allows the engine to consider host cpu threads 
as cores for the purposes of VM resource allocation.

This wiki page has further details, your comments are welcome!
http://www.ovirt.org/Features/cpu_overcommit

Thanks in advance,
Greg


From snmishra at linux.vnet.ibm.com  Mon Dec 17 16:09:17 2012
From: snmishra at linux.vnet.ibm.com (snmishra at linux.vnet.ibm.com)
Date: Mon, 17 Dec 2012 08:09:17 -0800
Subject: [Engine-devel] LDAP: Add support for simple authentication over SSL
Message-ID: <20121217080917.Horde.ABuXs5ir309Qz0OtoHFWcgA@imap.linux.ibm.com>


Hi,

    IBM Tivoli Directory Server (ITDS) supports simple authentication  
over SSL. What will it take to add this support? I can help with this  
work item but will need some guidance.

Regards
Sharad Mishra



From smizrahi at redhat.com  Mon Dec 17 16:13:37 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 17 Dec 2012 11:13:37 -0500 (EST)
Subject: [Engine-devel] [vdsm] RFC: New Storage API
In-Reply-To: <50CEA221.6000902@linux.vnet.ibm.com>
Message-ID: <1082511649.19310839.1355760817552.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Deepak C Shetty" <deepakcs at linux.vnet.ibm.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "Shu Ming" <shuming at linux.vnet.ibm.com>, "engine-devel" <engine-devel at ovirt.org>, "VDSM Project Development"
> <vdsm-devel at lists.fedorahosted.org>, "Deepak C Shetty" <deepakcs at linux.vnet.ibm.com>
> Sent: Sunday, December 16, 2012 11:40:01 PM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> On 12/08/2012 01:23 AM, Saggi Mizrahi wrote:
> >
> > ----- Original Message -----
> >> From: "Deepak C Shetty" <deepakcs at linux.vnet.ibm.com>
> >> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> >> Cc: "Shu Ming" <shuming at linux.vnet.ibm.com>, "engine-devel"
> >> <engine-devel at ovirt.org>, "VDSM Project Development"
> >> <vdsm-devel at lists.fedorahosted.org>, "Deepak C Shetty"
> >> <deepakcs at linux.vnet.ibm.com>
> >> Sent: Friday, December 7, 2012 12:23:15 AM
> >> Subject: Re: [vdsm] RFC: New Storage API
> >>
> >> On 12/06/2012 10:22 PM, Saggi Mizrahi wrote:
> >>> ----- Original Message -----
> >>>> From: "Shu Ming" <shuming at linux.vnet.ibm.com>
> >>>> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> >>>> Cc: "VDSM Project Development"
> >>>> <vdsm-devel at lists.fedorahosted.org>, "engine-devel"
> >>>> <engine-devel at ovirt.org>
> >>>> Sent: Thursday, December 6, 2012 11:02:02 AM
> >>>> Subject: Re: [vdsm] RFC: New Storage API
> >>>>
> >>>> Saggi,
> >>>>
> >>>> Thanks for sharing your thought and I get some comments below.
> >>>>
> >>>>
> >>>> Saggi Mizrahi:
> >>>>> I've been throwing a lot of bits out about the new storage API
> >>>>> and
> >>>>> I think it's time to talk a bit.
> >>>>> I will purposefully try and keep implementation details away
> >>>>> and
> >>>>> concentrate about how the API looks and how you use it.
> >>>>>
> >>>>> First major change is in terminology, there is no long a
> >>>>> storage
> >>>>> domain but a storage repository.
> >>>>> This change is done because so many things are already called
> >>>>> domain in the system and this will make things less confusing
> >>>>> for
> >>>>> new-commers with a libvirt background.
> >>>>>
> >>>>> One other changes is that repositories no longer have a UUID.
> >>>>> The UUID was only used in the pool members manifest and is no
> >>>>> longer needed.
> >>>>>
> >>>>>
> >>>>> connectStorageRepository(repoId, repoFormat,
> >>>>> connectionParameters={}):
> >>>>> repoId - is a transient name that will be used to refer to the
> >>>>> connected domain, it is not persisted and doesn't have to be
> >>>>> the
> >>>>> same across the cluster.
> >>>>> repoFormat - Similar to what used to be type (eg. localfs-1.0,
> >>>>> nfs-3.4, clvm-1.2).
> >>>>> connectionParameters - This is format specific and will used to
> >>>>> tell VDSM how to connect to the repo.
> >>>> Where does repoID come from? I think repoID doesn't exist before
> >>>> connectStorageRepository() return.  Isn't repoID a return value
> >>>> of
> >>>> connectStorageRepository()?
> >>> No, repoIDs are no longer part of the domain, they are just a
> >>> transient handle.
> >>> The user can put whatever it wants there as long as it isn't
> >>> already taken by another currently connected domain.
> >> So what happens when user mistakenly gives a repoID that is in use
> >> before.. there should be something in the return value that
> >> specifies
> >> the error and/or reason for error so that user can try with a
> >> new/diff
> >> repoID ?
> > Asi I said, connect fails if the repoId is in use ATM.
> >>>>> disconnectStorageRepository(self, repoId)
> >>>>>
> >>>>>
> >>>>> In the new API there are only images, some images are mutable
> >>>>> and
> >>>>> some are not.
> >>>>> mutable images are also called VirtualDisks
> >>>>> immutable images are also called Snapshots
> >>>>>
> >>>>> There are no explicit templates, you can create as many images
> >>>>> as
> >>>>> you want from any snapshot.
> >>>>>
> >>>>> There are 4 major image operations:
> >>>>>
> >>>>>
> >>>>> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> >>>>>                      userData={}, options={}):
> >>>>>
> >>>>> targetRepoId - ID of a connected repo where the disk will be
> >>>>> created
> >>>>> size - The size of the image you wish to create
> >>>>> baseSnapshotId - the ID of the snapshot you want the base the
> >>>>> new
> >>>>> virtual disk on
> >>>>> userData - optional data that will be attached to the new VD,
> >>>>> could
> >>>>> be anything that the user desires.
> >>>>> options - options to modify VDSMs default behavior
> >> IIUC, i can use options to do storage offloads ? For eg. I can
> >> create
> >> a
> >> LUN that represents this VD on my storage array based on the
> >> 'options'
> >> parameter ? Is this the intended way to use 'options' ?
> > No, this has nothing to do with offloads.
> > If by "offloads" you mean having other VDSM hosts to the heavy
> > lifting then this is what the option autoFix=False and the fix
> > mechanism is for.
> > If you are talking about advanced scsi features (ie. write same)
> > they will be used automatically whenever possible.
> > In any case, how we manage LUNs (if they are even used) is an
> > implementation detail.
> 
> I am a bit more interested in how storage array offloads ( by that I
> mean, offload VD creation, snapshot, clone etc to the storage array
> when
> available/possible) can be done from VDSM ?
> In the past there were talks of using libSM to do that. How does that
> strategy play in this new Storage API scenario ? I agree its implmn
> detail, but how & where does that implm sit and how it would be
> triggered is not very clear to me. Looking at createVD args, it
> sounded
> like 'options' seems to be a trigger point for deciding whether to
> use
> storage offloads or not, but you spoke otherwise :) Can you provide
> your
> vision on how VDSM can understand the storage array capabilities &
> exploit storgae array offloads in this New Storage API context ? --
> Thanks deepak
Some will be used automatically whenever possible (storage offloading).
Features that favor a specific strategy will be activated when the proper strategy (space, performance) option is selected.
In cases when only the user can know whether to use a feature or not we will have options to turn that on.
In any case every domain exports a capability list through GetRepositoryCapabilities() that returns a list off repository capabilities.
Some capabilities are VDSM specific like CLUSTERED or REQUIRES_SRM. Some are storage capabilities like NATIVE_SNAPSHOTS, NATIVE_THIN_PROVISIONING, SPARSE_VOLUMES, etc...

We are also considering an override mechanism where you can disable features in storage that supports it by setting it in the domain options. This will be done with NO_XXXXX (eg. NO_NATIVE_SNAPSHOTS). This will make the domain not use or expose the capability through the API. I assume it will only be used for testing or in cases where the storage array is known to have problems with a certain feature. Not everything can be disables as an example there is no real way to disable NATIVE_THING_PROVISIONING or SPARSE_VOLUMES.
> 
> 


From agl at us.ibm.com  Mon Dec 17 17:00:49 2012
From: agl at us.ibm.com (Adam Litke)
Date: Mon, 17 Dec 2012 11:00:49 -0600
Subject: [Engine-devel] Managing async tasks
Message-ID: <20121217170049.GA32397@aglitke>

On today's vdsm call we had a lively discussion around how asynchronous
operations should be handled in the future.  In an effort to include more people
in the discussion and to better capture the resulting conversation I would like
to continue that discussion here on the mailing list.

A lot of ideas were thrown around about how 'tasks' should be handled in the
future.  There are a lot of ways that it can be done.  To determine how we
should implement it, it's probably best if we start with a set of requirements.
If we can first agree on these, it should be easy to find a solution that meets
them.  I'll take a stab at identifying a first set of POSSIBLE requirements:

- Standardized method for determining the result of an operation

  This is a big one for me because it directly affects the consumability of the
  API.  If each verb has different semantics for discovering whether it has
  completed successfully, then the API will be nearly impossible to use easily.


Sorry.  That's my list :)  Hopefully others will be willing to add other
requirements for consideration.

>From my understanding, task recovery (stop, abort, rollback, etc) will not be
generally supported and should not be a requirement.



-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center



From smizrahi at redhat.com  Mon Dec 17 17:15:08 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 17 Dec 2012 12:15:08 -0500 (EST)
Subject: [Engine-devel] Managing async tasks
In-Reply-To: <20121217170049.GA32397@aglitke>
Message-ID: <2100160127.19338953.1355764508429.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Adam Litke" <agl at us.ibm.com>
> To: vdsm-devel at lists.fedorahosted.org
> Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron" <abaron at redhat.com>, "Saggi Mizrahi" <smizrahi at redhat.com>,
> "Federico Simoncelli" <fsimonce at redhat.com>, engine-devel at ovirt.org
> Sent: Monday, December 17, 2012 12:00:49 PM
> Subject: Managing async tasks
> 
> On today's vdsm call we had a lively discussion around how
> asynchronous
> operations should be handled in the future.  In an effort to include
> more people
> in the discussion and to better capture the resulting conversation I
> would like
> to continue that discussion here on the mailing list.
> 
> A lot of ideas were thrown around about how 'tasks' should be handled
> in the
> future.  There are a lot of ways that it can be done.  To determine
> how we
> should implement it, it's probably best if we start with a set of
> requirements.
> If we can first agree on these, it should be easy to find a solution
> that meets
> them.  I'll take a stab at identifying a first set of POSSIBLE
> requirements:
> 
> - Standardized method for determining the result of an operation
> 
>   This is a big one for me because it directly affects the
>   consumability of the
>   API.  If each verb has different semantics for discovering whether
>   it has
>   completed successfully, then the API will be nearly impossible to
>   use easily.
Since there is no way to assure if of some tasks completed successfully or failed, especially around the murky waters of storage, I say this requirement should be removed.
At least not in the context of a task.
> 
> 
> Sorry.  That's my list :)  Hopefully others will be willing to add
> other
> requirements for consideration.
> 
> From my understanding, task recovery (stop, abort, rollback, etc)
> will not be
> generally supported and should not be a requirement.
> 
> 
> 
> --
> Adam Litke <agl at us.ibm.com>
> IBM Linux Technology Center
> 
> 


From simon at redhat.com  Mon Dec 17 18:13:03 2012
From: simon at redhat.com (Simon Grinberg)
Date: Mon, 17 Dec 2012 13:13:03 -0500 (EST)
Subject: [Engine-devel] CPU Overcommit Feature
In-Reply-To: <50CF2E45.9080302@redhat.com>
Message-ID: <10629066.806.1355767936136.JavaMail.javamailuser@localhost>



----- Original Message -----
> From: "Greg Padgett" <gpadgett at redhat.com>
> To: "engine-devel" <engine-devel at ovirt.org>
> Sent: Monday, December 17, 2012 4:37:57 PM
> Subject: [Engine-devel] CPU Overcommit Feature
> 
> Hi,
> 
> I've been working on a feature to allow CPU Overcommitment of hosts
> in a
> cluster.  This first stage allows the engine to consider host cpu
> threads
> as cores for the purposes of VM resource allocation.
> 
> This wiki page has further details, your comments are welcome!
> http://www.ovirt.org/Features/cpu_overcommit

Basically looking good.
Hyperthread though is vendor specific. 

For AMD it's Clustered Multi-Thread while for Intel it's Hyper-Thread 
Official name is  simultaneous multithreading (SMT) but no one outside of the academy will recognize that.

in libvirt if I read it right it's <attribute name='thread_siblings'>

So why not just call it threads.
We'll have cpuSockets, cpiCores, and cpuThreads, should be clear when in CPU context.

In the GUI just change hyperthreads to CPU threads. While in the tool tip explain that it's either AMD Clustered Multi-Thread or Intel Hyperthread 



> 
> Thanks in advance,
> Greg
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From alonbl at redhat.com  Mon Dec 17 18:42:37 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Mon, 17 Dec 2012 13:42:37 -0500 (EST)
Subject: [Engine-devel] LDAP: Add support for simple authentication over
 SSL
In-Reply-To: <20121217080917.Horde.ABuXs5ir309Qz0OtoHFWcgA@imap.linux.ibm.com>
Message-ID: <299782770.5640234.1355769757605.JavaMail.root@redhat.com>



----- Original Message -----
> From: snmishra at linux.vnet.ibm.com
> To: engine-devel at ovirt.org
> Cc: snmishra at us.ibm.com
> Sent: Monday, December 17, 2012 6:09:17 PM
> Subject: [Engine-devel] LDAP: Add support for simple authentication over SSL
> 
> 
> Hi,
> 
>     IBM Tivoli Directory Server (ITDS) supports simple authentication
> over SSL. What will it take to add this support? I can help with this
> work item but will need some guidance.
> 
> Regards
> Sharad Mishra
> 

Hello,

There was a discussion recently regarding this.

I paste what I wrote then...

Alon

---

Hello Thierry,

If I understand correctly you wish to help in modifying the engine to support non GSSAPI authentication methods.

Following is a quick design goals for this implementation.

I will be glad to improve this.
Alon

---

Implementation should support the following transports:

1. LDAP (plain).
2. LDAP over TLS.
3. LDAP with StartTLS.

Implementation should support the following authentication methods:

1. Simple.
2. Digest-MD5 (plain and strong).

I believe the GSSAPI can be dropped, I see no advantage of using it.

A sample of low level implementation for transport and authentication is attached.

When adding a domain the following facts should be provided:

 1. Search user name.
 2. Search user password.
 3. Transport type (ldap, ldaps, ldap+startTLS)
 4. Authentication (simple, Digest-MD5)
 5. Sever selection policy (failover, round-robin, random).
 6. Server address type (explicit, DNS record)
 7. Server address set.
 8. Optional base DN.
 9. Optional root certificate.
10. Optional certificate chain.
11. Search page size.
10. Query timeout.
etc...

Within product there are two separate components that perform LDAP authentication:

1. User password validation.
2. User permission fetch.

These two components needs to work in share-nothing mode, meaning that each should communicate with directory independently with the other.

USER PASSWORD VALIDATION

Input: user
Input: domain
Input: password
Output: DN of user
Output: success/failure
Credentials used: user/password provided.
Notes: LDAP session should not be cached.
Logic: Perform LDAP bind.

USER PERMISSION FETCH

Input: DN of user (passed by user password validation)
Input: domain (passed by user password validation)
Output: A set of permissions
Credentials used: search user and password configured within system.
Notes: LDAP context can be cached.
Logic: Perform LDAP searches, this is most of current logic.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LDAPSearch.java
Type: text/x-java
Size: 7540 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121217/2c9e41cf/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: UnsecureSSLSocketFactory.java
Type: text/x-java
Size: 2292 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121217/2c9e41cf/attachment-0001.bin>

From agl at us.ibm.com  Mon Dec 17 19:16:25 2012
From: agl at us.ibm.com (Adam Litke)
Date: Mon, 17 Dec 2012 13:16:25 -0600
Subject: [Engine-devel] Managing async tasks
In-Reply-To: <2100160127.19338953.1355764508429.JavaMail.root@redhat.com>
References: <20121217170049.GA32397@aglitke>
	<2100160127.19338953.1355764508429.JavaMail.root@redhat.com>
Message-ID: <20121217191625.GB32397@aglitke>

On Mon, Dec 17, 2012 at 12:15:08PM -0500, Saggi Mizrahi wrote:
> 
> 
> ----- Original Message -----
> > From: "Adam Litke" <agl at us.ibm.com> To: vdsm-devel at lists.fedorahosted.org
> > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron" <abaron at redhat.com>,
> > "Saggi Mizrahi" <smizrahi at redhat.com>, "Federico Simoncelli"
> > <fsimonce at redhat.com>, engine-devel at ovirt.org Sent: Monday, December 17,
> > 2012 12:00:49 PM Subject: Managing async tasks
> > 
> > On today's vdsm call we had a lively discussion around how asynchronous
> > operations should be handled in the future.  In an effort to include more
> > people in the discussion and to better capture the resulting conversation I
> > would like to continue that discussion here on the mailing list.
> > 
> > A lot of ideas were thrown around about how 'tasks' should be handled in the
> > future.  There are a lot of ways that it can be done.  To determine how we
> > should implement it, it's probably best if we start with a set of
> > requirements.  If we can first agree on these, it should be easy to find a
> > solution that meets them.  I'll take a stab at identifying a first set of
> > POSSIBLE requirements:
> > 
> > - Standardized method for determining the result of an operation
> > 
> >   This is a big one for me because it directly affects the consumability of
> >   the API.  If each verb has different semantics for discovering whether it
> >   has completed successfully, then the API will be nearly impossible to use
> >   easily.
> Since there is no way to assure if of some tasks completed successfully or
> failed, especially around the murky waters of storage, I say this requirement
> should be removed.  At least not in the context of a task.

I don't agree.  Please feel free to convince me with some exampled.  If we
cannot provide feedback to a user as to whether their request has been satisfied
or not, then we have some bigger problems to solve.

> > 
> > 
> > Sorry.  That's my list :)  Hopefully others will be willing to add other
> > requirements for consideration.
> > 
> > From my understanding, task recovery (stop, abort, rollback, etc) will not
> > be generally supported and should not be a requirement.
> > 

-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center



From smizrahi at redhat.com  Mon Dec 17 19:52:06 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 17 Dec 2012 14:52:06 -0500 (EST)
Subject: [Engine-devel] Managing async tasks
In-Reply-To: <20121217191625.GB32397@aglitke>
Message-ID: <1929065958.19399160.1355773926677.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Adam Litke" <agl at us.ibm.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron" <abaron at redhat.com>, "Federico Simoncelli"
> <fsimonce at redhat.com>, engine-devel at ovirt.org, vdsm-devel at lists.fedorahosted.org
> Sent: Monday, December 17, 2012 2:16:25 PM
> Subject: Re: Managing async tasks
> 
> On Mon, Dec 17, 2012 at 12:15:08PM -0500, Saggi Mizrahi wrote:
> > 
> > 
> > ----- Original Message -----
> > > From: "Adam Litke" <agl at us.ibm.com> To:
> > > vdsm-devel at lists.fedorahosted.org
> > > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron"
> > > <abaron at redhat.com>,
> > > "Saggi Mizrahi" <smizrahi at redhat.com>, "Federico Simoncelli"
> > > <fsimonce at redhat.com>, engine-devel at ovirt.org Sent: Monday,
> > > December 17,
> > > 2012 12:00:49 PM Subject: Managing async tasks
> > > 
> > > On today's vdsm call we had a lively discussion around how
> > > asynchronous
> > > operations should be handled in the future.  In an effort to
> > > include more
> > > people in the discussion and to better capture the resulting
> > > conversation I
> > > would like to continue that discussion here on the mailing list.
> > > 
> > > A lot of ideas were thrown around about how 'tasks' should be
> > > handled in the
> > > future.  There are a lot of ways that it can be done.  To
> > > determine how we
> > > should implement it, it's probably best if we start with a set of
> > > requirements.  If we can first agree on these, it should be easy
> > > to find a
> > > solution that meets them.  I'll take a stab at identifying a
> > > first set of
> > > POSSIBLE requirements:
> > > 
> > > - Standardized method for determining the result of an operation
> > > 
> > >   This is a big one for me because it directly affects the
> > >   consumability of
> > >   the API.  If each verb has different semantics for discovering
> > >   whether it
> > >   has completed successfully, then the API will be nearly
> > >   impossible to use
> > >   easily.
> > Since there is no way to assure if of some tasks completed
> > successfully or
> > failed, especially around the murky waters of storage, I say this
> > requirement
> > should be removed.  At least not in the context of a task.
> 
> I don't agree.  Please feel free to convince me with some exampled.
>  If we
> cannot provide feedback to a user as to whether their request has
> been satisfied
> or not, then we have some bigger problems to solve.
If VDSM sends a write command to a storage server, and the connection hangs up before the ACK has returned.
The operation has been committed but VDSM has no way of knowing if that happened as far as VDSM is concerned it got an ETIMEO or EIO.
This is the same problem that the engine has with VDSM.
If VDSM creates an image\VM\network\repo but the connection hangs up before the response can be sent back as far as the engine is concerned the operation times out.
This is an inherent issue with clustering.
This is why I want to move away from tasks being *the* trackable objects.
Tasks should be short. As short as possible.
Run VM should just persist the VM information on the VDSM host and return. The rest of the tracking should be done using the VM ID.
Create image should return once VDSM persisted the information about the request on the repository and created the metadata files.
Tracking should be done on the repo or the imageId.
> 
> > > 
> > > 
> > > Sorry.  That's my list :)  Hopefully others will be willing to
> > > add other
> > > requirements for consideration.
> > > 
> > > From my understanding, task recovery (stop, abort, rollback, etc)
> > > will not
> > > be generally supported and should not be a requirement.
> > > 
> 
> --
> Adam Litke <agl at us.ibm.com>
> IBM Linux Technology Center
> 
> 


From smizrahi at redhat.com  Mon Dec 17 20:12:34 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 17 Dec 2012 15:12:34 -0500 (EST)
Subject: [Engine-devel] Managing async tasks
In-Reply-To: <1929065958.19399160.1355773926677.JavaMail.root@redhat.com>
Message-ID: <749477267.19409296.1355775154958.JavaMail.root@redhat.com>

This is an addendum to my previous email.

----- Original Message -----
> From: "Saggi Mizrahi" <smizrahi at redhat.com>
> To: "Adam Litke" <agl at us.ibm.com>
> Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron" <abaron at redhat.com>, "Federico Simoncelli"
> <fsimonce at redhat.com>, engine-devel at ovirt.org, vdsm-devel at lists.fedorahosted.org
> Sent: Monday, December 17, 2012 2:52:06 PM
> Subject: Re: Managing async tasks
> 
> 
> 
> ----- Original Message -----
> > From: "Adam Litke" <agl at us.ibm.com>
> > To: "Saggi Mizrahi" <smizrahi at redhat.com>
> > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron"
> > <abaron at redhat.com>, "Federico Simoncelli"
> > <fsimonce at redhat.com>, engine-devel at ovirt.org,
> > vdsm-devel at lists.fedorahosted.org
> > Sent: Monday, December 17, 2012 2:16:25 PM
> > Subject: Re: Managing async tasks
> > 
> > On Mon, Dec 17, 2012 at 12:15:08PM -0500, Saggi Mizrahi wrote:
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Adam Litke" <agl at us.ibm.com> To:
> > > > vdsm-devel at lists.fedorahosted.org
> > > > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron"
> > > > <abaron at redhat.com>,
> > > > "Saggi Mizrahi" <smizrahi at redhat.com>, "Federico Simoncelli"
> > > > <fsimonce at redhat.com>, engine-devel at ovirt.org Sent: Monday,
> > > > December 17,
> > > > 2012 12:00:49 PM Subject: Managing async tasks
> > > > 
> > > > On today's vdsm call we had a lively discussion around how
> > > > asynchronous
> > > > operations should be handled in the future.  In an effort to
> > > > include more
> > > > people in the discussion and to better capture the resulting
> > > > conversation I
> > > > would like to continue that discussion here on the mailing
> > > > list.
> > > > 
> > > > A lot of ideas were thrown around about how 'tasks' should be
> > > > handled in the
> > > > future.  There are a lot of ways that it can be done.  To
> > > > determine how we
> > > > should implement it, it's probably best if we start with a set
> > > > of
> > > > requirements.  If we can first agree on these, it should be
> > > > easy
> > > > to find a
> > > > solution that meets them.  I'll take a stab at identifying a
> > > > first set of
> > > > POSSIBLE requirements:
> > > > 
> > > > - Standardized method for determining the result of an
> > > > operation
> > > > 
> > > >   This is a big one for me because it directly affects the
> > > >   consumability of
> > > >   the API.  If each verb has different semantics for
> > > >   discovering
> > > >   whether it
> > > >   has completed successfully, then the API will be nearly
> > > >   impossible to use
> > > >   easily.
> > > Since there is no way to assure if of some tasks completed
> > > successfully or
> > > failed, especially around the murky waters of storage, I say this
> > > requirement
> > > should be removed.  At least not in the context of a task.
> > 
> > I don't agree.  Please feel free to convince me with some exampled.
> >  If we
> > cannot provide feedback to a user as to whether their request has
> > been satisfied
> > or not, then we have some bigger problems to solve.
> If VDSM sends a write command to a storage server, and the connection
> hangs up before the ACK has returned.
> The operation has been committed but VDSM has no way of knowing if
> that happened as far as VDSM is concerned it got an ETIMEO or EIO.
> This is the same problem that the engine has with VDSM.
> If VDSM creates an image\VM\network\repo but the connection hangs up
> before the response can be sent back as far as the engine is
> concerned the operation times out.
> This is an inherent issue with clustering.
> This is why I want to move away from tasks being *the* trackable
> objects.
> Tasks should be short. As short as possible.
> Run VM should just persist the VM information on the VDSM host and
> return. The rest of the tracking should be done using the VM ID.
> Create image should return once VDSM persisted the information about
> the request on the repository and created the metadata files.
> Tracking should be done on the repo or the imageId.

The thing is that I know how long a VM object should live (or an Image object).
So tracking it is straight forward. How long a task should live is very problematic and quite context specific.
It depends on what the task is.
I think it's quite confusing from an API standpoint to have every task have a different scope, id requirement and life-cycle.

In VDSM has two types of APIs

CRUD objects - VM, Image, Repository, Bridge, Storage Connections....
General transient methods - getBiosInfo(), getDeviceList()

The latter are quite simple to manage. They don't need any special handling. If you lost a getBiosInfo() call you just send another one, no harm done.
The same is even true with things that "change" the host like getDeviceList()

What we are really arguing about is fitting the CRUD objects to some generic task oriented scheme.
I'm saying it's a waste of time as you can quite easily have flows to recover from each operation.

Create - Check if the object exists
Read - Read again
Update - either update again or read and update if update didn't commit the first time
Delete - Check if object doesn't exist

Each of the objects we CRUD have different life-cycles and ownership semantics.

Danken raised the point that creation has a problem that if it fails there is no way to get why it failed.
This is why Create method should be minimal. They shouldn't create the object just the entry in the respective persistent storage.
Even now storage connections are persisted to disk and then the operation returns and the user polls to see the state of the connection.
The same should be done for everything. Do the minimum required to create the object entry and mark it as "not usable".
For storage connections it's "connecting"
For VMs it's "preparing for launch"
For new images it's "broken" and in some regards "degraded"

I hope this makes things clearer


> > 
> > > > 
> > > > 
> > > > Sorry.  That's my list :)  Hopefully others will be willing to
> > > > add other
> > > > requirements for consideration.
> > > > 
> > > > From my understanding, task recovery (stop, abort, rollback,
> > > > etc)
> > > > will not
> > > > be generally supported and should not be a requirement.
> > > > 
> > 
> > --
> > Adam Litke <agl at us.ibm.com>
> > IBM Linux Technology Center
> > 
> > 
> 


From abaron at redhat.com  Mon Dec 17 22:24:48 2012
From: abaron at redhat.com (Ayal Baron)
Date: Mon, 17 Dec 2012 17:24:48 -0500 (EST)
Subject: [Engine-devel] Managing async tasks
In-Reply-To: <749477267.19409296.1355775154958.JavaMail.root@redhat.com>
Message-ID: <1225731060.26490265.1355783088590.JavaMail.root@redhat.com>



----- Original Message -----
> This is an addendum to my previous email.
> 
> ----- Original Message -----
> > From: "Saggi Mizrahi" <smizrahi at redhat.com>
> > To: "Adam Litke" <agl at us.ibm.com>
> > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron"
> > <abaron at redhat.com>, "Federico Simoncelli"
> > <fsimonce at redhat.com>, engine-devel at ovirt.org,
> > vdsm-devel at lists.fedorahosted.org
> > Sent: Monday, December 17, 2012 2:52:06 PM
> > Subject: Re: Managing async tasks
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Adam Litke" <agl at us.ibm.com>
> > > To: "Saggi Mizrahi" <smizrahi at redhat.com>
> > > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron"
> > > <abaron at redhat.com>, "Federico Simoncelli"
> > > <fsimonce at redhat.com>, engine-devel at ovirt.org,
> > > vdsm-devel at lists.fedorahosted.org
> > > Sent: Monday, December 17, 2012 2:16:25 PM
> > > Subject: Re: Managing async tasks
> > > 
> > > On Mon, Dec 17, 2012 at 12:15:08PM -0500, Saggi Mizrahi wrote:
> > > > 
> > > > 
> > > > ----- Original Message -----
> > > > > From: "Adam Litke" <agl at us.ibm.com> To:
> > > > > vdsm-devel at lists.fedorahosted.org
> > > > > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron"
> > > > > <abaron at redhat.com>,
> > > > > "Saggi Mizrahi" <smizrahi at redhat.com>, "Federico Simoncelli"
> > > > > <fsimonce at redhat.com>, engine-devel at ovirt.org Sent: Monday,
> > > > > December 17,
> > > > > 2012 12:00:49 PM Subject: Managing async tasks
> > > > > 
> > > > > On today's vdsm call we had a lively discussion around how
> > > > > asynchronous
> > > > > operations should be handled in the future.  In an effort to
> > > > > include more
> > > > > people in the discussion and to better capture the resulting
> > > > > conversation I
> > > > > would like to continue that discussion here on the mailing
> > > > > list.
> > > > > 
> > > > > A lot of ideas were thrown around about how 'tasks' should be
> > > > > handled in the
> > > > > future.  There are a lot of ways that it can be done.  To
> > > > > determine how we
> > > > > should implement it, it's probably best if we start with a
> > > > > set
> > > > > of
> > > > > requirements.  If we can first agree on these, it should be
> > > > > easy
> > > > > to find a
> > > > > solution that meets them.  I'll take a stab at identifying a
> > > > > first set of
> > > > > POSSIBLE requirements:
> > > > > 
> > > > > - Standardized method for determining the result of an
> > > > > operation
> > > > > 
> > > > >   This is a big one for me because it directly affects the
> > > > >   consumability of
> > > > >   the API.  If each verb has different semantics for
> > > > >   discovering
> > > > >   whether it
> > > > >   has completed successfully, then the API will be nearly
> > > > >   impossible to use
> > > > >   easily.
> > > > Since there is no way to assure if of some tasks completed
> > > > successfully or
> > > > failed, especially around the murky waters of storage, I say
> > > > this
> > > > requirement
> > > > should be removed.  At least not in the context of a task.
> > > 
> > > I don't agree.  Please feel free to convince me with some
> > > exampled.
> > >  If we
> > > cannot provide feedback to a user as to whether their request has
> > > been satisfied
> > > or not, then we have some bigger problems to solve.
> > If VDSM sends a write command to a storage server, and the
> > connection
> > hangs up before the ACK has returned.
> > The operation has been committed but VDSM has no way of knowing if
> > that happened as far as VDSM is concerned it got an ETIMEO or EIO.
> > This is the same problem that the engine has with VDSM.
> > If VDSM creates an image\VM\network\repo but the connection hangs
> > up
> > before the response can be sent back as far as the engine is
> > concerned the operation times out.
> > This is an inherent issue with clustering.
> > This is why I want to move away from tasks being *the* trackable
> > objects.
> > Tasks should be short. As short as possible.
> > Run VM should just persist the VM information on the VDSM host and
> > return. The rest of the tracking should be done using the VM ID.
> > Create image should return once VDSM persisted the information
> > about
> > the request on the repository and created the metadata files.
> > Tracking should be done on the repo or the imageId.
> 
> The thing is that I know how long a VM object should live (or an
> Image object).
> So tracking it is straight forward. How long a task should live is
> very problematic and quite context specific.
> It depends on what the task is.
> I think it's quite confusing from an API standpoint to have every
> task have a different scope, id requirement and life-cycle.
> 
> In VDSM has two types of APIs
> 
> CRUD objects - VM, Image, Repository, Bridge, Storage Connections....
> General transient methods - getBiosInfo(), getDeviceList()
> 
> The latter are quite simple to manage. They don't need any special
> handling. If you lost a getBiosInfo() call you just send another
> one, no harm done.
> The same is even true with things that "change" the host like
> getDeviceList()
> 
> What we are really arguing about is fitting the CRUD objects to some
> generic task oriented scheme.
> I'm saying it's a waste of time as you can quite easily have flows to
> recover from each operation.
> 
> Create - Check if the object exists
> Read - Read again
> Update - either update again or read and update if update didn't
> commit the first time
> Delete - Check if object doesn't exist
> 
> Each of the objects we CRUD have different life-cycles and ownership
> semantics.
> 
> Danken raised the point that creation has a problem that if it fails
> there is no way to get why it failed.
> This is why Create method should be minimal. They shouldn't create
> the object just the entry in the respective persistent storage.
> Even now storage connections are persisted to disk and then the
> operation returns and the user polls to see the state of the
> connection.
> The same should be done for everything. Do the minimum required to
> create the object entry and mark it as "not usable".
> For storage connections it's "connecting"
> For VMs it's "preparing for launch"
> For new images it's "broken" and in some regards "degraded"
> 
> I hope this makes things clearer

Saggi,

When running an async operation (not task, operation), I want an indication of when it finishes.  This can be either an event sent to me or via polling or by divine intervention, but this is basic information that is required.

Polling for a specific end state is wrong because there can be multiple end states (success, failure 1, failure 2, maybe even multiple options for success, etc).
>From the call I get the feeling that you do support having this just not having it persisted across restarts of the service?
If so, then let's discuss the semantics of what can be reported while vdsm doesn't crash.
If not then in addition to not agreeing with you on this, I have additional problems.  For example, when an operation ends with failure, it is insufficient to know that it failed. I want to know *why* it failed.  Without changing something there is no reason to believe that trying again would succeed.  Without indication of reason of failure I'd just be shooting in the dark.

To keep the discussion focused I will stop here to let you comment.

> 
> 
> > > 
> > > > > 
> > > > > 
> > > > > Sorry.  That's my list :)  Hopefully others will be willing
> > > > > to
> > > > > add other
> > > > > requirements for consideration.
> > > > > 
> > > > > From my understanding, task recovery (stop, abort, rollback,
> > > > > etc)
> > > > > will not
> > > > > be generally supported and should not be a requirement.
> > > > > 
> > > 
> > > --
> > > Adam Litke <agl at us.ibm.com>
> > > IBM Linux Technology Center
> > > 
> > > 
> > 
> 


From smizrahi at redhat.com  Mon Dec 17 22:40:20 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 17 Dec 2012 17:40:20 -0500 (EST)
Subject: [Engine-devel] [Draft]Task Management API
In-Reply-To: <1412234732.19457593.1355783215039.JavaMail.root@redhat.com>
Message-ID: <1311028151.19459886.1355784020609.JavaMail.root@redhat.com>

Dan rightly suggested I'd be more specific about what the task system is
instead of what the task system isn't.

The problem is that I'm not completely sure how it's going to work.
It also depends on the events mechanism.
This is my current working draft:


TaskInfo:
id string
methodName string
kwargs json-object (string keys variant values) *filtered to remove sensitive
                                                 information

getRunningTasks(filter string, filterType enum{glob, regexp})
Returns a list of TaskInfo of all tasks that their id's match the filter


That's it, not even stopTask()

As explained, I would like to offload handling to the subsystems.
In order to make things easier for the clients every subsystem can choose a
filed of the object to be of type OperationInfo.
This is a generic structure that the user has a generic way to track all tasks
on all subsystem with a report interface. The extraData field is for subsystem
specific data. This is where the storage subsystem would put, for example,
imageState (broken, degraded, optimized) data.

OperationInfo:
operationDescription string - something out of an agreed enum of strings
                              vaguely describing the operation at hand for
			      example "Copying", "Merging", "Deleting",
			      "Configuring", "Stopped", "Paused", ....
			      They must be known to the client so it can in
			      turn translate it in the UI. The also have to
			      remain relatively vague as they are part of the
			      interface meaning that new values will break old
			      clients so they have to be reusable.
stageDescription - Similar to operation description in case you want more
                   granularity, optional.
stage (int, int) - (5, 10) means 5 out of 10. 1 out of 1 implies the UI to not
                   display stage widgets.
percentage - 0-100, -1 means unknown.
lastError - (code, message) the same errors that can return for regular calls
extraData - json-object


For example creatVM will return once the object is created in VDSM.
getVmInfo() would return, amongst other things, the operation info.
For the case of preparing for launch it will be:
  {"Creating", "configuring", (2, 4), 40, (0, ""),
   {state="preparing for launch"}}
In the case of VM paused on EIO:
  {"Paused", "Paused", (1, 1), -1, (123, "Error writing to disks"),
   {state="paused"}}

Migration is a tricky one, it will be reported as a task while it's in progress
but all the information is available on the image operationInfo.
In the case of Migration:
  {"Migration", "Configuring", (1, 3), -1, (0, ""), {status="Migrating"}}

For StorageConnection this is somewhat already the case but in simplified
version.

If you want to ask about any other operation I'd be more then happy to write my
suggestion for it.

Subsystems have complete freedom about how to set up the API.
For Storage you have Fixes() to start\stop operations.
Gluster is pretty autonomous once operations have been started.

Since operations return as soon as they are registered (persisted) or fail to
register, it makes synchronous programming a bit clunky.
vdsm.pauseVm(vmId) doesn't return when the VM is paused but when VDSM committed
it will try to pause it. This means you will have to poll in order to see if
the operation finished. For gluster, as an example, this is the only way we
can check that the operation finished.

For stuff we have a bit more control over vdsm will fire events using json-rpc
notifications sent to the clients. The will be in the form of:
{"method": "alert", "params": {
  "alertName": <subsystem>(.<objectType>)?.<object>.(<subobject>., ...),
  "operationInfo", OperationInfo}
}

The user can register to recive events using a glob or a regexp.
registering to vdsm.VM.* pop every time any VM has changed stage.
This means that whenever the task finishes, fails or gains significance progress
and VDSM is there to track it, an event will be sent to the client.

This means that the general flow is.
# Register operation
vmID = best_vm
host.VM.pauseVM(vmID)
while True:
    opInfo = None
    try:
       event = host.waitForEvent("vdsm.VM.best_vm", timeout=10)
       opInfo = event.opInfo
    except VdsmDisconnectionError:
       host.waitForReconnect()
       host.vm.getVmInfo(vmID)  # Double check that we didn't miss the event
       continue
    except Timeout:
       # This is a long operation, poll to see that we didn't miss any event
       # but more commonly, update percentage in the UI to show progress.
       vmInfo = host.vm.getVmInfo(vmID)
       opInfo = vmInfo.operationInfo

    if opInfo.stage.number != op.stage.total:
       # Operation in progress
       updateUI(opInfo)
    else:
       # Operation completed
       # Check that the state is what we expected it to be.
       if oInfo.extraData.state == "paused":
          return SUCCESS
       else:
          return opInfo.lastError


vdsm.waitForEvent(filterm, timeout) is a client side libvdsm helper operation.
Clients that access the raw API need to create thir own client side code to
filter out events and manage their distribution. I'm open to also defining
server side filters but I'm not sure whether it's worth it or just having
it be a boolean (all events or none) is sufficient.

This is a very simplified example but the general flow is clear.
Even if the connection is lost for 1 second or 4 days, the code
still works. Further more, the user can wait for multiple operations
in the same thread using:
  host.waitForEvent("vdsm.VM.(best_vm_ever|not_so_good_vm)")
This means that the client can wait for a 100 VMs or all VMs (using wildecards)
in a mechanism similar "poll()" with minimal overhead. This also means that if
The fact that operations are registered means that even if connections is lost
due to VDSM crashing or the network crashing, the manager doesn't need to care
one the original command returns as it know the operation registered.
This doesn't mean that every operation must retry forever. How persistent
every method is can and should change between the different operations.
Also, it means that manager that didn't initiate an operation track it in
the same way as those that did. This makes clustered managers a lot easier
to implement as if one goes down a second one can more or less immediately with
minimal extra code.


From agl at us.ibm.com  Mon Dec 17 22:41:00 2012
From: agl at us.ibm.com (Adam Litke)
Date: Mon, 17 Dec 2012 16:41:00 -0600
Subject: [Engine-devel] Managing async tasks
In-Reply-To: <749477267.19409296.1355775154958.JavaMail.root@redhat.com>
References: <1929065958.19399160.1355773926677.JavaMail.root@redhat.com>
	<749477267.19409296.1355775154958.JavaMail.root@redhat.com>
Message-ID: <20121217224100.GD32397@aglitke>

On Mon, Dec 17, 2012 at 03:12:34PM -0500, Saggi Mizrahi wrote:
> This is an addendum to my previous email.
> 
> ----- Original Message -----
> > From: "Saggi Mizrahi" <smizrahi at redhat.com>
> > To: "Adam Litke" <agl at us.ibm.com>
> > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron" <abaron at redhat.com>, "Federico Simoncelli"
> > <fsimonce at redhat.com>, engine-devel at ovirt.org, vdsm-devel at lists.fedorahosted.org
> > Sent: Monday, December 17, 2012 2:52:06 PM
> > Subject: Re: Managing async tasks
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Adam Litke" <agl at us.ibm.com>
> > > To: "Saggi Mizrahi" <smizrahi at redhat.com>
> > > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron"
> > > <abaron at redhat.com>, "Federico Simoncelli"
> > > <fsimonce at redhat.com>, engine-devel at ovirt.org,
> > > vdsm-devel at lists.fedorahosted.org
> > > Sent: Monday, December 17, 2012 2:16:25 PM
> > > Subject: Re: Managing async tasks
> > > 
> > > On Mon, Dec 17, 2012 at 12:15:08PM -0500, Saggi Mizrahi wrote:
> > > > 
> > > > 
> > > > ----- Original Message -----
> > > > > From: "Adam Litke" <agl at us.ibm.com> To:
> > > > > vdsm-devel at lists.fedorahosted.org
> > > > > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron"
> > > > > <abaron at redhat.com>,
> > > > > "Saggi Mizrahi" <smizrahi at redhat.com>, "Federico Simoncelli"
> > > > > <fsimonce at redhat.com>, engine-devel at ovirt.org Sent: Monday,
> > > > > December 17,
> > > > > 2012 12:00:49 PM Subject: Managing async tasks
> > > > > 
> > > > > On today's vdsm call we had a lively discussion around how
> > > > > asynchronous
> > > > > operations should be handled in the future.  In an effort to
> > > > > include more
> > > > > people in the discussion and to better capture the resulting
> > > > > conversation I
> > > > > would like to continue that discussion here on the mailing
> > > > > list.
> > > > > 
> > > > > A lot of ideas were thrown around about how 'tasks' should be
> > > > > handled in the
> > > > > future.  There are a lot of ways that it can be done.  To
> > > > > determine how we
> > > > > should implement it, it's probably best if we start with a set
> > > > > of
> > > > > requirements.  If we can first agree on these, it should be
> > > > > easy
> > > > > to find a
> > > > > solution that meets them.  I'll take a stab at identifying a
> > > > > first set of
> > > > > POSSIBLE requirements:
> > > > > 
> > > > > - Standardized method for determining the result of an
> > > > > operation
> > > > > 
> > > > >   This is a big one for me because it directly affects the
> > > > >   consumability of
> > > > >   the API.  If each verb has different semantics for
> > > > >   discovering
> > > > >   whether it
> > > > >   has completed successfully, then the API will be nearly
> > > > >   impossible to use
> > > > >   easily.
> > > > Since there is no way to assure if of some tasks completed
> > > > successfully or
> > > > failed, especially around the murky waters of storage, I say this
> > > > requirement
> > > > should be removed.  At least not in the context of a task.
> > > 
> > > I don't agree.  Please feel free to convince me with some exampled.
> > >  If we
> > > cannot provide feedback to a user as to whether their request has
> > > been satisfied
> > > or not, then we have some bigger problems to solve.
> > If VDSM sends a write command to a storage server, and the connection
> > hangs up before the ACK has returned.
> > The operation has been committed but VDSM has no way of knowing if
> > that happened as far as VDSM is concerned it got an ETIMEO or EIO.
> > This is the same problem that the engine has with VDSM.
> > If VDSM creates an image\VM\network\repo but the connection hangs up
> > before the response can be sent back as far as the engine is
> > concerned the operation times out.
> > This is an inherent issue with clustering.
> > This is why I want to move away from tasks being *the* trackable
> > objects.
> > Tasks should be short. As short as possible.
> > Run VM should just persist the VM information on the VDSM host and
> > return. The rest of the tracking should be done using the VM ID.
> > Create image should return once VDSM persisted the information about
> > the request on the repository and created the metadata files.
> > Tracking should be done on the repo or the imageId.
> 
> The thing is that I know how long a VM object should live (or an Image object).
> So tracking it is straight forward. How long a task should live is very problematic and quite context specific.
> It depends on what the task is.
> I think it's quite confusing from an API standpoint to have every task have a different scope, id requirement and life-cycle.
> 
> In VDSM has two types of APIs
> 
> CRUD objects - VM, Image, Repository, Bridge, Storage Connections....
> General transient methods - getBiosInfo(), getDeviceList()
> 
> The latter are quite simple to manage. They don't need any special handling. If you lost a getBiosInfo() call you just send another one, no harm done.
> The same is even true with things that "change" the host like getDeviceList()
> 
> What we are really arguing about is fitting the CRUD objects to some generic task oriented scheme.
> I'm saying it's a waste of time as you can quite easily have flows to recover from each operation.
> 
> Create - Check if the object exists
> Read - Read again
> Update - either update again or read and update if update didn't commit the first time
> Delete - Check if object doesn't exist
> 
> Each of the objects we CRUD have different life-cycles and ownership semantics.
> 
> Danken raised the point that creation has a problem that if it fails there is no way to get why it failed.
> This is why Create method should be minimal. They shouldn't create the object just the entry in the respective persistent storage.
> Even now storage connections are persisted to disk and then the operation returns and the user polls to see the state of the connection.
> The same should be done for everything. Do the minimum required to create the object entry and mark it as "not usable".
> For storage connections it's "connecting"
> For VMs it's "preparing for launch"
> For new images it's "broken" and in some regards "degraded"
> 
> I hope this makes things clearer

So, I tried to avoid using the word 'Task' in my initial email because I didn't
want to cause any confusion.  I actually like the ideas you mention above.  They
work especially well assuming we have no rollback/cancel operations.

I was hoping someone would volunteer some requirements that would be pertinent
to the 'setupNetwork' flow.  I don't think this one fits very well into the
model above.


-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center



From smizrahi at redhat.com  Mon Dec 17 22:44:55 2012
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 17 Dec 2012 17:44:55 -0500 (EST)
Subject: [Engine-devel] Managing async tasks
In-Reply-To: <1225731060.26490265.1355783088590.JavaMail.root@redhat.com>
Message-ID: <795398725.19460530.1355784295061.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Ayal Baron" <abaron at redhat.com>
> To: "Saggi Mizrahi" <smizrahi at redhat.com>
> Cc: "Dan Kenigsberg" <danken at redhat.com>, "Federico Simoncelli" <fsimonce at redhat.com>, engine-devel at ovirt.org,
> vdsm-devel at lists.fedorahosted.org, "Adam Litke" <agl at us.ibm.com>
> Sent: Monday, December 17, 2012 5:24:48 PM
> Subject: Re: Managing async tasks
> 
> 
> 
> ----- Original Message -----
> > This is an addendum to my previous email.
> > 
> > ----- Original Message -----
> > > From: "Saggi Mizrahi" <smizrahi at redhat.com>
> > > To: "Adam Litke" <agl at us.ibm.com>
> > > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron"
> > > <abaron at redhat.com>, "Federico Simoncelli"
> > > <fsimonce at redhat.com>, engine-devel at ovirt.org,
> > > vdsm-devel at lists.fedorahosted.org
> > > Sent: Monday, December 17, 2012 2:52:06 PM
> > > Subject: Re: Managing async tasks
> > > 
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Adam Litke" <agl at us.ibm.com>
> > > > To: "Saggi Mizrahi" <smizrahi at redhat.com>
> > > > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron"
> > > > <abaron at redhat.com>, "Federico Simoncelli"
> > > > <fsimonce at redhat.com>, engine-devel at ovirt.org,
> > > > vdsm-devel at lists.fedorahosted.org
> > > > Sent: Monday, December 17, 2012 2:16:25 PM
> > > > Subject: Re: Managing async tasks
> > > > 
> > > > On Mon, Dec 17, 2012 at 12:15:08PM -0500, Saggi Mizrahi wrote:
> > > > > 
> > > > > 
> > > > > ----- Original Message -----
> > > > > > From: "Adam Litke" <agl at us.ibm.com> To:
> > > > > > vdsm-devel at lists.fedorahosted.org
> > > > > > Cc: "Dan Kenigsberg" <danken at redhat.com>, "Ayal Baron"
> > > > > > <abaron at redhat.com>,
> > > > > > "Saggi Mizrahi" <smizrahi at redhat.com>, "Federico
> > > > > > Simoncelli"
> > > > > > <fsimonce at redhat.com>, engine-devel at ovirt.org Sent: Monday,
> > > > > > December 17,
> > > > > > 2012 12:00:49 PM Subject: Managing async tasks
> > > > > > 
> > > > > > On today's vdsm call we had a lively discussion around how
> > > > > > asynchronous
> > > > > > operations should be handled in the future.  In an effort
> > > > > > to
> > > > > > include more
> > > > > > people in the discussion and to better capture the
> > > > > > resulting
> > > > > > conversation I
> > > > > > would like to continue that discussion here on the mailing
> > > > > > list.
> > > > > > 
> > > > > > A lot of ideas were thrown around about how 'tasks' should
> > > > > > be
> > > > > > handled in the
> > > > > > future.  There are a lot of ways that it can be done.  To
> > > > > > determine how we
> > > > > > should implement it, it's probably best if we start with a
> > > > > > set
> > > > > > of
> > > > > > requirements.  If we can first agree on these, it should be
> > > > > > easy
> > > > > > to find a
> > > > > > solution that meets them.  I'll take a stab at identifying
> > > > > > a
> > > > > > first set of
> > > > > > POSSIBLE requirements:
> > > > > > 
> > > > > > - Standardized method for determining the result of an
> > > > > > operation
> > > > > > 
> > > > > >   This is a big one for me because it directly affects the
> > > > > >   consumability of
> > > > > >   the API.  If each verb has different semantics for
> > > > > >   discovering
> > > > > >   whether it
> > > > > >   has completed successfully, then the API will be nearly
> > > > > >   impossible to use
> > > > > >   easily.
> > > > > Since there is no way to assure if of some tasks completed
> > > > > successfully or
> > > > > failed, especially around the murky waters of storage, I say
> > > > > this
> > > > > requirement
> > > > > should be removed.  At least not in the context of a task.
> > > > 
> > > > I don't agree.  Please feel free to convince me with some
> > > > exampled.
> > > >  If we
> > > > cannot provide feedback to a user as to whether their request
> > > > has
> > > > been satisfied
> > > > or not, then we have some bigger problems to solve.
> > > If VDSM sends a write command to a storage server, and the
> > > connection
> > > hangs up before the ACK has returned.
> > > The operation has been committed but VDSM has no way of knowing
> > > if
> > > that happened as far as VDSM is concerned it got an ETIMEO or
> > > EIO.
> > > This is the same problem that the engine has with VDSM.
> > > If VDSM creates an image\VM\network\repo but the connection hangs
> > > up
> > > before the response can be sent back as far as the engine is
> > > concerned the operation times out.
> > > This is an inherent issue with clustering.
> > > This is why I want to move away from tasks being *the* trackable
> > > objects.
> > > Tasks should be short. As short as possible.
> > > Run VM should just persist the VM information on the VDSM host
> > > and
> > > return. The rest of the tracking should be done using the VM ID.
> > > Create image should return once VDSM persisted the information
> > > about
> > > the request on the repository and created the metadata files.
> > > Tracking should be done on the repo or the imageId.
> > 
> > The thing is that I know how long a VM object should live (or an
> > Image object).
> > So tracking it is straight forward. How long a task should live is
> > very problematic and quite context specific.
> > It depends on what the task is.
> > I think it's quite confusing from an API standpoint to have every
> > task have a different scope, id requirement and life-cycle.
> > 
> > In VDSM has two types of APIs
> > 
> > CRUD objects - VM, Image, Repository, Bridge, Storage
> > Connections....
> > General transient methods - getBiosInfo(), getDeviceList()
> > 
> > The latter are quite simple to manage. They don't need any special
> > handling. If you lost a getBiosInfo() call you just send another
> > one, no harm done.
> > The same is even true with things that "change" the host like
> > getDeviceList()
> > 
> > What we are really arguing about is fitting the CRUD objects to
> > some
> > generic task oriented scheme.
> > I'm saying it's a waste of time as you can quite easily have flows
> > to
> > recover from each operation.
> > 
> > Create - Check if the object exists
> > Read - Read again
> > Update - either update again or read and update if update didn't
> > commit the first time
> > Delete - Check if object doesn't exist
> > 
> > Each of the objects we CRUD have different life-cycles and
> > ownership
> > semantics.
> > 
> > Danken raised the point that creation has a problem that if it
> > fails
> > there is no way to get why it failed.
> > This is why Create method should be minimal. They shouldn't create
> > the object just the entry in the respective persistent storage.
> > Even now storage connections are persisted to disk and then the
> > operation returns and the user polls to see the state of the
> > connection.
> > The same should be done for everything. Do the minimum required to
> > create the object entry and mark it as "not usable".
> > For storage connections it's "connecting"
> > For VMs it's "preparing for launch"
> > For new images it's "broken" and in some regards "degraded"
> > 
> > I hope this makes things clearer
> 
> Saggi,
> 
> When running an async operation (not task, operation), I want an
> indication of when it finishes.  This can be either an event sent to
> me or via polling or by divine intervention, but this is basic
> information that is required.
> 
> Polling for a specific end state is wrong because there can be
> multiple end states (success, failure 1, failure 2, maybe even
> multiple options for success, etc).
> From the call I get the feeling that you do support having this just
> not having it persisted across restarts of the service?
> If so, then let's discuss the semantics of what can be reported while
> vdsm doesn't crash.
> If not then in addition to not agreeing with you on this, I have
> additional problems.  For example, when an operation ends with
> failure, it is insufficient to know that it failed. I want to know
> *why* it failed.  Without changing something there is no reason to
> believe that trying again would succeed.  Without indication of
> reason of failure I'd just be shooting in the dark.
> 
> To keep the discussion focused I will stop here to let you comment.
I agree, I sent a different email with my suggestion on how to solve all
these problems.
> 
> > 
> > 
> > > > 
> > > > > > 
> > > > > > 
> > > > > > Sorry.  That's my list :)  Hopefully others will be willing
> > > > > > to
> > > > > > add other
> > > > > > requirements for consideration.
> > > > > > 
> > > > > > From my understanding, task recovery (stop, abort,
> > > > > > rollback,
> > > > > > etc)
> > > > > > will not
> > > > > > be generally supported and should not be a requirement.
> > > > > > 
> > > > 
> > > > --
> > > > Adam Litke <agl at us.ibm.com>
> > > > IBM Linux Technology Center
> > > > 
> > > > 
> > > 
> > 
> 


From acathrow at redhat.com  Mon Dec 17 22:52:40 2012
From: acathrow at redhat.com (Andrew Cathrow)
Date: Mon, 17 Dec 2012 17:52:40 -0500 (EST)
Subject: [Engine-devel] CPU Overcommit Feature
In-Reply-To: <10629066.806.1355767936136.JavaMail.javamailuser@localhost>
Message-ID: <4484293.139.1355784756844.JavaMail.javamailuser@localhost>

... and let's not call this CPU overcommit feature. It's nothing like that - it's "Hyperthread handling"

----- Original Message -----
> From: "Simon Grinberg" <simon at redhat.com>
> To: "Greg Padgett" <gpadgett at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Monday, December 17, 2012 1:13:03 PM
> Subject: Re: [Engine-devel] CPU Overcommit Feature
> 
> 
> 
> ----- Original Message -----
> > From: "Greg Padgett" <gpadgett at redhat.com>
> > To: "engine-devel" <engine-devel at ovirt.org>
> > Sent: Monday, December 17, 2012 4:37:57 PM
> > Subject: [Engine-devel] CPU Overcommit Feature
> > 
> > Hi,
> > 
> > I've been working on a feature to allow CPU Overcommitment of hosts
> > in a
> > cluster.  This first stage allows the engine to consider host cpu
> > threads
> > as cores for the purposes of VM resource allocation.
> > 
> > This wiki page has further details, your comments are welcome!
> > http://www.ovirt.org/Features/cpu_overcommit
> 
> Basically looking good.
> Hyperthread though is vendor specific.
> 
> For AMD it's Clustered Multi-Thread while for Intel it's Hyper-Thread
> Official name is  simultaneous multithreading (SMT) but no one
> outside of the academy will recognize that.
> 
> in libvirt if I read it right it's <attribute name='thread_siblings'>
> 
> So why not just call it threads.
> We'll have cpuSockets, cpiCores, and cpuThreads, should be clear when
> in CPU context.
> 
> In the GUI just change hyperthreads to CPU threads. While in the tool
> tip explain that it's either AMD Clustered Multi-Thread or Intel
> Hyperthread
> 
> 
> 
> > 
> > Thanks in advance,
> > Greg
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > 
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From abaron at redhat.com  Mon Dec 17 23:11:09 2012
From: abaron at redhat.com (Ayal Baron)
Date: Mon, 17 Dec 2012 18:11:09 -0500 (EST)
Subject: [Engine-devel] [vdsm] [Draft]Task Management API
In-Reply-To: <1311028151.19459886.1355784020609.JavaMail.root@redhat.com>
Message-ID: <1193982073.26532529.1355785869699.JavaMail.root@redhat.com>



----- Original Message -----
> Dan rightly suggested I'd be more specific about what the task system
> is
> instead of what the task system isn't.
> 
> The problem is that I'm not completely sure how it's going to work.
> It also depends on the events mechanism.
> This is my current working draft:
> 
> 
> TaskInfo:
> id string
> methodName string
> kwargs json-object (string keys variant values) *filtered to remove
> sensitive
>                                                  information
> 
> getRunningTasks(filter string, filterType enum{glob, regexp})
> Returns a list of TaskInfo of all tasks that their id's match the
> filter
> 
> 
> That's it, not even stopTask()

So for each verb we need to implement a specific 'cancel' op if relevant?
e.g. copyImage -> cancelCopyImage ? (or just deleteImage on the target image in this case?)
for migrateVM -> cancelMigrateVM? etc.

> 
> As explained, I would like to offload handling to the subsystems.
> In order to make things easier for the clients every subsystem can
> choose a
> filed of the object to be of type OperationInfo.
> This is a generic structure that the user has a generic way to track
> all tasks
> on all subsystem with a report interface. The extraData field is for
> subsystem
> specific data. This is where the storage subsystem would put, for
> example,
> imageState (broken, degraded, optimized) data.
> 
> OperationInfo:
> operationDescription string - something out of an agreed enum of
> strings
>                               vaguely describing the operation at
>                               hand for
> 			      example "Copying", "Merging", "Deleting",
> 			      "Configuring", "Stopped", "Paused", ....
> 			      They must be known to the client so it can in
> 			      turn translate it in the UI. The also have to
> 			      remain relatively vague as they are part of the
> 			      interface meaning that new values will break old
> 			      clients so they have to be reusable.
> stageDescription - Similar to operation description in case you want
> more
>                    granularity, optional.
> stage (int, int) - (5, 10) means 5 out of 10. 1 out of 1 implies the
> UI to not
>                    display stage widgets.
> percentage - 0-100, -1 means unknown.

I'm assuming this is not overall percentage but relative to current stage?

> lastError - (code, message) the same errors that can return for
> regular calls
> extraData - json-object
> 
> 
> For example creatVM will return once the object is created in VDSM.
> getVmInfo() would return, amongst other things, the operation info.
> For the case of preparing for launch it will be:
>   {"Creating", "configuring", (2, 4), 40, (0, ""),
>    {state="preparing for launch"}}
> In the case of VM paused on EIO:
>   {"Paused", "Paused", (1, 1), -1, (123, "Error writing to disks"),
>    {state="paused"}}
> 
> Migration is a tricky one, it will be reported as a task while it's
> in progress
> but all the information is available on the image operationInfo.
> In the case of Migration:
>   {"Migration", "Configuring", (1, 3), -1, (0, ""),
>   {status="Migrating"}}

Since you wrote 'image' I'm assuming you're referring to storage migration here?
So the migrate command is successful the moment it starts? (or even a bit before, once it persisted info or something) and then to query the state of the migration user needs to call 'image operationInfo'?

> 
> For StorageConnection this is somewhat already the case but in
> simplified
> version.
> 
> If you want to ask about any other operation I'd be more then happy
> to write my
> suggestion for it.
> 
> Subsystems have complete freedom about how to set up the API.
> For Storage you have Fixes() to start\stop operations.

Fixes?

> Gluster is pretty autonomous once operations have been started.
> 
> Since operations return as soon as they are registered (persisted) or
> fail to
> register, it makes synchronous programming a bit clunky.
> vdsm.pauseVm(vmId) doesn't return when the VM is paused but when VDSM
> committed
> it will try to pause it. This means you will have to poll in order to
> see if
> the operation finished. For gluster, as an example, this is the only
> way we
> can check that the operation finished.
> 
> For stuff we have a bit more control over vdsm will fire events using
> json-rpc
> notifications sent to the clients. The will be in the form of:
> {"method": "alert", "params": {
>   "alertName": <subsystem>(.<objectType>)?.<object>.(<subobject>.,
>   ...),
>   "operationInfo", OperationInfo}
> }
> 
> The user can register to recive events using a glob or a regexp.
> registering to vdsm.VM.* pop every time any VM has changed stage.
> This means that whenever the task finishes, fails or gains
> significance progress
> and VDSM is there to track it, an event will be sent to the client.
> 
> This means that the general flow is.
> # Register operation
> vmID = best_vm
> host.VM.pauseVM(vmID)
> while True:
>     opInfo = None
>     try:
>        event = host.waitForEvent("vdsm.VM.best_vm", timeout=10)
>        opInfo = event.opInfo
>     except VdsmDisconnectionError:
>        host.waitForReconnect()
>        host.vm.getVmInfo(vmID)  # Double check that we didn't miss
>        the event
>        continue
>     except Timeout:
>        # This is a long operation, poll to see that we didn't miss
>        any event
>        # but more commonly, update percentage in the UI to show
>        progress.
>        vmInfo = host.vm.getVmInfo(vmID)
>        opInfo = vmInfo.operationInfo
> 
>     if opInfo.stage.number != op.stage.total:
>        # Operation in progress
>        updateUI(opInfo)
>     else:
>        # Operation completed
>        # Check that the state is what we expected it to be.
>        if oInfo.extraData.state == "paused":
>           return SUCCESS
>        else:
>           return opInfo.lastError
> 
> 
> vdsm.waitForEvent(filterm, timeout) is a client side libvdsm helper
> operation.
> Clients that access the raw API need to create thir own client side
> code to
> filter out events and manage their distribution. I'm open to also
> defining
> server side filters but I'm not sure whether it's worth it or just
> having
> it be a boolean (all events or none) is sufficient.
> 
> This is a very simplified example but the general flow is clear.
> Even if the connection is lost for 1 second or 4 days, the code
> still works. Further more, the user can wait for multiple operations
> in the same thread using:
>   host.waitForEvent("vdsm.VM.(best_vm_ever|not_so_good_vm)")
> This means that the client can wait for a 100 VMs or all VMs (using
> wildecards)
> in a mechanism similar "poll()" with minimal overhead. This also
> means that if
> The fact that operations are registered means that even if
> connections is lost
> due to VDSM crashing or the network crashing, the manager doesn't
> need to care
> one the original command returns as it know the operation registered.
> This doesn't mean that every operation must retry forever. How
> persistent
> every method is can and should change between the different
> operations.
> Also, it means that manager that didn't initiate an operation track
> it in
> the same way as those that did. This makes clustered managers a lot
> easier
> to implement as if one goes down a second one can more or less
> immediately with
> minimal extra code.
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 


From rgolan at redhat.com  Tue Dec 18 08:31:37 2012
From: rgolan at redhat.com (Roy Golan)
Date: Tue, 18 Dec 2012 10:31:37 +0200
Subject: [Engine-devel] Revisiting Java7
In-Reply-To: <1134847122.40563373.1354645506659.JavaMail.root@redhat.com>
References: <1134847122.40563373.1354645506659.JavaMail.root@redhat.com>
Message-ID: <50D029E9.70809@redhat.com>

another motive "End of Public Updates for Java SE 6 " 
https://blogs.oracle.com/java/entry/end_of_public_updates_for

On 12/04/2012 08:25 PM, Asaf Shakarchi wrote:
> ----- Original Message -----
>> On 12/04/2012 03:53 PM, Alon Bar-Lev wrote:
>>>
>>> ----- Original Message -----
>>>> From: "Itamar Heim" <iheim at redhat.com>
>>>> To: "Allon Mureinik" <amureini at redhat.com>
>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>> Sent: Tuesday, December 4, 2012 1:13:49 PM
>>>> Subject: Re: [Engine-devel] Revisiting Java7
>>>>
>>>> On 12/04/2012 11:20 AM, Allon Mureinik wrote:
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Itamar Heim" <iheim at redhat.com>
>>>>>> To: "Allon Mureinik" <amureini at redhat.com>
>>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>>>> Sent: Tuesday, December 4, 2012 1:40:10 AM
>>>>>> Subject: Re: [Engine-devel] Revisiting Java7
>>>>>>
>>>>>> On 12/03/2012 04:03 PM, Allon Mureinik wrote:
>>>>>>> Hi guys,
>>>>>>>
>>>>>>> Earlier today, Java6 compatibility was broken
>>>>>>> (http://gerrit.ovirt.org/#/c/9430/).
>>>>>>> This was picked up on pretty quickly, and easily fixed
>>>>>>> (http://gerrit.ovirt.org/#/c/9666/).
>>>>>>>
>>>>>>> However, I think this is a good opportunity to revisit our
>>>>>>> policy
>>>>>>> towards Java 7.
>>>>>>> Currently, we have an "odd" setup, where we recommend running a
>>>>>>> compiling /with/ Java 7 [1] but comply to Java 6 language level
>>>>>>> [2] and JDK [3].
>>>>>>> Inevitably, mistakes like the that happened today will happen.
>>>>>>>
>>>>>>> I know we're holding back due to GWT issues, but looking
>>>>>>> forward
>>>>>>> to
>>>>>>> oVirt 4.0, is Java 7 on our roadmap?
>>>>>> what was the GWT issue?
>>>>> GWT 2.3 that we use doesn't support java 7 syntax.
>>>>> The latest version, 2.5, doesn't either.
>>>>> I'm not sure, however, this is a good reason to enforce the java
>>>>> 6
>>>>> limitation on the entire project (including backed, rest, etc.)
>>>> i agree. you can limit it to the parts used by GWT for now.
>>> The major advantage to move to java7 is the try-with-resources
>>> statement.
>>>
>>> If we can do this for 3.2 it will be great. From what I understand
>>> from Allon, the only noisy change is to move the common and compat
>>> out of the backend directory into its own top-level directory, so
>>> we can apply a different maven policy on these two easily.
>>>
>>> This way the common/compat and frontend will be built using 1.6 and
>>> backend will be built using 1.7.
>> and searchbackend.
>> asaf, is there no easier way to enable java 7 compilation/enforcement
>> except for these 3?
> In the long run consider having a project layer such as :
> Frontend / Backend / Shared [common/compat/searchbackend] , this will allow us to use plugins inheritance cleanly [regarding versions],
>
> Of course inheritance is nicer but it requires a lot of movements (maybe in v4?)
>
>
> For now its easy - we can just use the maven-compiler to define source/target of [v1.7] for the root project, while defining explicitly different versions per frontend/shared module [v1.6]
> Checkstyles should be upgraded as well.
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel



From lhornyak at redhat.com  Tue Dec 18 13:55:39 2012
From: lhornyak at redhat.com (Laszlo Hornyak)
Date: Tue, 18 Dec 2012 08:55:39 -0500 (EST)
Subject: [Engine-devel] jpa annotations
In-Reply-To: <894082910.11417838.1355838785995.JavaMail.root@redhat.com>
Message-ID: <838809794.11418805.1355838939239.JavaMail.root@redhat.com>

Hi,

We have JPA annotations on the backend entity beans, but they have never been actually used as far as I know and now that I looked into VmTemplate for example we do not even have a vm_template table anymore, while VmTemplate still refers to it.
Any plans with the JPA?

Laszlo


From gchaplik at redhat.com  Tue Dec 18 14:03:32 2012
From: gchaplik at redhat.com (Gilad Chaplik)
Date: Tue, 18 Dec 2012 09:03:32 -0500 (EST)
Subject: [Engine-devel] Revisiting Java7
In-Reply-To: <50D029E9.70809@redhat.com>
Message-ID: <1545107992.2218432.1355839412776.JavaMail.root@redhat.com>

----- Original Message -----
> From: "Roy Golan" <rgolan at redhat.com>
> To: engine-devel at ovirt.org
> Sent: Tuesday, December 18, 2012 10:31:37 AM
> Subject: Re: [Engine-devel] Revisiting Java7
> 
> another motive "End of Public Updates for Java SE 6 "
> https://blogs.oracle.com/java/entry/end_of_public_updates_for
> 

I think we can wait with the change, there are no complains yet..


> On 12/04/2012 08:25 PM, Asaf Shakarchi wrote:
> > ----- Original Message -----
> >> On 12/04/2012 03:53 PM, Alon Bar-Lev wrote:
> >>>
> >>> ----- Original Message -----
> >>>> From: "Itamar Heim" <iheim at redhat.com>
> >>>> To: "Allon Mureinik" <amureini at redhat.com>
> >>>> Cc: "engine-devel" <engine-devel at ovirt.org>
> >>>> Sent: Tuesday, December 4, 2012 1:13:49 PM
> >>>> Subject: Re: [Engine-devel] Revisiting Java7
> >>>>
> >>>> On 12/04/2012 11:20 AM, Allon Mureinik wrote:
> >>>>>
> >>>>> ----- Original Message -----
> >>>>>> From: "Itamar Heim" <iheim at redhat.com>
> >>>>>> To: "Allon Mureinik" <amureini at redhat.com>
> >>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
> >>>>>> Sent: Tuesday, December 4, 2012 1:40:10 AM
> >>>>>> Subject: Re: [Engine-devel] Revisiting Java7
> >>>>>>
> >>>>>> On 12/03/2012 04:03 PM, Allon Mureinik wrote:
> >>>>>>> Hi guys,
> >>>>>>>
> >>>>>>> Earlier today, Java6 compatibility was broken
> >>>>>>> (http://gerrit.ovirt.org/#/c/9430/).
> >>>>>>> This was picked up on pretty quickly, and easily fixed
> >>>>>>> (http://gerrit.ovirt.org/#/c/9666/).
> >>>>>>>
> >>>>>>> However, I think this is a good opportunity to revisit our
> >>>>>>> policy
> >>>>>>> towards Java 7.
> >>>>>>> Currently, we have an "odd" setup, where we recommend running
> >>>>>>> a
> >>>>>>> compiling /with/ Java 7 [1] but comply to Java 6 language
> >>>>>>> level
> >>>>>>> [2] and JDK [3].
> >>>>>>> Inevitably, mistakes like the that happened today will
> >>>>>>> happen.
> >>>>>>>
> >>>>>>> I know we're holding back due to GWT issues, but looking
> >>>>>>> forward
> >>>>>>> to
> >>>>>>> oVirt 4.0, is Java 7 on our roadmap?
> >>>>>> what was the GWT issue?
> >>>>> GWT 2.3 that we use doesn't support java 7 syntax.
> >>>>> The latest version, 2.5, doesn't either.
> >>>>> I'm not sure, however, this is a good reason to enforce the
> >>>>> java
> >>>>> 6
> >>>>> limitation on the entire project (including backed, rest, etc.)
> >>>> i agree. you can limit it to the parts used by GWT for now.
> >>> The major advantage to move to java7 is the try-with-resources
> >>> statement.
> >>>
> >>> If we can do this for 3.2 it will be great. From what I
> >>> understand
> >>> from Allon, the only noisy change is to move the common and
> >>> compat
> >>> out of the backend directory into its own top-level directory, so
> >>> we can apply a different maven policy on these two easily.
> >>>
> >>> This way the common/compat and frontend will be built using 1.6
> >>> and
> >>> backend will be built using 1.7.
> >> and searchbackend.
> >> asaf, is there no easier way to enable java 7
> >> compilation/enforcement
> >> except for these 3?
> > In the long run consider having a project layer such as :
> > Frontend / Backend / Shared [common/compat/searchbackend] , this
> > will allow us to use plugins inheritance cleanly [regarding
> > versions],
> >
> > Of course inheritance is nicer but it requires a lot of movements
> > (maybe in v4?)
> >
> >
> > For now its easy - we can just use the maven-compiler to define
> > source/target of [v1.7] for the root project, while defining
> > explicitly different versions per frontend/shared module [v1.6]
> > Checkstyles should be upgraded as well.
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> 
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From gpadgett at redhat.com  Tue Dec 18 14:27:10 2012
From: gpadgett at redhat.com (Greg Padgett)
Date: Tue, 18 Dec 2012 09:27:10 -0500
Subject: [Engine-devel] CPU Overcommit Feature
In-Reply-To: <4484293.139.1355784756844.JavaMail.javamailuser@localhost>
References: <4484293.139.1355784756844.JavaMail.javamailuser@localhost>
Message-ID: <50D07D3E.2080209@redhat.com>

On 12/17/2012 05:52 PM, Andrew Cathrow wrote:
> ... and let's not call this CPU overcommit feature. It's nothing like that - it's "Hyperthread handling"
>
> ----- Original Message -----
>> From: "Simon Grinberg" <simon at redhat.com>
>> To: "Greg Padgett" <gpadgett at redhat.com>
>> Cc: "engine-devel" <engine-devel at ovirt.org>
>> Sent: Monday, December 17, 2012 1:13:03 PM
>> Subject: Re: [Engine-devel] CPU Overcommit Feature
>>
>>
>>
>> ----- Original Message -----
>>> From: "Greg Padgett" <gpadgett at redhat.com>
>>> To: "engine-devel" <engine-devel at ovirt.org>
>>> Sent: Monday, December 17, 2012 4:37:57 PM
>>> Subject: [Engine-devel] CPU Overcommit Feature
>>>
>>> Hi,
>>>
>>> I've been working on a feature to allow CPU Overcommitment of hosts
>>> in a
>>> cluster.  This first stage allows the engine to consider host cpu
>>> threads
>>> as cores for the purposes of VM resource allocation.
>>>
>>> This wiki page has further details, your comments are welcome!
>>> http://www.ovirt.org/Features/cpu_overcommit
>>
>> Basically looking good.
>> Hyperthread though is vendor specific.
>>
>> For AMD it's Clustered Multi-Thread while for Intel it's Hyper-Thread
>> Official name is  simultaneous multithreading (SMT) but no one
>> outside of the academy will recognize that.
>>
>> in libvirt if I read it right it's <attribute name='thread_siblings'>
>>
>> So why not just call it threads.
>> We'll have cpuSockets, cpiCores, and cpuThreads, should be clear when
>> in CPU context.
>>
>> In the GUI just change hyperthreads to CPU threads. While in the tool
>> tip explain that it's either AMD Clustered Multi-Thread or Intel
>> Hyperthread
>>
>>
>>
>>>
>>> Thanks in advance,
>>> Greg
>>> _______________________________________________
>>> Engine-devel mailing list
>>> Engine-devel at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>>>
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>>

Thanks Simon and Andrew.  I've moved the wiki page [1] (with a redirect at 
the old name), updated the terms within to not be vendor-specific, and 
will do the same with the implementation.

[1] http://www.ovirt.org/Features/cpu_thread_handling



From dennisml at conversis.de  Tue Dec 18 17:30:34 2012
From: dennisml at conversis.de (Dennis Jacobfeuerborn)
Date: Tue, 18 Dec 2012 18:30:34 +0100
Subject: [Engine-devel] CPU Overcommit Feature
In-Reply-To: <10629066.806.1355767936136.JavaMail.javamailuser@localhost>
References: <10629066.806.1355767936136.JavaMail.javamailuser@localhost>
Message-ID: <50D0A83A.2060407@conversis.de>

On 12/17/2012 07:13 PM, Simon Grinberg wrote:
> 
> 
> ----- Original Message -----
>> From: "Greg Padgett" <gpadgett at redhat.com>
>> To: "engine-devel" <engine-devel at ovirt.org>
>> Sent: Monday, December 17, 2012 4:37:57 PM
>> Subject: [Engine-devel] CPU Overcommit Feature
>>
>> Hi,
>>
>> I've been working on a feature to allow CPU Overcommitment of hosts
>> in a
>> cluster.  This first stage allows the engine to consider host cpu
>> threads
>> as cores for the purposes of VM resource allocation.
>>
>> This wiki page has further details, your comments are welcome!
>> http://www.ovirt.org/Features/cpu_overcommit
> 
> Basically looking good.
> Hyperthread though is vendor specific. 
> 
> For AMD it's Clustered Multi-Thread while for Intel it's Hyper-Thread 
> Official name is  simultaneous multithreading (SMT) but no one outside of the academy will recognize that.
> 
> in libvirt if I read it right it's <attribute name='thread_siblings'>
> 
> So why not just call it threads.
> We'll have cpuSockets, cpiCores, and cpuThreads, should be clear when in CPU context.
> 
> In the GUI just change hyperthreads to CPU threads. While in the tool tip explain that it's either AMD Clustered Multi-Thread or Intel Hyperthread 

Does this affect only the number of potential vCpus for the guests or does
this also have an impact on the actual scheduling?
So far I always disabled HT out of fear that a 2 vCpu guest might actually
be scheduled to run in 2 threads of the same core but now I'm not so sure
anymore. In the HT case does KVM know that two threads belong to the same
core and will it only schedule its vCpus on distinct cores? Is there some
documentation about this somewhere?

Regards,
  Dennis



From acathrow at redhat.com  Tue Dec 18 17:33:51 2012
From: acathrow at redhat.com (Andrew Cathrow)
Date: Tue, 18 Dec 2012 12:33:51 -0500 (EST)
Subject: [Engine-devel] CPU Overcommit Feature
In-Reply-To: <50D0A83A.2060407@conversis.de>
Message-ID: <1574204484.55205312.1355852031713.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Dennis Jacobfeuerborn" <dennisml at conversis.de>
> To: engine-devel at ovirt.org
> Sent: Tuesday, December 18, 2012 12:30:34 PM
> Subject: Re: [Engine-devel] CPU Overcommit Feature
> 
> On 12/17/2012 07:13 PM, Simon Grinberg wrote:
> > 
> > 
> > ----- Original Message -----
> >> From: "Greg Padgett" <gpadgett at redhat.com>
> >> To: "engine-devel" <engine-devel at ovirt.org>
> >> Sent: Monday, December 17, 2012 4:37:57 PM
> >> Subject: [Engine-devel] CPU Overcommit Feature
> >>
> >> Hi,
> >>
> >> I've been working on a feature to allow CPU Overcommitment of
> >> hosts
> >> in a
> >> cluster.  This first stage allows the engine to consider host cpu
> >> threads
> >> as cores for the purposes of VM resource allocation.
> >>
> >> This wiki page has further details, your comments are welcome!
> >> http://www.ovirt.org/Features/cpu_overcommit
> > 
> > Basically looking good.
> > Hyperthread though is vendor specific.
> > 
> > For AMD it's Clustered Multi-Thread while for Intel it's
> > Hyper-Thread
> > Official name is  simultaneous multithreading (SMT) but no one
> > outside of the academy will recognize that.
> > 
> > in libvirt if I read it right it's <attribute
> > name='thread_siblings'>
> > 
> > So why not just call it threads.
> > We'll have cpuSockets, cpiCores, and cpuThreads, should be clear
> > when in CPU context.
> > 
> > In the GUI just change hyperthreads to CPU threads. While in the
> > tool tip explain that it's either AMD Clustered Multi-Thread or
> > Intel Hyperthread
> 
> Does this affect only the number of potential vCpus for the guests or
> does
> this also have an impact on the actual scheduling?
> So far I always disabled HT out of fear that a 2 vCpu guest might
> actually
> be scheduled to run in 2 threads of the same core but now I'm not so
> sure
> anymore. In the HT case does KVM know that two threads belong to the
> same
> core and will it only schedule its vCpus on distinct cores? Is there
> some
> documentation about this somewhere?

This is about the maximum number of vCPUs we can give to a VM.
If the machine has 32 Physical cores that are hyperthreaded then do we say the max number of vCPUs for a single VM is 32 or 64.


> 
> Regards,
>   Dennis
> 
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From dennisml at conversis.de  Tue Dec 18 17:59:26 2012
From: dennisml at conversis.de (Dennis Jacobfeuerborn)
Date: Tue, 18 Dec 2012 18:59:26 +0100
Subject: [Engine-devel] CPU Overcommit Feature
In-Reply-To: <1574204484.55205312.1355852031713.JavaMail.root@redhat.com>
References: <1574204484.55205312.1355852031713.JavaMail.root@redhat.com>
Message-ID: <50D0AEFE.6010301@conversis.de>

On 12/18/2012 06:33 PM, Andrew Cathrow wrote:
> 
> 
> ----- Original Message -----
>> From: "Dennis Jacobfeuerborn" <dennisml at conversis.de>
>> To: engine-devel at ovirt.org
>> Sent: Tuesday, December 18, 2012 12:30:34 PM
>> Subject: Re: [Engine-devel] CPU Overcommit Feature
>>
>> On 12/17/2012 07:13 PM, Simon Grinberg wrote:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Greg Padgett" <gpadgett at redhat.com>
>>>> To: "engine-devel" <engine-devel at ovirt.org>
>>>> Sent: Monday, December 17, 2012 4:37:57 PM
>>>> Subject: [Engine-devel] CPU Overcommit Feature
>>>>
>>>> Hi,
>>>>
>>>> I've been working on a feature to allow CPU Overcommitment of
>>>> hosts
>>>> in a
>>>> cluster.  This first stage allows the engine to consider host cpu
>>>> threads
>>>> as cores for the purposes of VM resource allocation.
>>>>
>>>> This wiki page has further details, your comments are welcome!
>>>> http://www.ovirt.org/Features/cpu_overcommit
>>>
>>> Basically looking good.
>>> Hyperthread though is vendor specific.
>>>
>>> For AMD it's Clustered Multi-Thread while for Intel it's
>>> Hyper-Thread
>>> Official name is  simultaneous multithreading (SMT) but no one
>>> outside of the academy will recognize that.
>>>
>>> in libvirt if I read it right it's <attribute
>>> name='thread_siblings'>
>>>
>>> So why not just call it threads.
>>> We'll have cpuSockets, cpiCores, and cpuThreads, should be clear
>>> when in CPU context.
>>>
>>> In the GUI just change hyperthreads to CPU threads. While in the
>>> tool tip explain that it's either AMD Clustered Multi-Thread or
>>> Intel Hyperthread
>>
>> Does this affect only the number of potential vCpus for the guests or
>> does
>> this also have an impact on the actual scheduling?
>> So far I always disabled HT out of fear that a 2 vCpu guest might
>> actually
>> be scheduled to run in 2 threads of the same core but now I'm not so
>> sure
>> anymore. In the HT case does KVM know that two threads belong to the
>> same
>> core and will it only schedule its vCpus on distinct cores? Is there
>> some
>> documentation about this somewhere?
> 
> This is about the maximum number of vCPUs we can give to a VM.
> If the machine has 32 Physical cores that are hyperthreaded then do we say the max number of vCPUs for a single VM is 32 or 64.

If the actual scheduling of vCPUs cannot distinguish between threads and
cores then why would you even want to limit yourself to 32 in you example?
In that case the scheduling might end up being inefficient no matter how
many vCPUs you assign to a guest so why restrict the user? (You might of
course want to limit the user for policy reasons but that has nothing to to
with the thread/core topic.)

On the other hand if KVM does only schedule the vCPUs on distinct cores and
extending the count from 32 to 64 implies that this distinction is to be
disabled then this will have a performance impact for the guest.
In that case I might want to limit the guests to just the 32 physical cores
so two vCPUs of a single guest don't get scheduled on two threads of the
same core.

I've never really looked that closely into the scheduling issue but it
might play a role here so I asked if someone had any pointers to infos
about how exactly KVM makes its scheduling decisions.

Regards,
  Dennis



From dfediuck at redhat.com  Tue Dec 18 18:33:32 2012
From: dfediuck at redhat.com (Doron Fediuck)
Date: Tue, 18 Dec 2012 13:33:32 -0500 (EST)
Subject: [Engine-devel] CPU Overcommit Feature
In-Reply-To: <50D0AEFE.6010301@conversis.de>
Message-ID: <1466161084.55251501.1355855612486.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Dennis Jacobfeuerborn" <dennisml at conversis.de>
> To: "Andrew Cathrow" <acathrow at redhat.com>
> Cc: engine-devel at ovirt.org
> Sent: Tuesday, December 18, 2012 7:59:26 PM
> Subject: Re: [Engine-devel] CPU Overcommit Feature
> 
> On 12/18/2012 06:33 PM, Andrew Cathrow wrote:
> > 
> > 
> > ----- Original Message -----
> >> From: "Dennis Jacobfeuerborn" <dennisml at conversis.de>
> >> To: engine-devel at ovirt.org
> >> Sent: Tuesday, December 18, 2012 12:30:34 PM
> >> Subject: Re: [Engine-devel] CPU Overcommit Feature
> >>
> >> On 12/17/2012 07:13 PM, Simon Grinberg wrote:
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Greg Padgett" <gpadgett at redhat.com>
> >>>> To: "engine-devel" <engine-devel at ovirt.org>
> >>>> Sent: Monday, December 17, 2012 4:37:57 PM
> >>>> Subject: [Engine-devel] CPU Overcommit Feature
> >>>>
> >>>> Hi,
> >>>>
> >>>> I've been working on a feature to allow CPU Overcommitment of
> >>>> hosts
> >>>> in a
> >>>> cluster.  This first stage allows the engine to consider host
> >>>> cpu
> >>>> threads
> >>>> as cores for the purposes of VM resource allocation.
> >>>>
> >>>> This wiki page has further details, your comments are welcome!
> >>>> http://www.ovirt.org/Features/cpu_overcommit
> >>>
> >>> Basically looking good.
> >>> Hyperthread though is vendor specific.
> >>>
> >>> For AMD it's Clustered Multi-Thread while for Intel it's
> >>> Hyper-Thread
> >>> Official name is  simultaneous multithreading (SMT) but no one
> >>> outside of the academy will recognize that.
> >>>
> >>> in libvirt if I read it right it's <attribute
> >>> name='thread_siblings'>
> >>>
> >>> So why not just call it threads.
> >>> We'll have cpuSockets, cpiCores, and cpuThreads, should be clear
> >>> when in CPU context.
> >>>
> >>> In the GUI just change hyperthreads to CPU threads. While in the
> >>> tool tip explain that it's either AMD Clustered Multi-Thread or
> >>> Intel Hyperthread
> >>
> >> Does this affect only the number of potential vCpus for the guests
> >> or
> >> does
> >> this also have an impact on the actual scheduling?
> >> So far I always disabled HT out of fear that a 2 vCpu guest might
> >> actually
> >> be scheduled to run in 2 threads of the same core but now I'm not
> >> so
> >> sure
> >> anymore. In the HT case does KVM know that two threads belong to
> >> the
> >> same
> >> core and will it only schedule its vCpus on distinct cores? Is
> >> there
> >> some
> >> documentation about this somewhere?
> > 
> > This is about the maximum number of vCPUs we can give to a VM.
> > If the machine has 32 Physical cores that are hyperthreaded then do
> > we say the max number of vCPUs for a single VM is 32 or 64.
> 
> If the actual scheduling of vCPUs cannot distinguish between threads
> and
> cores then why would you even want to limit yourself to 32 in you
> example?
> In that case the scheduling might end up being inefficient no matter
> how
> many vCPUs you assign to a guest so why restrict the user? (You might
> of
> course want to limit the user for policy reasons but that has nothing
> to to
> with the thread/core topic.)
> 
> On the other hand if KVM does only schedule the vCPUs on distinct
> cores and
> extending the count from 32 to 64 implies that this distinction is to
> be
> disabled then this will have a performance impact for the guest.
> In that case I might want to limit the guests to just the 32 physical
> cores
> so two vCPUs of a single guest don't get scheduled on two threads of
> the
> same core.
> 
> I've never really looked that closely into the scheduling issue but
> it
> might play a role here so I asked if someone had any pointers to
> infos
> about how exactly KVM makes its scheduling decisions.
> 
> Regards,
>   Dennis
> 

Dennis,
first of all every virtual cpu is basically a qemu-thread which can
run on any cpu-thread. The scheduling is done by the kernel scheduler,
while we may control it using cpu pinning. ie- you may ask for
specific vcpu to run on a specific thread which is from the OS
point of view another core.
Indeed there are cases where this is not recommended, but other
cases where this will actually give you a performance boost,
as L1 cache is being shared by the sibling threads.
So it's really up to you to test your workload and decide id you
wish to utilize it or not. We're giving you powerful tools, and
you can decide if and how to use it.

Doron


From snmishra at linux.vnet.ibm.com  Tue Dec 18 21:21:45 2012
From: snmishra at linux.vnet.ibm.com (snmishra at linux.vnet.ibm.com)
Date: Tue, 18 Dec 2012 13:21:45 -0800
Subject: [Engine-devel] LDAP: Add support for simple authentication over
 SSL
In-Reply-To: <299782770.5640234.1355769757605.JavaMail.root@redhat.com>
References: <299782770.5640234.1355769757605.JavaMail.root@redhat.com>
Message-ID: <20121218132145.Horde.dmHwxpir309Q0N5pgyNGEkA@imap.linux.ibm.com>


Quoting Alon Bar-Lev <alonbl at redhat.com>:

> ----- Original Message -----
>> From: snmishra at linux.vnet.ibm.com
>> To: engine-devel at ovirt.org
>> Cc: snmishra at us.ibm.com
>> Sent: Monday, December 17, 2012 6:09:17 PM
>> Subject: [Engine-devel] LDAP: Add support for simple authentication over SSL
>>
>>
>> Hi,
>>
>>     IBM Tivoli Directory Server (ITDS) supports simple authentication
>> over SSL. What will it take to add this support? I can help with this
>> work item but will need some guidance.
>>
>> Regards
>> Sharad Mishra
>>
>
> Hello,
>
> There was a discussion recently regarding this.
>
> I paste what I wrote then...

Alon,

     Thanks for the prompt reply. Does it mean that we will now be  
passing LDAP protocol as an argument. Here is the patch that does it  
(not a working patch) -

@@@@@@@@@@@@@

---  
a/backend/manager/modules/utils/src/main/java/org/ovirt/engine/core/utils/kerberos/JndiAction.java
+++  
b/backend/manager/modules/utils/src/main/java/org/ovirt/engine/core/utils/kerberos/JndiAction.java
@@ -33,14 +33,16 @@ public class JndiAction implements PrivilegedAction {
      private final LdapProviderType ldapProviderType;
      private final StringBuffer userGuid;
      private DnsSRVResult ldapDnsResult;
+    private final String ldapProtocol;
      private final static Logger log = Logger.getLogger(JndiAction.class);

-    public JndiAction(String userName, String domainName,  
StringBuffer userGuid, LdapProviderType ldapProviderType, DnsSRVResult  
ldapDnsResult) {
+    public JndiAction(String userName, String domainName,  
StringBuffer userGuid, LdapProviderType ldapProviderType, DnsSRVResult  
ldapDnsResult, String ldapProtocol) {
          this.userName = userName;
          this.domainName = domainName;
          this.ldapProviderType = ldapProviderType;
          this.userGuid = userGuid;
          this.ldapDnsResult = ldapDnsResult;
+        this.ldapProtocol = ldapProtocol;
      }

      @Override
@@ -48,7 +50,7 @@ public class JndiAction implements PrivilegedAction {
          Hashtable env = new Hashtable(11);
          env.put(Context.INITIAL_CONTEXT_FACTORY,  
"com.sun.jndi.ldap.LdapCtxFactory");
          env.put("java.naming.ldap.attributes.binary", "objectGUID");
-        env.put(Context.SECURITY_AUTHENTICATION, "GSSAPI");
+        env.put(Context.SECURITY_AUTHENTICATION, ldapProtocol);
          env.put("javax.security.sasl.qop", "auth-conf");

          // Send an SRV record DNS query to retrieve all the LDAP  
servers in the domain

@@@@@@@@@@@@@

Thanks
Sharad Mishra

>
> Alon
>
> ---
>
> Hello Thierry,
>
> If I understand correctly you wish to help in modifying the engine  
> to support non GSSAPI authentication methods.
>
> Following is a quick design goals for this implementation.
>
> I will be glad to improve this.
> Alon
>
> ---
>
> Implementation should support the following transports:
>
> 1. LDAP (plain).
> 2. LDAP over TLS.
> 3. LDAP with StartTLS.
>
> Implementation should support the following authentication methods:
>
> 1. Simple.
> 2. Digest-MD5 (plain and strong).
>
> I believe the GSSAPI can be dropped, I see no advantage of using it.
>
> A sample of low level implementation for transport and  
> authentication is attached.
>
> When adding a domain the following facts should be provided:
>
>  1. Search user name.
>  2. Search user password.
>  3. Transport type (ldap, ldaps, ldap+startTLS)
>  4. Authentication (simple, Digest-MD5)
>  5. Sever selection policy (failover, round-robin, random).
>  6. Server address type (explicit, DNS record)
>  7. Server address set.
>  8. Optional base DN.
>  9. Optional root certificate.
> 10. Optional certificate chain.
> 11. Search page size.
> 10. Query timeout.
> etc...
>
> Within product there are two separate components that perform LDAP  
> authentication:
>
> 1. User password validation.
> 2. User permission fetch.
>
> These two components needs to work in share-nothing mode, meaning  
> that each should communicate with directory independently with the  
> other.
>
> USER PASSWORD VALIDATION
>
> Input: user
> Input: domain
> Input: password
> Output: DN of user
> Output: success/failure
> Credentials used: user/password provided.
> Notes: LDAP session should not be cached.
> Logic: Perform LDAP bind.
>
> USER PERMISSION FETCH
>
> Input: DN of user (passed by user password validation)
> Input: domain (passed by user password validation)
> Output: A set of permissions
> Credentials used: search user and password configured within system.
> Notes: LDAP context can be cached.
> Logic: Perform LDAP searches, this is most of current logic.





From alonbl at redhat.com  Tue Dec 18 21:28:36 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Tue, 18 Dec 2012 16:28:36 -0500 (EST)
Subject: [Engine-devel] LDAP: Add support for simple authentication over
 SSL
In-Reply-To: <20121218132145.Horde.dmHwxpir309Q0N5pgyNGEkA@imap.linux.ibm.com>
Message-ID: <94558981.5919785.1355866116938.JavaMail.root@redhat.com>



----- Original Message -----
> From: snmishra at linux.vnet.ibm.com
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: snmishra at us.ibm.com, engine-devel at ovirt.org
> Sent: Tuesday, December 18, 2012 11:21:45 PM
> Subject: Re: [Engine-devel] LDAP: Add support for simple authentication over SSL
> 
> 
> Quoting Alon Bar-Lev <alonbl at redhat.com>:
> 
> > ----- Original Message -----
> >> From: snmishra at linux.vnet.ibm.com
> >> To: engine-devel at ovirt.org
> >> Cc: snmishra at us.ibm.com
> >> Sent: Monday, December 17, 2012 6:09:17 PM
> >> Subject: [Engine-devel] LDAP: Add support for simple
> >> authentication over SSL
> >>
> >>
> >> Hi,
> >>
> >>     IBM Tivoli Directory Server (ITDS) supports simple
> >>     authentication
> >> over SSL. What will it take to add this support? I can help with
> >> this
> >> work item but will need some guidance.
> >>
> >> Regards
> >> Sharad Mishra
> >>
> >
> > Hello,
> >
> > There was a discussion recently regarding this.
> >
> > I paste what I wrote then...
> 
> Alon,
> 
>      Thanks for the prompt reply. Does it mean that we will now be
> passing LDAP protocol as an argument. Here is the patch that does it
> (not a working patch) -
> 

No... it is more than than, please see the attachment I sent.
We need to copy with all combinations of transport and authentication.
We should also add the selected transport and authentication to the manage-domains utility and database.
It is not something we can solve in few lines of code if we want to do that properly.

Regards,
Alon


From iheim at redhat.com  Tue Dec 18 21:52:39 2012
From: iheim at redhat.com (Itamar Heim)
Date: Tue, 18 Dec 2012 23:52:39 +0200
Subject: [Engine-devel] jpa annotations
In-Reply-To: <838809794.11418805.1355838939239.JavaMail.root@redhat.com>
References: <838809794.11418805.1355838939239.JavaMail.root@redhat.com>
Message-ID: <50D0E5A7.9060105@redhat.com>

On 12/18/2012 03:55 PM, Laszlo Hornyak wrote:
> Hi,
>
> We have JPA annotations on the backend entity beans, but they have never been actually used as far as I know and now that I looked into VmTemplate for example we do not even have a vm_template table anymore, while VmTemplate still refers to it.
> Any plans with the JPA?
>

I'm assuming they are from the work on moving to hibernate started a 
long time ago, and question is are we going to finish it anytime soon.




From dennisml at conversis.de  Wed Dec 19 11:49:24 2012
From: dennisml at conversis.de (Dennis Jacobfeuerborn)
Date: Wed, 19 Dec 2012 12:49:24 +0100
Subject: [Engine-devel] CPU Overcommit Feature
In-Reply-To: <1466161084.55251501.1355855612486.JavaMail.root@redhat.com>
References: <1466161084.55251501.1355855612486.JavaMail.root@redhat.com>
Message-ID: <50D1A9C4.3010000@conversis.de>

On 12/18/2012 07:33 PM, Doron Fediuck wrote:
> 
> 
> ----- Original Message -----
>> From: "Dennis Jacobfeuerborn" <dennisml at conversis.de>
>> To: "Andrew Cathrow" <acathrow at redhat.com>
>> Cc: engine-devel at ovirt.org
>> Sent: Tuesday, December 18, 2012 7:59:26 PM
>> Subject: Re: [Engine-devel] CPU Overcommit Feature
>>
>> On 12/18/2012 06:33 PM, Andrew Cathrow wrote:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Dennis Jacobfeuerborn" <dennisml at conversis.de>
>>>> To: engine-devel at ovirt.org
>>>> Sent: Tuesday, December 18, 2012 12:30:34 PM
>>>> Subject: Re: [Engine-devel] CPU Overcommit Feature
>>>>
>>>> On 12/17/2012 07:13 PM, Simon Grinberg wrote:
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Greg Padgett" <gpadgett at redhat.com>
>>>>>> To: "engine-devel" <engine-devel at ovirt.org>
>>>>>> Sent: Monday, December 17, 2012 4:37:57 PM
>>>>>> Subject: [Engine-devel] CPU Overcommit Feature
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I've been working on a feature to allow CPU Overcommitment of
>>>>>> hosts
>>>>>> in a
>>>>>> cluster.  This first stage allows the engine to consider host
>>>>>> cpu
>>>>>> threads
>>>>>> as cores for the purposes of VM resource allocation.
>>>>>>
>>>>>> This wiki page has further details, your comments are welcome!
>>>>>> http://www.ovirt.org/Features/cpu_overcommit
>>>>>
>>>>> Basically looking good.
>>>>> Hyperthread though is vendor specific.
>>>>>
>>>>> For AMD it's Clustered Multi-Thread while for Intel it's
>>>>> Hyper-Thread
>>>>> Official name is  simultaneous multithreading (SMT) but no one
>>>>> outside of the academy will recognize that.
>>>>>
>>>>> in libvirt if I read it right it's <attribute
>>>>> name='thread_siblings'>
>>>>>
>>>>> So why not just call it threads.
>>>>> We'll have cpuSockets, cpiCores, and cpuThreads, should be clear
>>>>> when in CPU context.
>>>>>
>>>>> In the GUI just change hyperthreads to CPU threads. While in the
>>>>> tool tip explain that it's either AMD Clustered Multi-Thread or
>>>>> Intel Hyperthread
>>>>
>>>> Does this affect only the number of potential vCpus for the guests
>>>> or
>>>> does
>>>> this also have an impact on the actual scheduling?
>>>> So far I always disabled HT out of fear that a 2 vCpu guest might
>>>> actually
>>>> be scheduled to run in 2 threads of the same core but now I'm not
>>>> so
>>>> sure
>>>> anymore. In the HT case does KVM know that two threads belong to
>>>> the
>>>> same
>>>> core and will it only schedule its vCpus on distinct cores? Is
>>>> there
>>>> some
>>>> documentation about this somewhere?
>>>
>>> This is about the maximum number of vCPUs we can give to a VM.
>>> If the machine has 32 Physical cores that are hyperthreaded then do
>>> we say the max number of vCPUs for a single VM is 32 or 64.
>>
>> If the actual scheduling of vCPUs cannot distinguish between threads
>> and
>> cores then why would you even want to limit yourself to 32 in you
>> example?
>> In that case the scheduling might end up being inefficient no matter
>> how
>> many vCPUs you assign to a guest so why restrict the user? (You might
>> of
>> course want to limit the user for policy reasons but that has nothing
>> to to
>> with the thread/core topic.)
>>
>> On the other hand if KVM does only schedule the vCPUs on distinct
>> cores and
>> extending the count from 32 to 64 implies that this distinction is to
>> be
>> disabled then this will have a performance impact for the guest.
>> In that case I might want to limit the guests to just the 32 physical
>> cores
>> so two vCPUs of a single guest don't get scheduled on two threads of
>> the
>> same core.
>>
>> I've never really looked that closely into the scheduling issue but
>> it
>> might play a role here so I asked if someone had any pointers to
>> infos
>> about how exactly KVM makes its scheduling decisions.
>>
>> Regards,
>>   Dennis
>>
> 
> Dennis,
> first of all every virtual cpu is basically a qemu-thread which can
> run on any cpu-thread. The scheduling is done by the kernel scheduler,
> while we may control it using cpu pinning. ie- you may ask for
> specific vcpu to run on a specific thread which is from the OS
> point of view another core.
> Indeed there are cases where this is not recommended, but other
> cases where this will actually give you a performance boost,
> as L1 cache is being shared by the sibling threads.
> So it's really up to you to test your workload and decide id you
> wish to utilize it or not. We're giving you powerful tools, and
> you can decide if and how to use it.

What I'm trying to get at is this: Isn't the "Count threads as physical
cores" setting superfluous?

If HT is disabled on the node this doesn't do anything anyway but if it is
enabled what is to be gained by disabling this option? As far as I can see
this makes the UI more complicated for no apparent reason.

Regards,
  Dennis


From mpastern at redhat.com  Wed Dec 19 13:08:42 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Wed, 19 Dec 2012 15:08:42 +0200
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new feature
	discussion/implementation.
Message-ID: <50D1BC5A.9030402@redhat.com>


Hi All,

In many cases OSS maintainers not always can be in the loop of different threads what may
cause them missing important decisions being taken,

As result later on during reviews of the patches they're not accepting (already implemented)
features, what is causing not once for feature to be re-designed and/or delayed, what is wrong
from the development cycle PoV.

Therefore I'd like to suggest establishing dev-rules for the new feature implementation,
what will make entire process much more easer for all of us:

1. discuss new feature on the mailing list (requirements/constraints/etc.)
2. summarise feature details in feature-doc
3. send feature-doc to review to:
   3.1 ML (engine-devel)
   3.2 MG (mailing group of maintainers of the relevant layers)
4. after feature-doc is accepted,
   4.1 implement the feature
   4.2 send it to gerrit for review to:
       4.2.1 lead maintainer/s (they will review/delegate it)


NOTE 3.2, 4.2.1 will require defining MGs such as:
      - engine-devel-core
      - engine-devel-ui
      - engine-devel-api
      - engine-devel-sdk
      - engine-devel-cli
      - engine-devel-vdsm ...


Thoughts?

-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D

-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From mpastern at redhat.com  Wed Dec 19 11:51:32 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Wed, 19 Dec 2012 13:51:32 +0200
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new feature
	discussion/implementation.
Message-ID: <50D1AA44.4040609@redhat.com>

Hi All,

In many cases OSS maintainers not always can be in the loop of different threads what may
cause them missing important decisions being taken,

As result later on during reviews of the patches they're not accepting (already implemented)
features, what is causing not once for feature to be re-designed and/or delayed, what is wrong
from the development cycle PoV.

Therefore I'd like to suggest establishing dev-rules for the new feature implementation,
what will make entire process much more easer for all of us:

1. discuss new feature on the mailing list (requirements/constraints/etc.)
2. summarise feature details in feature-doc
3. send feature-doc to review to:
   3.1 ML (engine-devel)
   3.2 MG (mailing group of maintainers of the relevant layers)
4. after feature-doc is accepted,
   4.1 implement the feature
   4.2 send it to gerrit for review to:
       4.2.1 lead maintainer/s (they will review/delegate it)


NOTE 3.2, 4.2.1 will require defining MGs such as:
      - engine-devel-core
      - engine-devel-ui
      - engine-devel-api
      - engine-devel-sdk
      - engine-devel-cli
      - engine-devel-vdsm ...


Thoughts?

-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From dfediuck at redhat.com  Wed Dec 19 12:18:18 2012
From: dfediuck at redhat.com (Doron Fediuck)
Date: Wed, 19 Dec 2012 07:18:18 -0500 (EST)
Subject: [Engine-devel] CPU Overcommit Feature
In-Reply-To: <50D1A9C4.3010000@conversis.de>
Message-ID: <1471357033.55643997.1355919498993.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Dennis Jacobfeuerborn" <dennisml at conversis.de>
> To: "Doron Fediuck" <dfediuck at redhat.com>
> Cc: engine-devel at ovirt.org, "Andrew Cathrow" <acathrow at redhat.com>
> Sent: Wednesday, December 19, 2012 1:49:24 PM
> Subject: Re: [Engine-devel] CPU Overcommit Feature
> 
> On 12/18/2012 07:33 PM, Doron Fediuck wrote:
> > 
> > 
> > ----- Original Message -----
> >> From: "Dennis Jacobfeuerborn" <dennisml at conversis.de>
> >> To: "Andrew Cathrow" <acathrow at redhat.com>
> >> Cc: engine-devel at ovirt.org
> >> Sent: Tuesday, December 18, 2012 7:59:26 PM
> >> Subject: Re: [Engine-devel] CPU Overcommit Feature
> >>
> >> On 12/18/2012 06:33 PM, Andrew Cathrow wrote:
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Dennis Jacobfeuerborn" <dennisml at conversis.de>
> >>>> To: engine-devel at ovirt.org
> >>>> Sent: Tuesday, December 18, 2012 12:30:34 PM
> >>>> Subject: Re: [Engine-devel] CPU Overcommit Feature
> >>>>
> >>>> On 12/17/2012 07:13 PM, Simon Grinberg wrote:
> >>>>>
> >>>>>
> >>>>> ----- Original Message -----
> >>>>>> From: "Greg Padgett" <gpadgett at redhat.com>
> >>>>>> To: "engine-devel" <engine-devel at ovirt.org>
> >>>>>> Sent: Monday, December 17, 2012 4:37:57 PM
> >>>>>> Subject: [Engine-devel] CPU Overcommit Feature
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I've been working on a feature to allow CPU Overcommitment of
> >>>>>> hosts
> >>>>>> in a
> >>>>>> cluster.  This first stage allows the engine to consider host
> >>>>>> cpu
> >>>>>> threads
> >>>>>> as cores for the purposes of VM resource allocation.
> >>>>>>
> >>>>>> This wiki page has further details, your comments are welcome!
> >>>>>> http://www.ovirt.org/Features/cpu_overcommit
> >>>>>
> >>>>> Basically looking good.
> >>>>> Hyperthread though is vendor specific.
> >>>>>
> >>>>> For AMD it's Clustered Multi-Thread while for Intel it's
> >>>>> Hyper-Thread
> >>>>> Official name is  simultaneous multithreading (SMT) but no one
> >>>>> outside of the academy will recognize that.
> >>>>>
> >>>>> in libvirt if I read it right it's <attribute
> >>>>> name='thread_siblings'>
> >>>>>
> >>>>> So why not just call it threads.
> >>>>> We'll have cpuSockets, cpiCores, and cpuThreads, should be
> >>>>> clear
> >>>>> when in CPU context.
> >>>>>
> >>>>> In the GUI just change hyperthreads to CPU threads. While in
> >>>>> the
> >>>>> tool tip explain that it's either AMD Clustered Multi-Thread or
> >>>>> Intel Hyperthread
> >>>>
> >>>> Does this affect only the number of potential vCpus for the
> >>>> guests
> >>>> or
> >>>> does
> >>>> this also have an impact on the actual scheduling?
> >>>> So far I always disabled HT out of fear that a 2 vCpu guest
> >>>> might
> >>>> actually
> >>>> be scheduled to run in 2 threads of the same core but now I'm
> >>>> not
> >>>> so
> >>>> sure
> >>>> anymore. In the HT case does KVM know that two threads belong to
> >>>> the
> >>>> same
> >>>> core and will it only schedule its vCpus on distinct cores? Is
> >>>> there
> >>>> some
> >>>> documentation about this somewhere?
> >>>
> >>> This is about the maximum number of vCPUs we can give to a VM.
> >>> If the machine has 32 Physical cores that are hyperthreaded then
> >>> do
> >>> we say the max number of vCPUs for a single VM is 32 or 64.
> >>
> >> If the actual scheduling of vCPUs cannot distinguish between
> >> threads
> >> and
> >> cores then why would you even want to limit yourself to 32 in you
> >> example?
> >> In that case the scheduling might end up being inefficient no
> >> matter
> >> how
> >> many vCPUs you assign to a guest so why restrict the user? (You
> >> might
> >> of
> >> course want to limit the user for policy reasons but that has
> >> nothing
> >> to to
> >> with the thread/core topic.)
> >>
> >> On the other hand if KVM does only schedule the vCPUs on distinct
> >> cores and
> >> extending the count from 32 to 64 implies that this distinction is
> >> to
> >> be
> >> disabled then this will have a performance impact for the guest.
> >> In that case I might want to limit the guests to just the 32
> >> physical
> >> cores
> >> so two vCPUs of a single guest don't get scheduled on two threads
> >> of
> >> the
> >> same core.
> >>
> >> I've never really looked that closely into the scheduling issue
> >> but
> >> it
> >> might play a role here so I asked if someone had any pointers to
> >> infos
> >> about how exactly KVM makes its scheduling decisions.
> >>
> >> Regards,
> >>   Dennis
> >>
> > 
> > Dennis,
> > first of all every virtual cpu is basically a qemu-thread which can
> > run on any cpu-thread. The scheduling is done by the kernel
> > scheduler,
> > while we may control it using cpu pinning. ie- you may ask for
> > specific vcpu to run on a specific thread which is from the OS
> > point of view another core.
> > Indeed there are cases where this is not recommended, but other
> > cases where this will actually give you a performance boost,
> > as L1 cache is being shared by the sibling threads.
> > So it's really up to you to test your workload and decide id you
> > wish to utilize it or not. We're giving you powerful tools, and
> > you can decide if and how to use it.
> 
> What I'm trying to get at is this: Isn't the "Count threads as
> physical
> cores" setting superfluous?
> 
> If HT is disabled on the node this doesn't do anything anyway but if
> it is
> enabled what is to be gained by disabling this option? As far as I
> can see
> this makes the UI more complicated for no apparent reason.
> 
> Regards,
>   Dennis
> 

Hi Dennis.
Let's take it one at a time;
UI wise, this is a cluster level policy, and falls into the optimization
tab. So it shouldn't complicate things unless you're looking for specific
optimization. In which-case, I'd expect you to know what HT is all about,
and what should be the proper settings. Additionally, you now may observe
the HT status in the general sub-tab of the hosts. 

Functionality wise, some users actually expects 8 cores to be counted as 16
when HT is on, since they need to utilize it based on their experience
and tests. The efficiency factor HT will provide is something which varies
from one workload to another and from one hardware to another as already
mentioned. The default state of HT optimization is off, so it shouldn't bother
you if you're not interested in such optimization.

Once you start utilizing it, if you turn off the HT in one or more of the
cluster hosts (BIOS setting, requires reboot!), vdsm will report it to the
engine, so the correct number of cores will be used when scheduling a target
host for running or migrating a VM.

Hope this helps,
Doron


From amureini at redhat.com  Wed Dec 19 11:08:27 2012
From: amureini at redhat.com (Allon Mureinik)
Date: Wed, 19 Dec 2012 06:08:27 -0500 (EST)
Subject: [Engine-devel] jpa annotations
In-Reply-To: <50D0E5A7.9060105@redhat.com>
Message-ID: <849327049.17660937.1355915307567.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Tuesday, December 18, 2012 11:52:39 PM
> Subject: Re: [Engine-devel] jpa annotations
> 
> On 12/18/2012 03:55 PM, Laszlo Hornyak wrote:
> > Hi,
> >
> > We have JPA annotations on the backend entity beans, but they have
> > never been actually used as far as I know and now that I looked
> > into VmTemplate for example we do not even have a vm_template
> > table anymore, while VmTemplate still refers to it.
> > Any plans with the JPA?
> >
> 
> I'm assuming they are from the work on moving to hibernate started a
> long time ago, and question is are we going to finish it anytime
> soon.
It's not a case of finishing it, it's a case of redoing it.
These annotations are not being maintainted anymore, and a large percent of them are now irrelevant.

 
> 
> 
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From danken at redhat.com  Wed Dec 19 13:59:11 2012
From: danken at redhat.com (Dan Kenigsberg)
Date: Wed, 19 Dec 2012 15:59:11 +0200
Subject: [Engine-devel] CPU Overcommit Feature
In-Reply-To: <50CF2E45.9080302@redhat.com>
References: <50CF2E45.9080302@redhat.com>
Message-ID: <20121219135911.GP4997@redhat.com>

On Mon, Dec 17, 2012 at 09:37:57AM -0500, Greg Padgett wrote:
> Hi,
> 
> I've been working on a feature to allow CPU Overcommitment of hosts
> in a cluster.  This first stage allows the engine to consider host
> cpu threads as cores for the purposes of VM resource allocation.
> 
> This wiki page has further details, your comments are welcome!
> http://www.ovirt.org/Features/cpu_overcommit

I've commented about the vdsm/engine API on
http://gerrit.ovirt.org/#/c/10144/ but it is probably better to
reiterate it here.

The suggested API is tightly coupled with an ugly hack we pushed to vdsm
in order not to solve the issue properly on the first strike.

If we had not have report_host_threads_as_cores, I think we'd have a
simpler API reporting only cpuThreads and cpuCores; with no funny
boolean flags.

Let us strive to that position as much as we can.

How about asking whoever used report_host_threads_as_cores to unset it
once they install Engine 3.2 ? I think that these are very few people,
that would not mind this very much.

If this is impossible, I'd add a cpuCores2, always reporting the true
number, to be used by new Engines. We may even report it only on the
very few cases of report_host_threads_as_cores being set.

Dan.


From dfediuck at redhat.com  Wed Dec 19 14:53:15 2012
From: dfediuck at redhat.com (Doron Fediuck)
Date: Wed, 19 Dec 2012 09:53:15 -0500 (EST)
Subject: [Engine-devel] CPU Overcommit Feature
In-Reply-To: <20121219135911.GP4997@redhat.com>
Message-ID: <880469598.55741237.1355928795067.JavaMail.root@redhat.com>


----- Original Message -----
> From: "Dan Kenigsberg" <danken at redhat.com>
> To: "Greg Padgett" <gpadgett at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, vdsm-devel at fedorahosted.org
> Sent: Wednesday, December 19, 2012 3:59:11 PM
> Subject: Re: [Engine-devel] CPU Overcommit Feature
> 
> On Mon, Dec 17, 2012 at 09:37:57AM -0500, Greg Padgett wrote:
> > Hi,
> > 
> > I've been working on a feature to allow CPU Overcommitment of hosts
> > in a cluster.  This first stage allows the engine to consider host
> > cpu threads as cores for the purposes of VM resource allocation.
> > 
> > This wiki page has further details, your comments are welcome!
> > http://www.ovirt.org/Features/cpu_overcommit
> 
> I've commented about the vdsm/engine API on
> http://gerrit.ovirt.org/#/c/10144/ but it is probably better to
> reiterate it here.
> 
> The suggested API is tightly coupled with an ugly hack we pushed to
> vdsm
> in order not to solve the issue properly on the first strike.
> 
> If we had not have report_host_threads_as_cores, I think we'd have a
> simpler API reporting only cpuThreads and cpuCores; with no funny
> boolean flags.
> 
> Let us strive to that position as much as we can.
> 
> How about asking whoever used report_host_threads_as_cores to unset
> it
> once they install Engine 3.2 ? I think that these are very few
> people,
> that would not mind this very much.
> 
> If this is impossible, I'd add a cpuCores2, always reporting the true
> number, to be used by new Engines. We may even report it only on the
> very few cases of report_host_threads_as_cores being set.
> 
> Dan.

Hi Dan,
Thanks for the review.

I agree simply reporting cores and threads would be the right solution.
However, when you have hyperthreading turned off you get cores=threads.
This is the same situation you have when hyperthreading turned on, and
someone used the vdsm configuration of reporting threads as cores.

So the engine won't know the real status of the host. We need to be
able to tell the difference. So this moves us to cpuCores2 suggestion.
This is one possibility (cpuRealCores?), and the alternative is an
indication of vdsm config (true/false) which may be removed in the future.
I suspect over time cpu and cpu2 will confuse people.

So I'd suggest having the boolean and removing it along with the vdsm 
configuration in the next ovirt version.

Doron



From snmishra at linux.vnet.ibm.com  Wed Dec 19 15:13:16 2012
From: snmishra at linux.vnet.ibm.com (snmishra at linux.vnet.ibm.com)
Date: Wed, 19 Dec 2012 07:13:16 -0800
Subject: [Engine-devel] LDAP: Add support for simple authentication over
 SSL
In-Reply-To: <94558981.5919785.1355866116938.JavaMail.root@redhat.com>
References: <94558981.5919785.1355866116938.JavaMail.root@redhat.com>
Message-ID: <20121219071316.Horde.RhZYQ5ir309Q0dmMcveGzXA@imap.linux.ibm.com>


Quoting Alon Bar-Lev <alonbl at redhat.com>:

> ----- Original Message -----
>> From: snmishra at linux.vnet.ibm.com
>> To: "Alon Bar-Lev" <alonbl at redhat.com>
>> Cc: snmishra at us.ibm.com, engine-devel at ovirt.org
>> Sent: Tuesday, December 18, 2012 11:21:45 PM
>> Subject: Re: [Engine-devel] LDAP: Add support for simple  
>> authentication over SSL
>>
>>
>> Quoting Alon Bar-Lev <alonbl at redhat.com>:
>>
>> > ----- Original Message -----
>> >> From: snmishra at linux.vnet.ibm.com
>> >> To: engine-devel at ovirt.org
>> >> Cc: snmishra at us.ibm.com
>> >> Sent: Monday, December 17, 2012 6:09:17 PM
>> >> Subject: [Engine-devel] LDAP: Add support for simple
>> >> authentication over SSL
>> >>
>> >>
>> >> Hi,
>> >>
>> >>     IBM Tivoli Directory Server (ITDS) supports simple
>> >>     authentication
>> >> over SSL. What will it take to add this support? I can help with
>> >> this
>> >> work item but will need some guidance.
>> >>
>> >> Regards
>> >> Sharad Mishra
>> >>
>> >
>> > Hello,
>> >
>> > There was a discussion recently regarding this.
>> >
>> > I paste what I wrote then...
>>
>> Alon,
>>
>>      Thanks for the prompt reply. Does it mean that we will now be
>> passing LDAP protocol as an argument. Here is the patch that does it
>> (not a working patch) -
>>
>
> No... it is more than than, please see the attachment I sent.
> We need to copy with all combinations of transport and authentication.
> We should also add the selected transport and authentication to the  
> manage-domains utility and database.
> It is not something we can solve in few lines of code if we want to  
> do that properly.
>

I never meant to indicate that the patch I attached is all I need. It  
will be lot more than what is in the patch. The idea behind the patch  
was to check if we need to pass ldap protocol as an argument like we  
do userid and password right now.

-Sharad
> Regards,
> Alon





From danken at redhat.com  Thu Dec 20 07:43:56 2012
From: danken at redhat.com (Dan Kenigsberg)
Date: Thu, 20 Dec 2012 09:43:56 +0200
Subject: [Engine-devel] CPU Overcommit Feature
In-Reply-To: <880469598.55741237.1355928795067.JavaMail.root@redhat.com>
References: <20121219135911.GP4997@redhat.com>
	<880469598.55741237.1355928795067.JavaMail.root@redhat.com>
Message-ID: <20121220074356.GA22339@redhat.com>

On Wed, Dec 19, 2012 at 09:53:15AM -0500, Doron Fediuck wrote:
> 
> ----- Original Message -----
> > From: "Dan Kenigsberg" <danken at redhat.com>
> > To: "Greg Padgett" <gpadgett at redhat.com>
> > Cc: "engine-devel" <engine-devel at ovirt.org>, vdsm-devel at fedorahosted.org
> > Sent: Wednesday, December 19, 2012 3:59:11 PM
> > Subject: Re: [Engine-devel] CPU Overcommit Feature
> > 
> > On Mon, Dec 17, 2012 at 09:37:57AM -0500, Greg Padgett wrote:
> > > Hi,
> > > 
> > > I've been working on a feature to allow CPU Overcommitment of hosts
> > > in a cluster.  This first stage allows the engine to consider host
> > > cpu threads as cores for the purposes of VM resource allocation.
> > > 
> > > This wiki page has further details, your comments are welcome!
> > > http://www.ovirt.org/Features/cpu_overcommit
> > 
> > I've commented about the vdsm/engine API on
> > http://gerrit.ovirt.org/#/c/10144/ but it is probably better to
> > reiterate it here.
> > 
> > The suggested API is tightly coupled with an ugly hack we pushed to
> > vdsm
> > in order not to solve the issue properly on the first strike.
> > 
> > If we had not have report_host_threads_as_cores, I think we'd have a
> > simpler API reporting only cpuThreads and cpuCores; with no funny
> > boolean flags.
> > 
> > Let us strive to that position as much as we can.
> > 
> > How about asking whoever used report_host_threads_as_cores to unset
> > it
> > once they install Engine 3.2 ? I think that these are very few
> > people,
> > that would not mind this very much.
> > 
> > If this is impossible, I'd add a cpuCores2, always reporting the true
> > number, to be used by new Engines. We may even report it only on the
> > very few cases of report_host_threads_as_cores being set.
> > 
> > Dan.
> 
> Hi Dan,
> Thanks for the review.
> 
> I agree simply reporting cores and threads would be the right solution.
> However, when you have hyperthreading turned off you get cores=threads.
> This is the same situation you have when hyperthreading turned on, and
> someone used the vdsm configuration of reporting threads as cores.
> 
> So the engine won't know the real status of the host.

This is not surprising, as report_host_threads_as_cores means in blunt
English "lie to Engine about the number of cores". The newly suggested
flag says "don't believe what I said in cpuCores, since I'm lying". Next
thing we'd have is another flag saying that the former flag was a lie,
and cpuCores is actually trustworthy.

Instead of dancing this dance, I suggest to stop lying.

report_host_threads_as_cores was a hack to assist a older Engine
versions. Engine users that needed it had to set it out-of-band on their
hosts. Now if they upgrade their Engine, they can -- as easily -- reset
that value.

If they forget, nothing devastating happens beyond Engine assuming that
hyperthreading is off.

Please consider this suggestion. I find it the simplest for all involved
parties.

Dan.

> We need to be
> able to tell the difference. So this moves us to cpuCores2 suggestion.
> This is one possibility (cpuRealCores?), and the alternative is an
> indication of vdsm config (true/false) which may be removed in the future.
> I suspect over time cpu and cpu2 will confuse people.
> 
> So I'd suggest having the boolean and removing it along with the vdsm 
> configuration in the next ovirt version.


From iheim at redhat.com  Thu Dec 20 08:23:55 2012
From: iheim at redhat.com (Itamar Heim)
Date: Thu, 20 Dec 2012 10:23:55 +0200
Subject: [Engine-devel] [vdsm]  CPU Overcommit Feature
In-Reply-To: <20121220074356.GA22339@redhat.com>
References: <20121219135911.GP4997@redhat.com>
	<880469598.55741237.1355928795067.JavaMail.root@redhat.com>
	<20121220074356.GA22339@redhat.com>
Message-ID: <50D2CB1B.1040809@redhat.com>

On 12/20/2012 09:43 AM, Dan Kenigsberg wrote:
> On Wed, Dec 19, 2012 at 09:53:15AM -0500, Doron Fediuck wrote:
>>
>> ----- Original Message -----
>>> From: "Dan Kenigsberg" <danken at redhat.com>
>>> To: "Greg Padgett" <gpadgett at redhat.com>
>>> Cc: "engine-devel" <engine-devel at ovirt.org>, vdsm-devel at fedorahosted.org
>>> Sent: Wednesday, December 19, 2012 3:59:11 PM
>>> Subject: Re: [Engine-devel] CPU Overcommit Feature
>>>
>>> On Mon, Dec 17, 2012 at 09:37:57AM -0500, Greg Padgett wrote:
>>>> Hi,
>>>>
>>>> I've been working on a feature to allow CPU Overcommitment of hosts
>>>> in a cluster.  This first stage allows the engine to consider host
>>>> cpu threads as cores for the purposes of VM resource allocation.
>>>>
>>>> This wiki page has further details, your comments are welcome!
>>>> http://www.ovirt.org/Features/cpu_overcommit
>>>
>>> I've commented about the vdsm/engine API on
>>> http://gerrit.ovirt.org/#/c/10144/ but it is probably better to
>>> reiterate it here.
>>>
>>> The suggested API is tightly coupled with an ugly hack we pushed to
>>> vdsm
>>> in order not to solve the issue properly on the first strike.
>>>
>>> If we had not have report_host_threads_as_cores, I think we'd have a
>>> simpler API reporting only cpuThreads and cpuCores; with no funny
>>> boolean flags.
>>>
>>> Let us strive to that position as much as we can.
>>>
>>> How about asking whoever used report_host_threads_as_cores to unset
>>> it
>>> once they install Engine 3.2 ? I think that these are very few
>>> people,
>>> that would not mind this very much.
>>>
>>> If this is impossible, I'd add a cpuCores2, always reporting the true
>>> number, to be used by new Engines. We may even report it only on the
>>> very few cases of report_host_threads_as_cores being set.
>>>
>>> Dan.
>>
>> Hi Dan,
>> Thanks for the review.
>>
>> I agree simply reporting cores and threads would be the right solution.
>> However, when you have hyperthreading turned off you get cores=threads.
>> This is the same situation you have when hyperthreading turned on, and
>> someone used the vdsm configuration of reporting threads as cores.
>>
>> So the engine won't know the real status of the host.
>
> This is not surprising, as report_host_threads_as_cores means in blunt
> English "lie to Engine about the number of cores". The newly suggested
> flag says "don't believe what I said in cpuCores, since I'm lying". Next
> thing we'd have is another flag saying that the former flag was a lie,
> and cpuCores is actually trustworthy.
>
> Instead of dancing this dance, I suggest to stop lying.
>
> report_host_threads_as_cores was a hack to assist a older Engine
> versions. Engine users that needed it had to set it out-of-band on their
> hosts. Now if they upgrade their Engine, they can -- as easily -- reset
> that value.
>
> If they forget, nothing devastating happens beyond Engine assuming that
> hyperthreading is off.
>
> Please consider this suggestion. I find it the simplest for all involved
> parties.

the only problem is the new vdsm doesn't know which engine may be using it.
if engine would say "getVdsCaps engineVersion=3.2", then vdsm could know 
engine no longer needs lying to and ignore the flag, re-using same field.

>
> Dan.
>
>> We need to be
>> able to tell the difference. So this moves us to cpuCores2 suggestion.
>> This is one possibility (cpuRealCores?), and the alternative is an
>> indication of vdsm config (true/false) which may be removed in the future.
>> I suspect over time cpu and cpu2 will confuse people.
>>
>> So I'd suggest having the boolean and removing it along with the vdsm
>> configuration in the next ovirt version.
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>




From danken at redhat.com  Thu Dec 20 09:55:10 2012
From: danken at redhat.com (Dan Kenigsberg)
Date: Thu, 20 Dec 2012 11:55:10 +0200
Subject: [Engine-devel] [vdsm]  CPU Overcommit Feature
In-Reply-To: <50D2CB1B.1040809@redhat.com>
References: <20121219135911.GP4997@redhat.com>
	<880469598.55741237.1355928795067.JavaMail.root@redhat.com>
	<20121220074356.GA22339@redhat.com> <50D2CB1B.1040809@redhat.com>
Message-ID: <20121220095510.GI22339@redhat.com>

On Thu, Dec 20, 2012 at 10:23:55AM +0200, Itamar Heim wrote:
> On 12/20/2012 09:43 AM, Dan Kenigsberg wrote:
> >On Wed, Dec 19, 2012 at 09:53:15AM -0500, Doron Fediuck wrote:
> >>
> >>----- Original Message -----
> >>>From: "Dan Kenigsberg" <danken at redhat.com>
> >>>To: "Greg Padgett" <gpadgett at redhat.com>
> >>>Cc: "engine-devel" <engine-devel at ovirt.org>, vdsm-devel at fedorahosted.org
> >>>Sent: Wednesday, December 19, 2012 3:59:11 PM
> >>>Subject: Re: [Engine-devel] CPU Overcommit Feature
> >>>
> >>>On Mon, Dec 17, 2012 at 09:37:57AM -0500, Greg Padgett wrote:
> >>>>Hi,
> >>>>
> >>>>I've been working on a feature to allow CPU Overcommitment of hosts
> >>>>in a cluster.  This first stage allows the engine to consider host
> >>>>cpu threads as cores for the purposes of VM resource allocation.
> >>>>
> >>>>This wiki page has further details, your comments are welcome!
> >>>>http://www.ovirt.org/Features/cpu_overcommit
> >>>
> >>>I've commented about the vdsm/engine API on
> >>>http://gerrit.ovirt.org/#/c/10144/ but it is probably better to
> >>>reiterate it here.
> >>>
> >>>The suggested API is tightly coupled with an ugly hack we pushed to
> >>>vdsm
> >>>in order not to solve the issue properly on the first strike.
> >>>
> >>>If we had not have report_host_threads_as_cores, I think we'd have a
> >>>simpler API reporting only cpuThreads and cpuCores; with no funny
> >>>boolean flags.
> >>>
> >>>Let us strive to that position as much as we can.
> >>>
> >>>How about asking whoever used report_host_threads_as_cores to unset
> >>>it
> >>>once they install Engine 3.2 ? I think that these are very few
> >>>people,
> >>>that would not mind this very much.
> >>>
> >>>If this is impossible, I'd add a cpuCores2, always reporting the true
> >>>number, to be used by new Engines. We may even report it only on the
> >>>very few cases of report_host_threads_as_cores being set.
> >>>
> >>>Dan.
> >>
> >>Hi Dan,
> >>Thanks for the review.
> >>
> >>I agree simply reporting cores and threads would be the right solution.
> >>However, when you have hyperthreading turned off you get cores=threads.
> >>This is the same situation you have when hyperthreading turned on, and
> >>someone used the vdsm configuration of reporting threads as cores.
> >>
> >>So the engine won't know the real status of the host.
> >
> >This is not surprising, as report_host_threads_as_cores means in blunt
> >English "lie to Engine about the number of cores". The newly suggested
> >flag says "don't believe what I said in cpuCores, since I'm lying". Next
> >thing we'd have is another flag saying that the former flag was a lie,
> >and cpuCores is actually trustworthy.
> >
> >Instead of dancing this dance, I suggest to stop lying.
> >
> >report_host_threads_as_cores was a hack to assist a older Engine
> >versions. Engine users that needed it had to set it out-of-band on their
> >hosts. Now if they upgrade their Engine, they can -- as easily -- reset
> >that value.
> >
> >If they forget, nothing devastating happens beyond Engine assuming that
> >hyperthreading is off.
> >
> >Please consider this suggestion. I find it the simplest for all involved
> >parties.
> 
> the only problem is the new vdsm doesn't know which engine may be using it.
> if engine would say "getVdsCaps engineVersion=3.2", then vdsm could
> know engine no longer needs lying to and ignore the flag, re-using
> same field.

Note that I do not suggest to drop report_host_threads_as_cores now.
I am suggesting to keep on lying even to new Engine.
If someone thinks that lying is bad, he should reset
report_host_threads_as_cores.

It seems to me that the suggested API is being coerced by a very limited
use case, that is not going to be really harmed by a straight-forward API.

Dan.


From dfediuck at redhat.com  Thu Dec 20 12:14:45 2012
From: dfediuck at redhat.com (Doron Fediuck)
Date: Thu, 20 Dec 2012 07:14:45 -0500 (EST)
Subject: [Engine-devel] [vdsm]  CPU Overcommit Feature
In-Reply-To: <20121220095510.GI22339@redhat.com>
Message-ID: <1939334656.56179607.1356005685698.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Dan Kenigsberg" <danken at redhat.com>
> To: "Itamar Heim" <iheim at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, vdsm-devel at fedorahosted.org
> Sent: Thursday, December 20, 2012 11:55:10 AM
> Subject: Re: [Engine-devel] [vdsm]  CPU Overcommit Feature
> 
> On Thu, Dec 20, 2012 at 10:23:55AM +0200, Itamar Heim wrote:
> > On 12/20/2012 09:43 AM, Dan Kenigsberg wrote:
> > >On Wed, Dec 19, 2012 at 09:53:15AM -0500, Doron Fediuck wrote:
> > >>
> > >>----- Original Message -----
> > >>>From: "Dan Kenigsberg" <danken at redhat.com>
> > >>>To: "Greg Padgett" <gpadgett at redhat.com>
> > >>>Cc: "engine-devel" <engine-devel at ovirt.org>,
> > >>>vdsm-devel at fedorahosted.org
> > >>>Sent: Wednesday, December 19, 2012 3:59:11 PM
> > >>>Subject: Re: [Engine-devel] CPU Overcommit Feature
> > >>>
> > >>>On Mon, Dec 17, 2012 at 09:37:57AM -0500, Greg Padgett wrote:
> > >>>>Hi,
> > >>>>
> > >>>>I've been working on a feature to allow CPU Overcommitment of
> > >>>>hosts
> > >>>>in a cluster.  This first stage allows the engine to consider
> > >>>>host
> > >>>>cpu threads as cores for the purposes of VM resource
> > >>>>allocation.
> > >>>>
> > >>>>This wiki page has further details, your comments are welcome!
> > >>>>http://www.ovirt.org/Features/cpu_overcommit
> > >>>
> > >>>I've commented about the vdsm/engine API on
> > >>>http://gerrit.ovirt.org/#/c/10144/ but it is probably better to
> > >>>reiterate it here.
> > >>>
> > >>>The suggested API is tightly coupled with an ugly hack we pushed
> > >>>to
> > >>>vdsm
> > >>>in order not to solve the issue properly on the first strike.
> > >>>
> > >>>If we had not have report_host_threads_as_cores, I think we'd
> > >>>have a
> > >>>simpler API reporting only cpuThreads and cpuCores; with no
> > >>>funny
> > >>>boolean flags.
> > >>>
> > >>>Let us strive to that position as much as we can.
> > >>>
> > >>>How about asking whoever used report_host_threads_as_cores to
> > >>>unset
> > >>>it
> > >>>once they install Engine 3.2 ? I think that these are very few
> > >>>people,
> > >>>that would not mind this very much.
> > >>>
> > >>>If this is impossible, I'd add a cpuCores2, always reporting the
> > >>>true
> > >>>number, to be used by new Engines. We may even report it only on
> > >>>the
> > >>>very few cases of report_host_threads_as_cores being set.
> > >>>
> > >>>Dan.
> > >>
> > >>Hi Dan,
> > >>Thanks for the review.
> > >>
> > >>I agree simply reporting cores and threads would be the right
> > >>solution.
> > >>However, when you have hyperthreading turned off you get
> > >>cores=threads.
> > >>This is the same situation you have when hyperthreading turned
> > >>on, and
> > >>someone used the vdsm configuration of reporting threads as
> > >>cores.
> > >>
> > >>So the engine won't know the real status of the host.
> > >
> > >This is not surprising, as report_host_threads_as_cores means in
> > >blunt
> > >English "lie to Engine about the number of cores". The newly
> > >suggested
> > >flag says "don't believe what I said in cpuCores, since I'm
> > >lying". Next
> > >thing we'd have is another flag saying that the former flag was a
> > >lie,
> > >and cpuCores is actually trustworthy.
> > >
> > >Instead of dancing this dance, I suggest to stop lying.
> > >
> > >report_host_threads_as_cores was a hack to assist a older Engine
> > >versions. Engine users that needed it had to set it out-of-band on
> > >their
> > >hosts. Now if they upgrade their Engine, they can -- as easily --
> > >reset
> > >that value.
> > >
> > >If they forget, nothing devastating happens beyond Engine assuming
> > >that
> > >hyperthreading is off.
> > >
> > >Please consider this suggestion. I find it the simplest for all
> > >involved
> > >parties.
> > 
> > the only problem is the new vdsm doesn't know which engine may be
> > using it.
> > if engine would say "getVdsCaps engineVersion=3.2", then vdsm could
> > know engine no longer needs lying to and ignore the flag, re-using
> > same field.
> 
> Note that I do not suggest to drop report_host_threads_as_cores now.
> I am suggesting to keep on lying even to new Engine.
> If someone thinks that lying is bad, he should reset
> report_host_threads_as_cores.
> 
> It seems to me that the suggested API is being coerced by a very
> limited
> use case, that is not going to be really harmed by a straight-forward
> API.
> 
> Dan.

Dan,
Did some further checking, and we can go with it;
So basically now we add cpuThreads. Additionally, if the report_host_threads_as_cores
is turned on, an additional cpuCoresReal will be reported.
Since this is a 3.2 cluster feature, the engine will make sure to update
the DB with the real numbers, and older clusters will use the information
vdsm reports, while blocking the engine side optimization.
An update path is already available here: http://gerrit.ovirt.org/#/c/10144/4

I hope that for cluster 3.3 we'll be able to drop the vdsm configuration.
A review would be highly appreciated due to the holidays proximity.

Doron


From simon at redhat.com  Thu Dec 20 14:56:14 2012
From: simon at redhat.com (Simon Grinberg)
Date: Thu, 20 Dec 2012 09:56:14 -0500 (EST)
Subject: [Engine-devel] [vdsm]  CPU Overcommit Feature
In-Reply-To: <1939334656.56179607.1356005685698.JavaMail.root@redhat.com>
Message-ID: <9336026.968.1356015327538.JavaMail.javamailuser@localhost>



----- Original Message -----
> From: "Doron Fediuck" <dfediuck at redhat.com>
> To: "Dan Kenigsberg" <danken at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, vdsm-devel at fedorahosted.org
> Sent: Thursday, December 20, 2012 2:14:45 PM
> Subject: Re: [Engine-devel] [vdsm]  CPU Overcommit Feature
> 
> 
> 
> ----- Original Message -----
> > From: "Dan Kenigsberg" <danken at redhat.com>
> > To: "Itamar Heim" <iheim at redhat.com>
> > Cc: "engine-devel" <engine-devel at ovirt.org>,
> > vdsm-devel at fedorahosted.org
> > Sent: Thursday, December 20, 2012 11:55:10 AM
> > Subject: Re: [Engine-devel] [vdsm]  CPU Overcommit Feature
> > 
> > On Thu, Dec 20, 2012 at 10:23:55AM +0200, Itamar Heim wrote:
> > > On 12/20/2012 09:43 AM, Dan Kenigsberg wrote:
> > > >On Wed, Dec 19, 2012 at 09:53:15AM -0500, Doron Fediuck wrote:
> > > >>
> > > >>----- Original Message -----
> > > >>>From: "Dan Kenigsberg" <danken at redhat.com>
> > > >>>To: "Greg Padgett" <gpadgett at redhat.com>
> > > >>>Cc: "engine-devel" <engine-devel at ovirt.org>,
> > > >>>vdsm-devel at fedorahosted.org
> > > >>>Sent: Wednesday, December 19, 2012 3:59:11 PM
> > > >>>Subject: Re: [Engine-devel] CPU Overcommit Feature
> > > >>>
> > > >>>On Mon, Dec 17, 2012 at 09:37:57AM -0500, Greg Padgett wrote:
> > > >>>>Hi,
> > > >>>>
> > > >>>>I've been working on a feature to allow CPU Overcommitment of
> > > >>>>hosts
> > > >>>>in a cluster.  This first stage allows the engine to consider
> > > >>>>host
> > > >>>>cpu threads as cores for the purposes of VM resource
> > > >>>>allocation.
> > > >>>>
> > > >>>>This wiki page has further details, your comments are
> > > >>>>welcome!
> > > >>>>http://www.ovirt.org/Features/cpu_overcommit
> > > >>>
> > > >>>I've commented about the vdsm/engine API on
> > > >>>http://gerrit.ovirt.org/#/c/10144/ but it is probably better
> > > >>>to
> > > >>>reiterate it here.
> > > >>>
> > > >>>The suggested API is tightly coupled with an ugly hack we
> > > >>>pushed
> > > >>>to
> > > >>>vdsm
> > > >>>in order not to solve the issue properly on the first strike.
> > > >>>
> > > >>>If we had not have report_host_threads_as_cores, I think we'd
> > > >>>have a
> > > >>>simpler API reporting only cpuThreads and cpuCores; with no
> > > >>>funny
> > > >>>boolean flags.
> > > >>>
> > > >>>Let us strive to that position as much as we can.
> > > >>>
> > > >>>How about asking whoever used report_host_threads_as_cores to
> > > >>>unset
> > > >>>it
> > > >>>once they install Engine 3.2 ? I think that these are very few
> > > >>>people,
> > > >>>that would not mind this very much.
> > > >>>
> > > >>>If this is impossible, I'd add a cpuCores2, always reporting
> > > >>>the
> > > >>>true
> > > >>>number, to be used by new Engines. We may even report it only
> > > >>>on
> > > >>>the
> > > >>>very few cases of report_host_threads_as_cores being set.
> > > >>>
> > > >>>Dan.
> > > >>
> > > >>Hi Dan,
> > > >>Thanks for the review.
> > > >>
> > > >>I agree simply reporting cores and threads would be the right
> > > >>solution.
> > > >>However, when you have hyperthreading turned off you get
> > > >>cores=threads.
> > > >>This is the same situation you have when hyperthreading turned
> > > >>on, and
> > > >>someone used the vdsm configuration of reporting threads as
> > > >>cores.
> > > >>
> > > >>So the engine won't know the real status of the host.
> > > >
> > > >This is not surprising, as report_host_threads_as_cores means in
> > > >blunt
> > > >English "lie to Engine about the number of cores". The newly
> > > >suggested
> > > >flag says "don't believe what I said in cpuCores, since I'm
> > > >lying". Next
> > > >thing we'd have is another flag saying that the former flag was
> > > >a
> > > >lie,
> > > >and cpuCores is actually trustworthy.
> > > >
> > > >Instead of dancing this dance, I suggest to stop lying.
> > > >
> > > >report_host_threads_as_cores was a hack to assist a older Engine
> > > >versions. Engine users that needed it had to set it out-of-band
> > > >on
> > > >their
> > > >hosts. Now if they upgrade their Engine, they can -- as easily
> > > >--
> > > >reset
> > > >that value.
> > > >
> > > >If they forget, nothing devastating happens beyond Engine
> > > >assuming
> > > >that
> > > >hyperthreading is off.
> > > >
> > > >Please consider this suggestion. I find it the simplest for all
> > > >involved
> > > >parties.
> > > 
> > > the only problem is the new vdsm doesn't know which engine may be
> > > using it.
> > > if engine would say "getVdsCaps engineVersion=3.2", then vdsm
> > > could
> > > know engine no longer needs lying to and ignore the flag,
> > > re-using
> > > same field.
> > 
> > Note that I do not suggest to drop report_host_threads_as_cores
> > now.
> > I am suggesting to keep on lying even to new Engine.
> > If someone thinks that lying is bad, he should reset
> > report_host_threads_as_cores.
> > 
> > It seems to me that the suggested API is being coerced by a very
> > limited
> > use case, that is not going to be really harmed by a
> > straight-forward
> > API.
> > 
> > Dan.
> 
> Dan,
> Did some further checking, and we can go with it;
> So basically now we add cpuThreads. Additionally, if the
> report_host_threads_as_cores
> is turned on, an additional cpuCoresReal will be reported.

No need for that.
There is only one problematic state where VDSM cheats and reports cores == threads.
This does not happen by mistake, the user specifically asked for it.

The above condition above also happens if threads is really off or the processor does not have threads, so it's a state we need to handle in any case.

So just report threads=off, whenever cores == threads and treat it as such. If the user is unhappy then he should just turn off the cheat mode.   


> Since this is a 3.2 cluster feature, the engine will make sure to
> update
> the DB with the real numbers, and older clusters will use the
> information
> vdsm reports, while blocking the engine side optimization.
> An update path is already available here:
> http://gerrit.ovirt.org/#/c/10144/4
> 
> I hope that for cluster 3.3 we'll be able to drop the vdsm
> configuration.
> A review would be highly appreciated due to the holidays proximity.
> 
> Doron
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From dfediuck at redhat.com  Thu Dec 20 15:17:50 2012
From: dfediuck at redhat.com (Doron Fediuck)
Date: Thu, 20 Dec 2012 10:17:50 -0500 (EST)
Subject: [Engine-devel] [vdsm]  CPU Overcommit Feature
In-Reply-To: <9336026.968.1356015327538.JavaMail.javamailuser@localhost>
Message-ID: <508374473.56293343.1356016670851.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Simon Grinberg" <simon at redhat.com>
> To: "Doron Fediuck" <dfediuck at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, vdsm-devel at fedorahosted.org
> Sent: Thursday, December 20, 2012 4:56:14 PM
> Subject: Re: [Engine-devel] [vdsm]  CPU Overcommit Feature
> 
> 
> 
> ----- Original Message -----
> > From: "Doron Fediuck" <dfediuck at redhat.com>
> > To: "Dan Kenigsberg" <danken at redhat.com>
> > Cc: "engine-devel" <engine-devel at ovirt.org>,
> > vdsm-devel at fedorahosted.org
> > Sent: Thursday, December 20, 2012 2:14:45 PM
> > Subject: Re: [Engine-devel] [vdsm]  CPU Overcommit Feature
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Dan Kenigsberg" <danken at redhat.com>
> > > To: "Itamar Heim" <iheim at redhat.com>
> > > Cc: "engine-devel" <engine-devel at ovirt.org>,
> > > vdsm-devel at fedorahosted.org
> > > Sent: Thursday, December 20, 2012 11:55:10 AM
> > > Subject: Re: [Engine-devel] [vdsm]  CPU Overcommit Feature
> > > 
> > > On Thu, Dec 20, 2012 at 10:23:55AM +0200, Itamar Heim wrote:
> > > > On 12/20/2012 09:43 AM, Dan Kenigsberg wrote:
> > > > >On Wed, Dec 19, 2012 at 09:53:15AM -0500, Doron Fediuck wrote:
> > > > >>
> > > > >>----- Original Message -----
> > > > >>>From: "Dan Kenigsberg" <danken at redhat.com>
> > > > >>>To: "Greg Padgett" <gpadgett at redhat.com>
> > > > >>>Cc: "engine-devel" <engine-devel at ovirt.org>,
> > > > >>>vdsm-devel at fedorahosted.org
> > > > >>>Sent: Wednesday, December 19, 2012 3:59:11 PM
> > > > >>>Subject: Re: [Engine-devel] CPU Overcommit Feature
> > > > >>>
> > > > >>>On Mon, Dec 17, 2012 at 09:37:57AM -0500, Greg Padgett
> > > > >>>wrote:
> > > > >>>>Hi,
> > > > >>>>
> > > > >>>>I've been working on a feature to allow CPU Overcommitment
> > > > >>>>of
> > > > >>>>hosts
> > > > >>>>in a cluster.  This first stage allows the engine to
> > > > >>>>consider
> > > > >>>>host
> > > > >>>>cpu threads as cores for the purposes of VM resource
> > > > >>>>allocation.
> > > > >>>>
> > > > >>>>This wiki page has further details, your comments are
> > > > >>>>welcome!
> > > > >>>>http://www.ovirt.org/Features/cpu_overcommit
> > > > >>>
> > > > >>>I've commented about the vdsm/engine API on
> > > > >>>http://gerrit.ovirt.org/#/c/10144/ but it is probably better
> > > > >>>to
> > > > >>>reiterate it here.
> > > > >>>
> > > > >>>The suggested API is tightly coupled with an ugly hack we
> > > > >>>pushed
> > > > >>>to
> > > > >>>vdsm
> > > > >>>in order not to solve the issue properly on the first
> > > > >>>strike.
> > > > >>>
> > > > >>>If we had not have report_host_threads_as_cores, I think
> > > > >>>we'd
> > > > >>>have a
> > > > >>>simpler API reporting only cpuThreads and cpuCores; with no
> > > > >>>funny
> > > > >>>boolean flags.
> > > > >>>
> > > > >>>Let us strive to that position as much as we can.
> > > > >>>
> > > > >>>How about asking whoever used report_host_threads_as_cores
> > > > >>>to
> > > > >>>unset
> > > > >>>it
> > > > >>>once they install Engine 3.2 ? I think that these are very
> > > > >>>few
> > > > >>>people,
> > > > >>>that would not mind this very much.
> > > > >>>
> > > > >>>If this is impossible, I'd add a cpuCores2, always reporting
> > > > >>>the
> > > > >>>true
> > > > >>>number, to be used by new Engines. We may even report it
> > > > >>>only
> > > > >>>on
> > > > >>>the
> > > > >>>very few cases of report_host_threads_as_cores being set.
> > > > >>>
> > > > >>>Dan.
> > > > >>
> > > > >>Hi Dan,
> > > > >>Thanks for the review.
> > > > >>
> > > > >>I agree simply reporting cores and threads would be the right
> > > > >>solution.
> > > > >>However, when you have hyperthreading turned off you get
> > > > >>cores=threads.
> > > > >>This is the same situation you have when hyperthreading
> > > > >>turned
> > > > >>on, and
> > > > >>someone used the vdsm configuration of reporting threads as
> > > > >>cores.
> > > > >>
> > > > >>So the engine won't know the real status of the host.
> > > > >
> > > > >This is not surprising, as report_host_threads_as_cores means
> > > > >in
> > > > >blunt
> > > > >English "lie to Engine about the number of cores". The newly
> > > > >suggested
> > > > >flag says "don't believe what I said in cpuCores, since I'm
> > > > >lying". Next
> > > > >thing we'd have is another flag saying that the former flag
> > > > >was
> > > > >a
> > > > >lie,
> > > > >and cpuCores is actually trustworthy.
> > > > >
> > > > >Instead of dancing this dance, I suggest to stop lying.
> > > > >
> > > > >report_host_threads_as_cores was a hack to assist a older
> > > > >Engine
> > > > >versions. Engine users that needed it had to set it
> > > > >out-of-band
> > > > >on
> > > > >their
> > > > >hosts. Now if they upgrade their Engine, they can -- as easily
> > > > >--
> > > > >reset
> > > > >that value.
> > > > >
> > > > >If they forget, nothing devastating happens beyond Engine
> > > > >assuming
> > > > >that
> > > > >hyperthreading is off.
> > > > >
> > > > >Please consider this suggestion. I find it the simplest for
> > > > >all
> > > > >involved
> > > > >parties.
> > > > 
> > > > the only problem is the new vdsm doesn't know which engine may
> > > > be
> > > > using it.
> > > > if engine would say "getVdsCaps engineVersion=3.2", then vdsm
> > > > could
> > > > know engine no longer needs lying to and ignore the flag,
> > > > re-using
> > > > same field.
> > > 
> > > Note that I do not suggest to drop report_host_threads_as_cores
> > > now.
> > > I am suggesting to keep on lying even to new Engine.
> > > If someone thinks that lying is bad, he should reset
> > > report_host_threads_as_cores.
> > > 
> > > It seems to me that the suggested API is being coerced by a very
> > > limited
> > > use case, that is not going to be really harmed by a
> > > straight-forward
> > > API.
> > > 
> > > Dan.
> > 
> > Dan,
> > Did some further checking, and we can go with it;
> > So basically now we add cpuThreads. Additionally, if the
> > report_host_threads_as_cores
> > is turned on, an additional cpuCoresReal will be reported.
> 
> No need for that.
> There is only one problematic state where VDSM cheats and reports
> cores == threads.
> This does not happen by mistake, the user specifically asked for it.
> 
> The above condition above also happens if threads is really off or
> the processor does not have threads, so it's a state we need to
> handle in any case.
> 
> So just report threads=off, whenever cores == threads and treat it as
> such. If the user is unhappy then he should just turn off the cheat
> mode.
> 

ack.

* <3.2 clusters are not supported.
* >=3.2 clusters will assume HT off unless cores < threads.

Some release note is needed for this, so users will be advised to
turn off vdsm cheating when upgrading to new 3.2 engine, in case they
happen to use it.

> 
> > Since this is a 3.2 cluster feature, the engine will make sure to
> > update
> > the DB with the real numbers, and older clusters will use the
> > information
> > vdsm reports, while blocking the engine side optimization.
> > An update path is already available here:
> > http://gerrit.ovirt.org/#/c/10144/4
> > 
> > I hope that for cluster 3.3 we'll be able to drop the vdsm
> > configuration.
> > A review would be highly appreciated due to the holidays proximity.
> > 
> > Doron



From Christopher.Morrissey at netapp.com  Thu Dec 20 15:18:02 2012
From: Christopher.Morrissey at netapp.com (Morrissey, Christopher)
Date: Thu, 20 Dec 2012 15:18:02 +0000
Subject: [Engine-devel] FW: Querying for and registering unknown disk images
 on a storage domain
Message-ID: <D290AD8432118048947689BA3AE8A9B3094E3FDC@SACEXCMBX04-PRD.hq.netapp.com>

Hi All,

I've been working on a bit of functionality for the engine that will allow a user to query a domain for new disk images (GetUnregisteredImagesQuery) for which the engine was previously unaware and a separate command to register those images (ImportImageCommand). These commands will be exposed through the REST API.

This functionality is needed as we are developing an extension/plugin to oVirt that will allow a NetApp storage controller to handle cloning the actual disks outside of oVirt and need to import them once they are cloned. We'll be using other existing APIs to attach the disk to the necessary VM once the disk is cloned. On the NetApp side, we'll ensure the disk is coalesced before cloning so as to avoid the issues of registering snapshots.

GetUnregisteredImagesQuery will be accessible through the disks resource collection on a storage domain. A "disks" resource collection does not yet exist and will need to be added. To access the unregistered images, a parameter (maybe "unregistered=true") would be passed. So the path to "GET" the unregistered disk images on a domain would be something like /api/storagedomains/f0dbcb33-69d3-4899-9352-8e8a02f01bbd/disks?unregistered=true. This will return a list of disk images that can be each used as input to the ImportImageCommand to get them added to oVirt.

ImportImageCommand will be accessible through "POST"ing a disk to /api/disks?import=true. The disk will be added to the oVirt DB based on the information supplied and afterward would be available to attach to a VM.

When querying for unregistered disk images, the GetUnregisteredImagesQuery command will use the getImagesList() VDSM command. Currently this only reports the GUIDs of all disk images in a domain. I had been using the getVolumesList() and getVolumeInfo() VDSM commands to fill in the information so that valid disk image objects could be registered in oVirt. It seems these two functions are set to be removed since they are too invasive into the internal VDSM workings. The VDSM team will need to either return more information about each disk as part of the getImagesList() function or add a new function getImageInfo() that will give the same information for a given image GUID.

Note that much of this work had originally been submitted under patch http://gerrit.ovirt.org/#/c/9603/. After several reviews it was found to be lacking in its design and was using deprecated APIs that did not yet have replacements. I'm reworking the code now to conform to this design and asking for further input from the VDSM, core, and restapi teams to ensure we can get this done quickly and correctly as it is needed for the 3.2 release.

-Chris

Chris Morrissey
Software Engineer
NetApp Inc.
919.476.4428

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121220/9776c523/attachment.html>

From ovedo at redhat.com  Thu Dec 20 16:35:22 2012
From: ovedo at redhat.com (Oved Ourfalli)
Date: Thu, 20 Dec 2012 11:35:22 -0500 (EST)
Subject: [Engine-devel] oVirt-Foreman UI plugin
In-Reply-To: <1398031618.67136945.1356020092522.JavaMail.root@redhat.com>
Message-ID: <1319400620.67161934.1356021322831.JavaMail.root@redhat.com>

Hey all,

I've been working on an oVirt-Foreman UI-plugin recently, using oVirt's new UI-plugin infrastructure.
It allows users to see Foreman data on oVirt entities, from inside Webadmin.

I wrote a blog post describing this plugin, what can he do, some challenges that I faced and solutions, some screenshots, future work and more.

Check it out at:
ovedou.blogpost.com

I'll be happy to hear your feedback both on the blog and on the plugin itself!
If you need help installing the plugin feel free to contact me :-)

Cheers,
Oved


From iheim at redhat.com  Thu Dec 20 16:39:00 2012
From: iheim at redhat.com (Itamar Heim)
Date: Thu, 20 Dec 2012 18:39:00 +0200
Subject: [Engine-devel] Proposal to add Shireesh Anjal as maintainer	to
 engine-gluster
In-Reply-To: <1984198008.43138201.1354696441836.JavaMail.root@redhat.com>
References: <1984198008.43138201.1354696441836.JavaMail.root@redhat.com>
Message-ID: <50D33F24.1020801@redhat.com>

On 12/05/2012 10:34 AM, Yair Zaslavsky wrote:
>

Shireesh added as maintainer for engine-gluster.

Thanks,
    Itamar


From iheim at redhat.com  Thu Dec 20 17:34:28 2012
From: iheim at redhat.com (Itamar Heim)
Date: Thu, 20 Dec 2012 19:34:28 +0200
Subject: [Engine-devel] oVirt-Foreman UI plugin
In-Reply-To: <1319400620.67161934.1356021322831.JavaMail.root@redhat.com>
References: <1319400620.67161934.1356021322831.JavaMail.root@redhat.com>
Message-ID: <50D34C24.8030901@redhat.com>

On 12/20/2012 06:35 PM, Oved Ourfalli wrote:
> Hey all,
>
> I've been working on an oVirt-Foreman UI-plugin recently, using oVirt's new UI-plugin infrastructure.
> It allows users to see Foreman data on oVirt entities, from inside Webadmin.
>
> I wrote a blog post describing this plugin, what can he do, some challenges that I faced and solutions, some screenshots, future work and more.
>
> Check it out at:
> ovedou.blogpost.com

link is broken...
http://ovedou.blogspot.co.il/
http://ovedou.blogspot.co.il/2012/12/ovirt-foreman-ui-plugin.html


>
> I'll be happy to hear your feedback both on the blog and on the plugin itself!
> If you need help installing the plugin feel free to contact me :-)
>
> Cheers,
> Oved
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
>




From ovedo at redhat.com  Thu Dec 20 17:51:53 2012
From: ovedo at redhat.com (Oved Ourfalli)
Date: Thu, 20 Dec 2012 12:51:53 -0500 (EST)
Subject: [Engine-devel] [Users]  oVirt-Foreman UI plugin
In-Reply-To: <50D34C24.8030901@redhat.com>
Message-ID: <79664673.67255133.1356025913948.JavaMail.root@redhat.com>

All links below work for me, from several devices. Is it still broken for someone?

----- Itamar Heim <iheim at redhat.com> wrote:
> On 12/20/2012 06:35 PM, Oved Ourfalli wrote:
> > Hey all,
> >
> > I've been working on an oVirt-Foreman UI-plugin recently, using oVirt's new UI-plugin infrastructure.
> > It allows users to see Foreman data on oVirt entities, from inside Webadmin.
> >
> > I wrote a blog post describing this plugin, what can he do, some challenges that I faced and solutions, some screenshots, future work and more.
> >
> > Check it out at:
> > ovedou.blogpost.com
> 
> link is broken...
> http://ovedou.blogspot.co.il/
> http://ovedou.blogspot.co.il/2012/12/ovirt-foreman-ui-plugin.html
> 
> 
> >
> > I'll be happy to hear your feedback both on the blog and on the plugin itself!
> > If you need help installing the plugin feel free to contact me :-)
> >
> > Cheers,
> > Oved
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> >
> 
> 
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users



From Christopher.Morrissey at netapp.com  Thu Dec 20 18:00:21 2012
From: Christopher.Morrissey at netapp.com (Morrissey, Christopher)
Date: Thu, 20 Dec 2012 18:00:21 +0000
Subject: [Engine-devel] [Users]  oVirt-Foreman UI plugin
In-Reply-To: <79664673.67255133.1356025913948.JavaMail.root@redhat.com>
References: <50D34C24.8030901@redhat.com>
	<79664673.67255133.1356025913948.JavaMail.root@redhat.com>
Message-ID: <D290AD8432118048947689BA3AE8A9B3094E43B0@SACEXCMBX04-PRD.hq.netapp.com>

The issue was "blogpost.com" vs. "blogspot.com".

-Chris


> -----Original Message-----
> From: engine-devel-bounces at ovirt.org [mailto:engine-devel-
> bounces at ovirt.org] On Behalf Of Oved Ourfalli
> Sent: Thursday, December 20, 2012 12:52 PM
> To: Itamar Heim
> Cc: Ohad Levy; engine-devel; users at oVirt.org; Joseph Magen
> Subject: Re: [Engine-devel] [Users] oVirt-Foreman UI plugin
> 
> All links below work for me, from several devices. Is it still broken for
> someone?
> 
> ----- Itamar Heim <iheim at redhat.com> wrote:
> > On 12/20/2012 06:35 PM, Oved Ourfalli wrote:
> > > Hey all,
> > >
> > > I've been working on an oVirt-Foreman UI-plugin recently, using oVirt's
> new UI-plugin infrastructure.
> > > It allows users to see Foreman data on oVirt entities, from inside
> Webadmin.
> > >
> > > I wrote a blog post describing this plugin, what can he do, some
> challenges that I faced and solutions, some screenshots, future work and
> more.
> > >
> > > Check it out at:
> > > ovedou.blogpost.com
> >
> > link is broken...
> > http://ovedou.blogspot.co.il/
> > http://ovedou.blogspot.co.il/2012/12/ovirt-foreman-ui-plugin.html
> >
> >
> > >
> > > I'll be happy to hear your feedback both on the blog and on the plugin
> itself!
> > > If you need help installing the plugin feel free to contact me :-)
> > >
> > > Cheers,
> > > Oved
> > > _______________________________________________
> > > Engine-devel mailing list
> > > Engine-devel at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > >
> >
> >
> > _______________________________________________
> > Users mailing list
> > Users at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> 
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel


From ovedo at redhat.com  Thu Dec 20 18:05:22 2012
From: ovedo at redhat.com (Oved Ourfalli)
Date: Thu, 20 Dec 2012 13:05:22 -0500 (EST)
Subject: [Engine-devel] [Users] oVirt-Foreman UI plugin
In-Reply-To: <D290AD8432118048947689BA3AE8A9B3094E43B0@SACEXCMBX04-PRD.hq.netapp.com>
Message-ID: <871608998.67264142.1356026722928.JavaMail.root@redhat.com>


----- Christopher Morrissey <Christopher.Morrissey at netapp.com> wrote:
> The issue was "blogpost.com" vs. "blogspot.com".
> 
Thank you for pointing it out.
So the correct link is:
http://ovedou.blogspot.com
and there you will see the new post.

(both links Itamar wrote below will work as well).

Sorry for the inconvenience,
Oved

> -Chris
> 
> 
> > -----Original Message-----
> > From: engine-devel-bounces at ovirt.org [mailto:engine-devel-
> > bounces at ovirt.org] On Behalf Of Oved Ourfalli
> > Sent: Thursday, December 20, 2012 12:52 PM
> > To: Itamar Heim
> > Cc: Ohad Levy; engine-devel; users at oVirt.org; Joseph Magen
> > Subject: Re: [Engine-devel] [Users] oVirt-Foreman UI plugin
> > 
> > All links below work for me, from several devices. Is it still broken for
> > someone?
> > 
> > ----- Itamar Heim <iheim at redhat.com> wrote:
> > > On 12/20/2012 06:35 PM, Oved Ourfalli wrote:
> > > > Hey all,
> > > >
> > > > I've been working on an oVirt-Foreman UI-plugin recently, using oVirt's
> > new UI-plugin infrastructure.
> > > > It allows users to see Foreman data on oVirt entities, from inside
> > Webadmin.
> > > >
> > > > I wrote a blog post describing this plugin, what can he do, some
> > challenges that I faced and solutions, some screenshots, future work and
> > more.
> > > >
> > > > Check it out at:
> > > > ovedou.blogpost.com
> > >
> > > link is broken...
> > > http://ovedou.blogspot.co.il/
> > > http://ovedou.blogspot.co.il/2012/12/ovirt-foreman-ui-plugin.html
> > >
> > >
> > > >
> > > > I'll be happy to hear your feedback both on the blog and on the plugin
> > itself!
> > > > If you need help installing the plugin feel free to contact me :-)
> > > >
> > > > Cheers,
> > > > Oved
> > > > _______________________________________________
> > > > Engine-devel mailing list
> > > > Engine-devel at ovirt.org
> > > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > >
> > >
> > >
> > > _______________________________________________
> > > Users mailing list
> > > Users at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/users
> > 
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel



From danken at redhat.com  Thu Dec 20 19:26:49 2012
From: danken at redhat.com (Dan Kenigsberg)
Date: Thu, 20 Dec 2012 21:26:49 +0200
Subject: [Engine-devel] [vdsm]  CPU Overcommit Feature
In-Reply-To: <508374473.56293343.1356016670851.JavaMail.root@redhat.com>
References: <9336026.968.1356015327538.JavaMail.javamailuser@localhost>
	<508374473.56293343.1356016670851.JavaMail.root@redhat.com>
Message-ID: <20121220192649.GA16556@redhat.com>

On Thu, Dec 20, 2012 at 10:17:50AM -0500, Doron Fediuck wrote:
> 
> 
> ----- Original Message -----
> > From: "Simon Grinberg" <simon at redhat.com>
> > To: "Doron Fediuck" <dfediuck at redhat.com>
> > Cc: "engine-devel" <engine-devel at ovirt.org>, vdsm-devel at fedorahosted.org
> > Sent: Thursday, December 20, 2012 4:56:14 PM
> > Subject: Re: [Engine-devel] [vdsm]  CPU Overcommit Feature
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Doron Fediuck" <dfediuck at redhat.com>
> > > To: "Dan Kenigsberg" <danken at redhat.com>
> > > Cc: "engine-devel" <engine-devel at ovirt.org>,
> > > vdsm-devel at fedorahosted.org
> > > Sent: Thursday, December 20, 2012 2:14:45 PM
> > > Subject: Re: [Engine-devel] [vdsm]  CPU Overcommit Feature
> > > 
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Dan Kenigsberg" <danken at redhat.com>
> > > > To: "Itamar Heim" <iheim at redhat.com>
> > > > Cc: "engine-devel" <engine-devel at ovirt.org>,
> > > > vdsm-devel at fedorahosted.org
> > > > Sent: Thursday, December 20, 2012 11:55:10 AM
> > > > Subject: Re: [Engine-devel] [vdsm]  CPU Overcommit Feature
> > > > 
> > > > On Thu, Dec 20, 2012 at 10:23:55AM +0200, Itamar Heim wrote:
> > > > > On 12/20/2012 09:43 AM, Dan Kenigsberg wrote:
> > > > > >On Wed, Dec 19, 2012 at 09:53:15AM -0500, Doron Fediuck wrote:
> > > > > >>
> > > > > >>----- Original Message -----
> > > > > >>>From: "Dan Kenigsberg" <danken at redhat.com>
> > > > > >>>To: "Greg Padgett" <gpadgett at redhat.com>
> > > > > >>>Cc: "engine-devel" <engine-devel at ovirt.org>,
> > > > > >>>vdsm-devel at fedorahosted.org
> > > > > >>>Sent: Wednesday, December 19, 2012 3:59:11 PM
> > > > > >>>Subject: Re: [Engine-devel] CPU Overcommit Feature
> > > > > >>>
> > > > > >>>On Mon, Dec 17, 2012 at 09:37:57AM -0500, Greg Padgett
> > > > > >>>wrote:
> > > > > >>>>Hi,
> > > > > >>>>
> > > > > >>>>I've been working on a feature to allow CPU Overcommitment
> > > > > >>>>of
> > > > > >>>>hosts
> > > > > >>>>in a cluster.  This first stage allows the engine to
> > > > > >>>>consider
> > > > > >>>>host
> > > > > >>>>cpu threads as cores for the purposes of VM resource
> > > > > >>>>allocation.
> > > > > >>>>
> > > > > >>>>This wiki page has further details, your comments are
> > > > > >>>>welcome!
> > > > > >>>>http://www.ovirt.org/Features/cpu_overcommit
> > > > > >>>
> > > > > >>>I've commented about the vdsm/engine API on
> > > > > >>>http://gerrit.ovirt.org/#/c/10144/ but it is probably better
> > > > > >>>to
> > > > > >>>reiterate it here.
> > > > > >>>
> > > > > >>>The suggested API is tightly coupled with an ugly hack we
> > > > > >>>pushed
> > > > > >>>to
> > > > > >>>vdsm
> > > > > >>>in order not to solve the issue properly on the first
> > > > > >>>strike.
> > > > > >>>
> > > > > >>>If we had not have report_host_threads_as_cores, I think
> > > > > >>>we'd
> > > > > >>>have a
> > > > > >>>simpler API reporting only cpuThreads and cpuCores; with no
> > > > > >>>funny
> > > > > >>>boolean flags.
> > > > > >>>
> > > > > >>>Let us strive to that position as much as we can.
> > > > > >>>
> > > > > >>>How about asking whoever used report_host_threads_as_cores
> > > > > >>>to
> > > > > >>>unset
> > > > > >>>it
> > > > > >>>once they install Engine 3.2 ? I think that these are very
> > > > > >>>few
> > > > > >>>people,
> > > > > >>>that would not mind this very much.
> > > > > >>>
> > > > > >>>If this is impossible, I'd add a cpuCores2, always reporting
> > > > > >>>the
> > > > > >>>true
> > > > > >>>number, to be used by new Engines. We may even report it
> > > > > >>>only
> > > > > >>>on
> > > > > >>>the
> > > > > >>>very few cases of report_host_threads_as_cores being set.
> > > > > >>>
> > > > > >>>Dan.
> > > > > >>
> > > > > >>Hi Dan,
> > > > > >>Thanks for the review.
> > > > > >>
> > > > > >>I agree simply reporting cores and threads would be the right
> > > > > >>solution.
> > > > > >>However, when you have hyperthreading turned off you get
> > > > > >>cores=threads.
> > > > > >>This is the same situation you have when hyperthreading
> > > > > >>turned
> > > > > >>on, and
> > > > > >>someone used the vdsm configuration of reporting threads as
> > > > > >>cores.
> > > > > >>
> > > > > >>So the engine won't know the real status of the host.
> > > > > >
> > > > > >This is not surprising, as report_host_threads_as_cores means
> > > > > >in
> > > > > >blunt
> > > > > >English "lie to Engine about the number of cores". The newly
> > > > > >suggested
> > > > > >flag says "don't believe what I said in cpuCores, since I'm
> > > > > >lying". Next
> > > > > >thing we'd have is another flag saying that the former flag
> > > > > >was
> > > > > >a
> > > > > >lie,
> > > > > >and cpuCores is actually trustworthy.
> > > > > >
> > > > > >Instead of dancing this dance, I suggest to stop lying.
> > > > > >
> > > > > >report_host_threads_as_cores was a hack to assist a older
> > > > > >Engine
> > > > > >versions. Engine users that needed it had to set it
> > > > > >out-of-band
> > > > > >on
> > > > > >their
> > > > > >hosts. Now if they upgrade their Engine, they can -- as easily
> > > > > >--
> > > > > >reset
> > > > > >that value.
> > > > > >
> > > > > >If they forget, nothing devastating happens beyond Engine
> > > > > >assuming
> > > > > >that
> > > > > >hyperthreading is off.
> > > > > >
> > > > > >Please consider this suggestion. I find it the simplest for
> > > > > >all
> > > > > >involved
> > > > > >parties.
> > > > > 
> > > > > the only problem is the new vdsm doesn't know which engine may
> > > > > be
> > > > > using it.
> > > > > if engine would say "getVdsCaps engineVersion=3.2", then vdsm
> > > > > could
> > > > > know engine no longer needs lying to and ignore the flag,
> > > > > re-using
> > > > > same field.
> > > > 
> > > > Note that I do not suggest to drop report_host_threads_as_cores
> > > > now.
> > > > I am suggesting to keep on lying even to new Engine.
> > > > If someone thinks that lying is bad, he should reset
> > > > report_host_threads_as_cores.
> > > > 
> > > > It seems to me that the suggested API is being coerced by a very
> > > > limited
> > > > use case, that is not going to be really harmed by a
> > > > straight-forward
> > > > API.
> > > > 
> > > > Dan.
> > > 
> > > Dan,
> > > Did some further checking, and we can go with it;
> > > So basically now we add cpuThreads. Additionally, if the
> > > report_host_threads_as_cores
> > > is turned on, an additional cpuCoresReal will be reported.
> > 
> > No need for that.
> > There is only one problematic state where VDSM cheats and reports
> > cores == threads.
> > This does not happen by mistake, the user specifically asked for it.
> > 
> > The above condition above also happens if threads is really off or
> > the processor does not have threads, so it's a state we need to
> > handle in any case.
> > 
> > So just report threads=off, whenever cores == threads and treat it as
> > such. If the user is unhappy then he should just turn off the cheat
> > mode.
> > 
> 
> ack.
> 
> * <3.2 clusters are not supported.
> * >=3.2 clusters will assume HT off unless cores < threads.
> 
> Some release note is needed for this, so users will be advised to
> turn off vdsm cheating when upgrading to new 3.2 engine, in case they
> happen to use it.

Plain and simple. vdsm side accepted to master branch.
Please backport it to the ovirt-3.2 branch in order to get into the
release.


From iheim at redhat.com  Thu Dec 20 21:42:37 2012
From: iheim at redhat.com (Itamar Heim)
Date: Thu, 20 Dec 2012 23:42:37 +0200
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <50D1BC5A.9030402@redhat.com>
References: <50D1BC5A.9030402@redhat.com>
Message-ID: <50D3864D.6030501@redhat.com>

On 12/19/2012 03:08 PM, Michael Pasternak wrote:
>
> Hi All,
>
> In many cases OSS maintainers not always can be in the loop of different threads what may
> cause them missing important decisions being taken,
>
> As result later on during reviews of the patches they're not accepting (already implemented)
> features, what is causing not once for feature to be re-designed and/or delayed, what is wrong
> from the development cycle PoV.
>
> Therefore I'd like to suggest establishing dev-rules for the new feature implementation,
> what will make entire process much more easer for all of us:
>
> 1. discuss new feature on the mailing list (requirements/constraints/etc.)
> 2. summarise feature details in feature-doc
> 3. send feature-doc to review to:
>     3.1 ML (engine-devel)
>     3.2 MG (mailing group of maintainers of the relevant layers)
> 4. after feature-doc is accepted,
>     4.1 implement the feature
>     4.2 send it to gerrit for review to:
>         4.2.1 lead maintainer/s (they will review/delegate it)
>
>
> NOTE 3.2, 4.2.1 will require defining MGs such as:
>        - engine-devel-core
>        - engine-devel-ui
>        - engine-devel-api
>        - engine-devel-sdk
>        - engine-devel-cli
>        - engine-devel-vdsm ...
>
>
> Thoughts?
>

i thought this is what we have the arch mailing list for, since any 
feature is going to cut through multiple layers/components, unless they 
are very specific, they should be sent to arch, and all maintainers 
should follow arch.


From alonbl at redhat.com  Thu Dec 20 22:12:03 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Thu, 20 Dec 2012 17:12:03 -0500 (EST)
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <50D3864D.6030501@redhat.com>
Message-ID: <1488374349.6339258.1356041523739.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Michael Pasternak" <mpastern at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>
> Sent: Thursday, December 20, 2012 11:42:37 PM
> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
> 
> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
> >
> > Hi All,
> >
> > In many cases OSS maintainers not always can be in the loop of
> > different threads what may
> > cause them missing important decisions being taken,
> >
> > As result later on during reviews of the patches they're not
> > accepting (already implemented)
> > features, what is causing not once for feature to be re-designed
> > and/or delayed, what is wrong
> > from the development cycle PoV.
> >
> > Therefore I'd like to suggest establishing dev-rules for the new
> > feature implementation,
> > what will make entire process much more easer for all of us:
> >
> > 1. discuss new feature on the mailing list
> > (requirements/constraints/etc.)
> > 2. summarise feature details in feature-doc
> > 3. send feature-doc to review to:
> >     3.1 ML (engine-devel)
> >     3.2 MG (mailing group of maintainers of the relevant layers)
> > 4. after feature-doc is accepted,
> >     4.1 implement the feature
> >     4.2 send it to gerrit for review to:
> >         4.2.1 lead maintainer/s (they will review/delegate it)
> >
> >
> > NOTE 3.2, 4.2.1 will require defining MGs such as:
> >        - engine-devel-core
> >        - engine-devel-ui
> >        - engine-devel-api
> >        - engine-devel-sdk
> >        - engine-devel-cli
> >        - engine-devel-vdsm ...
> >
> >
> > Thoughts?
> >
> 
> i thought this is what we have the arch mailing list for, since any
> feature is going to cut through multiple layers/components, unless
> they
> are very specific, they should be sent to arch, and all maintainers
> should follow arch.

What is missing is upstream bugzilla.

A feature, after initiate stage should be represented with a bug.
The bug should be assigned to the right designated milestone.
All document references (including versions) should be attached or referred by the bug.
Dependency between features can be established using bug dependencies.
Status can be acquired from buzilla at any time, progress reports should be input into bugzilla.
Contact details for the feature can be acquired too, relevant and interested parties can be CCed explicitly.
I guess I can add more benefits.

Mailing list is good for idea initiation, but not for lifecycle management, nor for people to join at implementation phase and understand why, how and when.

Having upstream bugzilla will also help us plan ahead, and manage the break the project into smaller components, to assign core developers for each.

Regards,
Alon


From iheim at redhat.com  Thu Dec 20 22:15:13 2012
From: iheim at redhat.com (Itamar Heim)
Date: Fri, 21 Dec 2012 00:15:13 +0200
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <1488374349.6339258.1356041523739.JavaMail.root@redhat.com>
References: <1488374349.6339258.1356041523739.JavaMail.root@redhat.com>
Message-ID: <50D38DF1.7050809@redhat.com>

On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
>
>
> ----- Original Message -----
>> From: "Itamar Heim" <iheim at redhat.com>
>> To: "Michael Pasternak" <mpastern at redhat.com>
>> Cc: "engine-devel" <engine-devel at ovirt.org>
>> Sent: Thursday, December 20, 2012 11:42:37 PM
>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
>>
>> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
>>>
>>> Hi All,
>>>
>>> In many cases OSS maintainers not always can be in the loop of
>>> different threads what may
>>> cause them missing important decisions being taken,
>>>
>>> As result later on during reviews of the patches they're not
>>> accepting (already implemented)
>>> features, what is causing not once for feature to be re-designed
>>> and/or delayed, what is wrong
>>> from the development cycle PoV.
>>>
>>> Therefore I'd like to suggest establishing dev-rules for the new
>>> feature implementation,
>>> what will make entire process much more easer for all of us:
>>>
>>> 1. discuss new feature on the mailing list
>>> (requirements/constraints/etc.)
>>> 2. summarise feature details in feature-doc
>>> 3. send feature-doc to review to:
>>>      3.1 ML (engine-devel)
>>>      3.2 MG (mailing group of maintainers of the relevant layers)
>>> 4. after feature-doc is accepted,
>>>      4.1 implement the feature
>>>      4.2 send it to gerrit for review to:
>>>          4.2.1 lead maintainer/s (they will review/delegate it)
>>>
>>>
>>> NOTE 3.2, 4.2.1 will require defining MGs such as:
>>>         - engine-devel-core
>>>         - engine-devel-ui
>>>         - engine-devel-api
>>>         - engine-devel-sdk
>>>         - engine-devel-cli
>>>         - engine-devel-vdsm ...
>>>
>>>
>>> Thoughts?
>>>
>>
>> i thought this is what we have the arch mailing list for, since any
>> feature is going to cut through multiple layers/components, unless
>> they
>> are very specific, they should be sent to arch, and all maintainers
>> should follow arch.
>
> What is missing is upstream bugzilla.
>
> A feature, after initiate stage should be represented with a bug.
> The bug should be assigned to the right designated milestone.
> All document references (including versions) should be attached or referred by the bug.
> Dependency between features can be established using bug dependencies.
> Status can be acquired from buzilla at any time, progress reports should be input into bugzilla.
> Contact details for the feature can be acquired too, relevant and interested parties can be CCed explicitly.
> I guess I can add more benefits.
>
> Mailing list is good for idea initiation, but not for lifecycle management, nor for people to join at implementation phase and understand why, how and when.
>
> Having upstream bugzilla will also help us plan ahead, and manage the break the project into smaller components, to assign core developers for each.

it still won't help tracking all relevant maintainers from rest api, ui, 
engine, vdsm saw the bug. we have multiple components.
arch is mostly to cover cross component issues, such as features.
(or we could create a feature mailing list).

but i do agree we could use bugzilla for tracking features per version, 
if wiki isn't tracking this correctly.


From alonbl at redhat.com  Thu Dec 20 22:18:22 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Thu, 20 Dec 2012 17:18:22 -0500 (EST)
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <50D38DF1.7050809@redhat.com>
Message-ID: <2017107161.6340319.1356041902365.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, "Michael Pasternak" <mpastern at redhat.com>
> Sent: Friday, December 21, 2012 12:15:13 AM
> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
> 
> On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Itamar Heim" <iheim at redhat.com>
> >> To: "Michael Pasternak" <mpastern at redhat.com>
> >> Cc: "engine-devel" <engine-devel at ovirt.org>
> >> Sent: Thursday, December 20, 2012 11:42:37 PM
> >> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
> >> the new feature discussion/implementation.
> >>
> >> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
> >>>
> >>> Hi All,
> >>>
> >>> In many cases OSS maintainers not always can be in the loop of
> >>> different threads what may
> >>> cause them missing important decisions being taken,
> >>>
> >>> As result later on during reviews of the patches they're not
> >>> accepting (already implemented)
> >>> features, what is causing not once for feature to be re-designed
> >>> and/or delayed, what is wrong
> >>> from the development cycle PoV.
> >>>
> >>> Therefore I'd like to suggest establishing dev-rules for the new
> >>> feature implementation,
> >>> what will make entire process much more easer for all of us:
> >>>
> >>> 1. discuss new feature on the mailing list
> >>> (requirements/constraints/etc.)
> >>> 2. summarise feature details in feature-doc
> >>> 3. send feature-doc to review to:
> >>>      3.1 ML (engine-devel)
> >>>      3.2 MG (mailing group of maintainers of the relevant layers)
> >>> 4. after feature-doc is accepted,
> >>>      4.1 implement the feature
> >>>      4.2 send it to gerrit for review to:
> >>>          4.2.1 lead maintainer/s (they will review/delegate it)
> >>>
> >>>
> >>> NOTE 3.2, 4.2.1 will require defining MGs such as:
> >>>         - engine-devel-core
> >>>         - engine-devel-ui
> >>>         - engine-devel-api
> >>>         - engine-devel-sdk
> >>>         - engine-devel-cli
> >>>         - engine-devel-vdsm ...
> >>>
> >>>
> >>> Thoughts?
> >>>
> >>
> >> i thought this is what we have the arch mailing list for, since
> >> any
> >> feature is going to cut through multiple layers/components, unless
> >> they
> >> are very specific, they should be sent to arch, and all
> >> maintainers
> >> should follow arch.
> >
> > What is missing is upstream bugzilla.
> >
> > A feature, after initiate stage should be represented with a bug.
> > The bug should be assigned to the right designated milestone.
> > All document references (including versions) should be attached or
> > referred by the bug.
> > Dependency between features can be established using bug
> > dependencies.
> > Status can be acquired from buzilla at any time, progress reports
> > should be input into bugzilla.
> > Contact details for the feature can be acquired too, relevant and
> > interested parties can be CCed explicitly.
> > I guess I can add more benefits.
> >
> > Mailing list is good for idea initiation, but not for lifecycle
> > management, nor for people to join at implementation phase and
> > understand why, how and when.
> >
> > Having upstream bugzilla will also help us plan ahead, and manage
> > the break the project into smaller components, to assign core
> > developers for each.
> 
> it still won't help tracking all relevant maintainers from rest api,
> ui,
> engine, vdsm saw the bug. we have multiple components.
> arch is mostly to cover cross component issues, such as features.
> (or we could create a feature mailing list).

Yes it is... for project width feature you have parent feature of the whole distribution and blockers for each component.

> but i do agree we could use bugzilla for tracking features per
> version,
> if wiki isn't tracking this correctly.

wiki is not the right tool for release engineering.

Regards,
Alon


From mpastern at redhat.com  Fri Dec 21 10:44:51 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Fri, 21 Dec 2012 12:44:51 +0200
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <50D38DF1.7050809@redhat.com>
References: <1488374349.6339258.1356041523739.JavaMail.root@redhat.com>
	<50D38DF1.7050809@redhat.com>
Message-ID: <50D43DA3.2010300@redhat.com>

On 12/21/2012 12:15 AM, Itamar Heim wrote:
> On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
>>
>>
>> ----- Original Message -----
>>> From: "Itamar Heim" <iheim at redhat.com>
>>> To: "Michael Pasternak" <mpastern at redhat.com>
>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>> Sent: Thursday, December 20, 2012 11:42:37 PM
>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
>>>
>>> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
>>>>
>>>> Hi All,
>>>>
>>>> In many cases OSS maintainers not always can be in the loop of
>>>> different threads what may
>>>> cause them missing important decisions being taken,
>>>>
>>>> As result later on during reviews of the patches they're not
>>>> accepting (already implemented)
>>>> features, what is causing not once for feature to be re-designed
>>>> and/or delayed, what is wrong
>>>> from the development cycle PoV.
>>>>
>>>> Therefore I'd like to suggest establishing dev-rules for the new
>>>> feature implementation,
>>>> what will make entire process much more easer for all of us:
>>>>
>>>> 1. discuss new feature on the mailing list
>>>> (requirements/constraints/etc.)
>>>> 2. summarise feature details in feature-doc
>>>> 3. send feature-doc to review to:
>>>>      3.1 ML (engine-devel)
>>>>      3.2 MG (mailing group of maintainers of the relevant layers)
>>>> 4. after feature-doc is accepted,
>>>>      4.1 implement the feature
>>>>      4.2 send it to gerrit for review to:
>>>>          4.2.1 lead maintainer/s (they will review/delegate it)
>>>>
>>>>
>>>> NOTE 3.2, 4.2.1 will require defining MGs such as:
>>>>         - engine-devel-core
>>>>         - engine-devel-ui
>>>>         - engine-devel-api
>>>>         - engine-devel-sdk
>>>>         - engine-devel-cli
>>>>         - engine-devel-vdsm ...
>>>>
>>>>
>>>> Thoughts?
>>>>
>>>
>>> i thought this is what we have the arch mailing list for, since any
>>> feature is going to cut through multiple layers/components, unless
>>> they
>>> are very specific, they should be sent to arch, and all maintainers
>>> should follow arch.
>>
>> What is missing is upstream bugzilla.
>>
>> A feature, after initiate stage should be represented with a bug.
>> The bug should be assigned to the right designated milestone.
>> All document references (including versions) should be attached or referred by the bug.
>> Dependency between features can be established using bug dependencies.
>> Status can be acquired from buzilla at any time, progress reports should be input into bugzilla.
>> Contact details for the feature can be acquired too, relevant and interested parties can be CCed explicitly.
>> I guess I can add more benefits.
>>
>> Mailing list is good for idea initiation, but not for lifecycle management, nor for people to join at implementation phase and understand why, how and when.
>>
>> Having upstream bugzilla will also help us plan ahead, and manage the break the project into smaller components, to assign core developers for each.
> 
> it still won't help tracking all relevant maintainers from rest api, ui, engine, vdsm saw the bug. we have multiple components.
> arch is mostly to cover cross component issues, such as features.

this is why i'm suggesting creating mailing-groups peer component,
feature owner knows what layers are involved and should be responsible to CC
relevant MGs in both BZ & thread.

> (or we could create a feature mailing list).

ML/MG peer feature is too much i think.

> 
> but i do agree we could use bugzilla for tracking features per version, if wiki isn't tracking this correctly.

+1


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From alonbl at redhat.com  Fri Dec 21 11:57:05 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Fri, 21 Dec 2012 06:57:05 -0500 (EST)
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <50D43DA3.2010300@redhat.com>
Message-ID: <652889674.6421746.1356091025982.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Michael Pasternak" <mpastern at redhat.com>
> To: "Itamar Heim" <iheim at redhat.com>
> Cc: "Alon Bar-Lev" <alonbl at redhat.com>, "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org
> Sent: Friday, December 21, 2012 12:44:51 PM
> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
> 
> On 12/21/2012 12:15 AM, Itamar Heim wrote:
> > On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
> >>
> >>
> >> ----- Original Message -----
> >>> From: "Itamar Heim" <iheim at redhat.com>
> >>> To: "Michael Pasternak" <mpastern at redhat.com>
> >>> Cc: "engine-devel" <engine-devel at ovirt.org>
> >>> Sent: Thursday, December 20, 2012 11:42:37 PM
> >>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
> >>> the new feature discussion/implementation.
> >>>
> >>> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
> >>>>
> >>>> Hi All,
> >>>>
> >>>> In many cases OSS maintainers not always can be in the loop of
> >>>> different threads what may
> >>>> cause them missing important decisions being taken,
> >>>>
> >>>> As result later on during reviews of the patches they're not
> >>>> accepting (already implemented)
> >>>> features, what is causing not once for feature to be re-designed
> >>>> and/or delayed, what is wrong
> >>>> from the development cycle PoV.
> >>>>
> >>>> Therefore I'd like to suggest establishing dev-rules for the new
> >>>> feature implementation,
> >>>> what will make entire process much more easer for all of us:
> >>>>
> >>>> 1. discuss new feature on the mailing list
> >>>> (requirements/constraints/etc.)
> >>>> 2. summarise feature details in feature-doc
> >>>> 3. send feature-doc to review to:
> >>>>      3.1 ML (engine-devel)
> >>>>      3.2 MG (mailing group of maintainers of the relevant
> >>>>      layers)
> >>>> 4. after feature-doc is accepted,
> >>>>      4.1 implement the feature
> >>>>      4.2 send it to gerrit for review to:
> >>>>          4.2.1 lead maintainer/s (they will review/delegate it)
> >>>>
> >>>>
> >>>> NOTE 3.2, 4.2.1 will require defining MGs such as:
> >>>>         - engine-devel-core
> >>>>         - engine-devel-ui
> >>>>         - engine-devel-api
> >>>>         - engine-devel-sdk
> >>>>         - engine-devel-cli
> >>>>         - engine-devel-vdsm ...
> >>>>
> >>>>
> >>>> Thoughts?
> >>>>
> >>>
> >>> i thought this is what we have the arch mailing list for, since
> >>> any
> >>> feature is going to cut through multiple layers/components,
> >>> unless
> >>> they
> >>> are very specific, they should be sent to arch, and all
> >>> maintainers
> >>> should follow arch.
> >>
> >> What is missing is upstream bugzilla.
> >>
> >> A feature, after initiate stage should be represented with a bug.
> >> The bug should be assigned to the right designated milestone.
> >> All document references (including versions) should be attached or
> >> referred by the bug.
> >> Dependency between features can be established using bug
> >> dependencies.
> >> Status can be acquired from buzilla at any time, progress reports
> >> should be input into bugzilla.
> >> Contact details for the feature can be acquired too, relevant and
> >> interested parties can be CCed explicitly.
> >> I guess I can add more benefits.
> >>
> >> Mailing list is good for idea initiation, but not for lifecycle
> >> management, nor for people to join at implementation phase and
> >> understand why, how and when.
> >>
> >> Having upstream bugzilla will also help us plan ahead, and manage
> >> the break the project into smaller components, to assign core
> >> developers for each.
> > 
> > it still won't help tracking all relevant maintainers from rest
> > api, ui, engine, vdsm saw the bug. we have multiple components.
> > arch is mostly to cover cross component issues, such as features.
> 
> this is why i'm suggesting creating mailing-groups peer component,
> feature owner knows what layers are involved and should be
> responsible to CC
> relevant MGs in both BZ & thread.

mailing group is something nobody can manage.
you can consider a group as all people who are CC to a bug.

> 
> > (or we could create a feature mailing list).
> 
> ML/MG peer feature is too much i think.
> 
> > 
> > but i do agree we could use bugzilla for tracking features per
> > version, if wiki isn't tracking this correctly.
> 
> +1
> 
> 
> --
> 
> Michael Pasternak
> RedHat, ENG-Virtualization R&D
> 


From mpastern at redhat.com  Sun Dec 23 06:25:23 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Sun, 23 Dec 2012 08:25:23 +0200
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <652889674.6421746.1356091025982.JavaMail.root@redhat.com>
References: <652889674.6421746.1356091025982.JavaMail.root@redhat.com>
Message-ID: <50D6A3D3.7010701@redhat.com>

On 12/21/2012 01:57 PM, Alon Bar-Lev wrote:
> 
> 
> ----- Original Message -----
>> From: "Michael Pasternak" <mpastern at redhat.com>
>> To: "Itamar Heim" <iheim at redhat.com>
>> Cc: "Alon Bar-Lev" <alonbl at redhat.com>, "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org
>> Sent: Friday, December 21, 2012 12:44:51 PM
>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
>>
>> On 12/21/2012 12:15 AM, Itamar Heim wrote:
>>> On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Itamar Heim" <iheim at redhat.com>
>>>>> To: "Michael Pasternak" <mpastern at redhat.com>
>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>>> Sent: Thursday, December 20, 2012 11:42:37 PM
>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
>>>>> the new feature discussion/implementation.
>>>>>
>>>>> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> In many cases OSS maintainers not always can be in the loop of
>>>>>> different threads what may
>>>>>> cause them missing important decisions being taken,
>>>>>>
>>>>>> As result later on during reviews of the patches they're not
>>>>>> accepting (already implemented)
>>>>>> features, what is causing not once for feature to be re-designed
>>>>>> and/or delayed, what is wrong
>>>>>> from the development cycle PoV.
>>>>>>
>>>>>> Therefore I'd like to suggest establishing dev-rules for the new
>>>>>> feature implementation,
>>>>>> what will make entire process much more easer for all of us:
>>>>>>
>>>>>> 1. discuss new feature on the mailing list
>>>>>> (requirements/constraints/etc.)
>>>>>> 2. summarise feature details in feature-doc
>>>>>> 3. send feature-doc to review to:
>>>>>>      3.1 ML (engine-devel)
>>>>>>      3.2 MG (mailing group of maintainers of the relevant
>>>>>>      layers)
>>>>>> 4. after feature-doc is accepted,
>>>>>>      4.1 implement the feature
>>>>>>      4.2 send it to gerrit for review to:
>>>>>>          4.2.1 lead maintainer/s (they will review/delegate it)
>>>>>>
>>>>>>
>>>>>> NOTE 3.2, 4.2.1 will require defining MGs such as:
>>>>>>         - engine-devel-core
>>>>>>         - engine-devel-ui
>>>>>>         - engine-devel-api
>>>>>>         - engine-devel-sdk
>>>>>>         - engine-devel-cli
>>>>>>         - engine-devel-vdsm ...
>>>>>>
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>
>>>>> i thought this is what we have the arch mailing list for, since
>>>>> any
>>>>> feature is going to cut through multiple layers/components,
>>>>> unless
>>>>> they
>>>>> are very specific, they should be sent to arch, and all
>>>>> maintainers
>>>>> should follow arch.
>>>>
>>>> What is missing is upstream bugzilla.
>>>>
>>>> A feature, after initiate stage should be represented with a bug.
>>>> The bug should be assigned to the right designated milestone.
>>>> All document references (including versions) should be attached or
>>>> referred by the bug.
>>>> Dependency between features can be established using bug
>>>> dependencies.
>>>> Status can be acquired from buzilla at any time, progress reports
>>>> should be input into bugzilla.
>>>> Contact details for the feature can be acquired too, relevant and
>>>> interested parties can be CCed explicitly.
>>>> I guess I can add more benefits.
>>>>
>>>> Mailing list is good for idea initiation, but not for lifecycle
>>>> management, nor for people to join at implementation phase and
>>>> understand why, how and when.
>>>>
>>>> Having upstream bugzilla will also help us plan ahead, and manage
>>>> the break the project into smaller components, to assign core
>>>> developers for each.
>>>
>>> it still won't help tracking all relevant maintainers from rest
>>> api, ui, engine, vdsm saw the bug. we have multiple components.
>>> arch is mostly to cover cross component issues, such as features.
>>
>> this is why i'm suggesting creating mailing-groups peer component,
>> feature owner knows what layers are involved and should be
>> responsible to CC
>> relevant MGs in both BZ & thread.
> 
> mailing group is something nobody can manage.

why?, we do not have many layers in the system, creating:
"engine{core, db, network, storage/gluster}, api, ui, sdk, cli, vdsm, node"
mailing groups will do the trick.

> you can consider a group as all people who are CC to a bug.

CC'ing people manually can't guarantee that all relevant people will
be included in the email (thread).

> 
>>
>>> (or we could create a feature mailing list).
>>
>> ML/MG peer feature is too much i think.
>>
>>>
>>> but i do agree we could use bugzilla for tracking features per
>>> version, if wiki isn't tracking this correctly.
>>
>> +1
>>
>>
>> --
>>
>> Michael Pasternak
>> RedHat, ENG-Virtualization R&D
>>


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From alonbl at redhat.com  Sun Dec 23 09:07:23 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Sun, 23 Dec 2012 04:07:23 -0500 (EST)
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <50D6A3D3.7010701@redhat.com>
Message-ID: <2069177993.6577063.1356253643361.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Michael Pasternak" <mpastern at redhat.com>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org, "Itamar Heim" <iheim at redhat.com>
> Sent: Sunday, December 23, 2012 8:25:23 AM
> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
> 
> On 12/21/2012 01:57 PM, Alon Bar-Lev wrote:
> > 
> > 
> > ----- Original Message -----
> >> From: "Michael Pasternak" <mpastern at redhat.com>
> >> To: "Itamar Heim" <iheim at redhat.com>
> >> Cc: "Alon Bar-Lev" <alonbl at redhat.com>, "engine-devel"
> >> <engine-devel at ovirt.org>, arch at ovirt.org
> >> Sent: Friday, December 21, 2012 12:44:51 PM
> >> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
> >> the new feature discussion/implementation.
> >>
> >> On 12/21/2012 12:15 AM, Itamar Heim wrote:
> >>> On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
> >>>>
> >>>>
> >>>> ----- Original Message -----
> >>>>> From: "Itamar Heim" <iheim at redhat.com>
> >>>>> To: "Michael Pasternak" <mpastern at redhat.com>
> >>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
> >>>>> Sent: Thursday, December 20, 2012 11:42:37 PM
> >>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
> >>>>> the new feature discussion/implementation.
> >>>>>
> >>>>> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
> >>>>>>
> >>>>>> Hi All,
> >>>>>>
> >>>>>> In many cases OSS maintainers not always can be in the loop of
> >>>>>> different threads what may
> >>>>>> cause them missing important decisions being taken,
> >>>>>>
> >>>>>> As result later on during reviews of the patches they're not
> >>>>>> accepting (already implemented)
> >>>>>> features, what is causing not once for feature to be
> >>>>>> re-designed
> >>>>>> and/or delayed, what is wrong
> >>>>>> from the development cycle PoV.
> >>>>>>
> >>>>>> Therefore I'd like to suggest establishing dev-rules for the
> >>>>>> new
> >>>>>> feature implementation,
> >>>>>> what will make entire process much more easer for all of us:
> >>>>>>
> >>>>>> 1. discuss new feature on the mailing list
> >>>>>> (requirements/constraints/etc.)
> >>>>>> 2. summarise feature details in feature-doc
> >>>>>> 3. send feature-doc to review to:
> >>>>>>      3.1 ML (engine-devel)
> >>>>>>      3.2 MG (mailing group of maintainers of the relevant
> >>>>>>      layers)
> >>>>>> 4. after feature-doc is accepted,
> >>>>>>      4.1 implement the feature
> >>>>>>      4.2 send it to gerrit for review to:
> >>>>>>          4.2.1 lead maintainer/s (they will review/delegate
> >>>>>>          it)
> >>>>>>
> >>>>>>
> >>>>>> NOTE 3.2, 4.2.1 will require defining MGs such as:
> >>>>>>         - engine-devel-core
> >>>>>>         - engine-devel-ui
> >>>>>>         - engine-devel-api
> >>>>>>         - engine-devel-sdk
> >>>>>>         - engine-devel-cli
> >>>>>>         - engine-devel-vdsm ...
> >>>>>>
> >>>>>>
> >>>>>> Thoughts?
> >>>>>>
> >>>>>
> >>>>> i thought this is what we have the arch mailing list for, since
> >>>>> any
> >>>>> feature is going to cut through multiple layers/components,
> >>>>> unless
> >>>>> they
> >>>>> are very specific, they should be sent to arch, and all
> >>>>> maintainers
> >>>>> should follow arch.
> >>>>
> >>>> What is missing is upstream bugzilla.
> >>>>
> >>>> A feature, after initiate stage should be represented with a
> >>>> bug.
> >>>> The bug should be assigned to the right designated milestone.
> >>>> All document references (including versions) should be attached
> >>>> or
> >>>> referred by the bug.
> >>>> Dependency between features can be established using bug
> >>>> dependencies.
> >>>> Status can be acquired from buzilla at any time, progress
> >>>> reports
> >>>> should be input into bugzilla.
> >>>> Contact details for the feature can be acquired too, relevant
> >>>> and
> >>>> interested parties can be CCed explicitly.
> >>>> I guess I can add more benefits.
> >>>>
> >>>> Mailing list is good for idea initiation, but not for lifecycle
> >>>> management, nor for people to join at implementation phase and
> >>>> understand why, how and when.
> >>>>
> >>>> Having upstream bugzilla will also help us plan ahead, and
> >>>> manage
> >>>> the break the project into smaller components, to assign core
> >>>> developers for each.
> >>>
> >>> it still won't help tracking all relevant maintainers from rest
> >>> api, ui, engine, vdsm saw the bug. we have multiple components.
> >>> arch is mostly to cover cross component issues, such as features.
> >>
> >> this is why i'm suggesting creating mailing-groups peer component,
> >> feature owner knows what layers are involved and should be
> >> responsible to CC
> >> relevant MGs in both BZ & thread.
> > 
> > mailing group is something nobody can manage.
> 
> why?, we do not have many layers in the system, creating:
> "engine{core, db, network, storage/gluster}, api, ui, sdk, cli, vdsm,
> node"
> mailing groups will do the trick.

Because in open source you don't know who is actually out there.
This is very different from proprietary.
Also, features can be cross component / cross products / cross projects.

You can have default CC in bugzilla for the people you do know, but you must enable other people to subscribe.

> 
> > you can consider a group as all people who are CC to a bug.
> 
> CC'ing people manually can't guarantee that all relevant people will
> be included in the email (thread).
> 
> > 
> >>
> >>> (or we could create a feature mailing list).
> >>
> >> ML/MG peer feature is too much i think.
> >>
> >>>
> >>> but i do agree we could use bugzilla for tracking features per
> >>> version, if wiki isn't tracking this correctly.
> >>
> >> +1
> >>
> >>
> >> --
> >>
> >> Michael Pasternak
> >> RedHat, ENG-Virtualization R&D
> >>
> 
> 
> --
> 
> Michael Pasternak
> RedHat, ENG-Virtualization R&D
> 


From mpastern at redhat.com  Sun Dec 23 09:28:13 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Sun, 23 Dec 2012 11:28:13 +0200
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <2069177993.6577063.1356253643361.JavaMail.root@redhat.com>
References: <2069177993.6577063.1356253643361.JavaMail.root@redhat.com>
Message-ID: <50D6CEAD.5020600@redhat.com>

On 12/23/2012 11:07 AM, Alon Bar-Lev wrote:
> 
> 
> ----- Original Message -----
>> From: "Michael Pasternak" <mpastern at redhat.com>
>> To: "Alon Bar-Lev" <alonbl at redhat.com>
>> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org, "Itamar Heim" <iheim at redhat.com>
>> Sent: Sunday, December 23, 2012 8:25:23 AM
>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
>>
>> On 12/21/2012 01:57 PM, Alon Bar-Lev wrote:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Michael Pasternak" <mpastern at redhat.com>
>>>> To: "Itamar Heim" <iheim at redhat.com>
>>>> Cc: "Alon Bar-Lev" <alonbl at redhat.com>, "engine-devel"
>>>> <engine-devel at ovirt.org>, arch at ovirt.org
>>>> Sent: Friday, December 21, 2012 12:44:51 PM
>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
>>>> the new feature discussion/implementation.
>>>>
>>>> On 12/21/2012 12:15 AM, Itamar Heim wrote:
>>>>> On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Itamar Heim" <iheim at redhat.com>
>>>>>>> To: "Michael Pasternak" <mpastern at redhat.com>
>>>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>>>>> Sent: Thursday, December 20, 2012 11:42:37 PM
>>>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
>>>>>>> the new feature discussion/implementation.
>>>>>>>
>>>>>>> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> In many cases OSS maintainers not always can be in the loop of
>>>>>>>> different threads what may
>>>>>>>> cause them missing important decisions being taken,
>>>>>>>>
>>>>>>>> As result later on during reviews of the patches they're not
>>>>>>>> accepting (already implemented)
>>>>>>>> features, what is causing not once for feature to be
>>>>>>>> re-designed
>>>>>>>> and/or delayed, what is wrong
>>>>>>>> from the development cycle PoV.
>>>>>>>>
>>>>>>>> Therefore I'd like to suggest establishing dev-rules for the
>>>>>>>> new
>>>>>>>> feature implementation,
>>>>>>>> what will make entire process much more easer for all of us:
>>>>>>>>
>>>>>>>> 1. discuss new feature on the mailing list
>>>>>>>> (requirements/constraints/etc.)
>>>>>>>> 2. summarise feature details in feature-doc
>>>>>>>> 3. send feature-doc to review to:
>>>>>>>>      3.1 ML (engine-devel)
>>>>>>>>      3.2 MG (mailing group of maintainers of the relevant
>>>>>>>>      layers)
>>>>>>>> 4. after feature-doc is accepted,
>>>>>>>>      4.1 implement the feature
>>>>>>>>      4.2 send it to gerrit for review to:
>>>>>>>>          4.2.1 lead maintainer/s (they will review/delegate
>>>>>>>>          it)
>>>>>>>>
>>>>>>>>
>>>>>>>> NOTE 3.2, 4.2.1 will require defining MGs such as:
>>>>>>>>         - engine-devel-core
>>>>>>>>         - engine-devel-ui
>>>>>>>>         - engine-devel-api
>>>>>>>>         - engine-devel-sdk
>>>>>>>>         - engine-devel-cli
>>>>>>>>         - engine-devel-vdsm ...
>>>>>>>>
>>>>>>>>
>>>>>>>> Thoughts?
>>>>>>>>
>>>>>>>
>>>>>>> i thought this is what we have the arch mailing list for, since
>>>>>>> any
>>>>>>> feature is going to cut through multiple layers/components,
>>>>>>> unless
>>>>>>> they
>>>>>>> are very specific, they should be sent to arch, and all
>>>>>>> maintainers
>>>>>>> should follow arch.
>>>>>>
>>>>>> What is missing is upstream bugzilla.
>>>>>>
>>>>>> A feature, after initiate stage should be represented with a
>>>>>> bug.
>>>>>> The bug should be assigned to the right designated milestone.
>>>>>> All document references (including versions) should be attached
>>>>>> or
>>>>>> referred by the bug.
>>>>>> Dependency between features can be established using bug
>>>>>> dependencies.
>>>>>> Status can be acquired from buzilla at any time, progress
>>>>>> reports
>>>>>> should be input into bugzilla.
>>>>>> Contact details for the feature can be acquired too, relevant
>>>>>> and
>>>>>> interested parties can be CCed explicitly.
>>>>>> I guess I can add more benefits.
>>>>>>
>>>>>> Mailing list is good for idea initiation, but not for lifecycle
>>>>>> management, nor for people to join at implementation phase and
>>>>>> understand why, how and when.
>>>>>>
>>>>>> Having upstream bugzilla will also help us plan ahead, and
>>>>>> manage
>>>>>> the break the project into smaller components, to assign core
>>>>>> developers for each.
>>>>>
>>>>> it still won't help tracking all relevant maintainers from rest
>>>>> api, ui, engine, vdsm saw the bug. we have multiple components.
>>>>> arch is mostly to cover cross component issues, such as features.
>>>>
>>>> this is why i'm suggesting creating mailing-groups peer component,
>>>> feature owner knows what layers are involved and should be
>>>> responsible to CC
>>>> relevant MGs in both BZ & thread.
>>>
>>> mailing group is something nobody can manage.
>>
>> why?, we do not have many layers in the system, creating:
>> "engine{core, db, network, storage/gluster}, api, ui, sdk, cli, vdsm,
>> node"
>> mailing groups will do the trick.
> 
> Because in open source you don't know who is actually out there.

this is not about of including everyone in mailing-group, but maintainers,
and you always know who maintainers are.

> This is very different from proprietary.
> Also, features can be cross component / cross products / cross projects.

the one who implementing the feature knows what layers he is touching,
and can trigger relevant mailing-groups (this way feature owner does not
have to be aware of actual layer/s maintainers)

> 
> You can have default CC in bugzilla for the people you do know, but you must enable other people to subscribe.

i'm not talking about bagzila, but the standard process of introducing new
feature, - e.g discussion cannot take place in the bugzila (it's a tool to summarise the feature, etc.),
but i'd like to male sure all relevant maintainers get involved in feature design/discussion
before it get written and not only when they have to review the code, and mailing-groups is a
right tool for that in my view.

> 
>>
>>> you can consider a group as all people who are CC to a bug.
>>
>> CC'ing people manually can't guarantee that all relevant people will
>> be included in the email (thread).
>>
>>>
>>>>
>>>>> (or we could create a feature mailing list).
>>>>
>>>> ML/MG peer feature is too much i think.
>>>>
>>>>>
>>>>> but i do agree we could use bugzilla for tracking features per
>>>>> version, if wiki isn't tracking this correctly.
>>>>
>>>> +1
>>>>
>>>>
>>>> --
>>>>
>>>> Michael Pasternak
>>>> RedHat, ENG-Virtualization R&D
>>>>
>>
>>
>> --
>>
>> Michael Pasternak
>> RedHat, ENG-Virtualization R&D
>>


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From iheim at redhat.com  Sun Dec 23 09:35:40 2012
From: iheim at redhat.com (Itamar Heim)
Date: Sun, 23 Dec 2012 11:35:40 +0200
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <50D6CEAD.5020600@redhat.com>
References: <2069177993.6577063.1356253643361.JavaMail.root@redhat.com>
	<50D6CEAD.5020600@redhat.com>
Message-ID: <50D6D06C.6020003@redhat.com>

On 12/23/2012 11:28 AM, Michael Pasternak wrote:
> On 12/23/2012 11:07 AM, Alon Bar-Lev wrote:
>>
>>
>> ----- Original Message -----
>>> From: "Michael Pasternak" <mpastern at redhat.com>
>>> To: "Alon Bar-Lev" <alonbl at redhat.com>
>>> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org, "Itamar Heim" <iheim at redhat.com>
>>> Sent: Sunday, December 23, 2012 8:25:23 AM
>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
>>>
>>> On 12/21/2012 01:57 PM, Alon Bar-Lev wrote:
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Michael Pasternak" <mpastern at redhat.com>
>>>>> To: "Itamar Heim" <iheim at redhat.com>
>>>>> Cc: "Alon Bar-Lev" <alonbl at redhat.com>, "engine-devel"
>>>>> <engine-devel at ovirt.org>, arch at ovirt.org
>>>>> Sent: Friday, December 21, 2012 12:44:51 PM
>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
>>>>> the new feature discussion/implementation.
>>>>>
>>>>> On 12/21/2012 12:15 AM, Itamar Heim wrote:
>>>>>> On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Itamar Heim" <iheim at redhat.com>
>>>>>>>> To: "Michael Pasternak" <mpastern at redhat.com>
>>>>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>>>>>> Sent: Thursday, December 20, 2012 11:42:37 PM
>>>>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
>>>>>>>> the new feature discussion/implementation.
>>>>>>>>
>>>>>>>> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
>>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> In many cases OSS maintainers not always can be in the loop of
>>>>>>>>> different threads what may
>>>>>>>>> cause them missing important decisions being taken,
>>>>>>>>>
>>>>>>>>> As result later on during reviews of the patches they're not
>>>>>>>>> accepting (already implemented)
>>>>>>>>> features, what is causing not once for feature to be
>>>>>>>>> re-designed
>>>>>>>>> and/or delayed, what is wrong
>>>>>>>>> from the development cycle PoV.
>>>>>>>>>
>>>>>>>>> Therefore I'd like to suggest establishing dev-rules for the
>>>>>>>>> new
>>>>>>>>> feature implementation,
>>>>>>>>> what will make entire process much more easer for all of us:
>>>>>>>>>
>>>>>>>>> 1. discuss new feature on the mailing list
>>>>>>>>> (requirements/constraints/etc.)
>>>>>>>>> 2. summarise feature details in feature-doc
>>>>>>>>> 3. send feature-doc to review to:
>>>>>>>>>       3.1 ML (engine-devel)
>>>>>>>>>       3.2 MG (mailing group of maintainers of the relevant
>>>>>>>>>       layers)
>>>>>>>>> 4. after feature-doc is accepted,
>>>>>>>>>       4.1 implement the feature
>>>>>>>>>       4.2 send it to gerrit for review to:
>>>>>>>>>           4.2.1 lead maintainer/s (they will review/delegate
>>>>>>>>>           it)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> NOTE 3.2, 4.2.1 will require defining MGs such as:
>>>>>>>>>          - engine-devel-core
>>>>>>>>>          - engine-devel-ui
>>>>>>>>>          - engine-devel-api
>>>>>>>>>          - engine-devel-sdk
>>>>>>>>>          - engine-devel-cli
>>>>>>>>>          - engine-devel-vdsm ...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>>>
>>>>>>>>
>>>>>>>> i thought this is what we have the arch mailing list for, since
>>>>>>>> any
>>>>>>>> feature is going to cut through multiple layers/components,
>>>>>>>> unless
>>>>>>>> they
>>>>>>>> are very specific, they should be sent to arch, and all
>>>>>>>> maintainers
>>>>>>>> should follow arch.
>>>>>>>
>>>>>>> What is missing is upstream bugzilla.
>>>>>>>
>>>>>>> A feature, after initiate stage should be represented with a
>>>>>>> bug.
>>>>>>> The bug should be assigned to the right designated milestone.
>>>>>>> All document references (including versions) should be attached
>>>>>>> or
>>>>>>> referred by the bug.
>>>>>>> Dependency between features can be established using bug
>>>>>>> dependencies.
>>>>>>> Status can be acquired from buzilla at any time, progress
>>>>>>> reports
>>>>>>> should be input into bugzilla.
>>>>>>> Contact details for the feature can be acquired too, relevant
>>>>>>> and
>>>>>>> interested parties can be CCed explicitly.
>>>>>>> I guess I can add more benefits.
>>>>>>>
>>>>>>> Mailing list is good for idea initiation, but not for lifecycle
>>>>>>> management, nor for people to join at implementation phase and
>>>>>>> understand why, how and when.
>>>>>>>
>>>>>>> Having upstream bugzilla will also help us plan ahead, and
>>>>>>> manage
>>>>>>> the break the project into smaller components, to assign core
>>>>>>> developers for each.
>>>>>>
>>>>>> it still won't help tracking all relevant maintainers from rest
>>>>>> api, ui, engine, vdsm saw the bug. we have multiple components.
>>>>>> arch is mostly to cover cross component issues, such as features.
>>>>>
>>>>> this is why i'm suggesting creating mailing-groups peer component,
>>>>> feature owner knows what layers are involved and should be
>>>>> responsible to CC
>>>>> relevant MGs in both BZ & thread.
>>>>
>>>> mailing group is something nobody can manage.
>>>
>>> why?, we do not have many layers in the system, creating:
>>> "engine{core, db, network, storage/gluster}, api, ui, sdk, cli, vdsm,
>>> node"
>>> mailing groups will do the trick.
>>
>> Because in open source you don't know who is actually out there.
>
> this is not about of including everyone in mailing-group, but maintainers,
> and you always know who maintainers are.
>
>> This is very different from proprietary.
>> Also, features can be cross component / cross products / cross projects.
>
> the one who implementing the feature knows what layers he is touching,
> and can trigger relevant mailing-groups (this way feature owner does not
> have to be aware of actual layer/s maintainers)
>
>>
>> You can have default CC in bugzilla for the people you do know, but you must enable other people to subscribe.
>
> i'm not talking about bagzila, but the standard process of introducing new
> feature, - e.g discussion cannot take place in the bugzila (it's a tool to summarise the feature, etc.),
> but i'd like to male sure all relevant maintainers get involved in feature design/discussion
> before it get written and not only when they have to review the code, and mailing-groups is a
> right tool for that in my view.

i really expect all maintainers to follow the arch mailing list for this.

>
>>
>>>
>>>> you can consider a group as all people who are CC to a bug.
>>>
>>> CC'ing people manually can't guarantee that all relevant people will
>>> be included in the email (thread).
>>>
>>>>
>>>>>
>>>>>> (or we could create a feature mailing list).
>>>>>
>>>>> ML/MG peer feature is too much i think.
>>>>>
>>>>>>
>>>>>> but i do agree we could use bugzilla for tracking features per
>>>>>> version, if wiki isn't tracking this correctly.
>>>>>
>>>>> +1
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Michael Pasternak
>>>>> RedHat, ENG-Virtualization R&D
>>>>>
>>>
>>>
>>> --
>>>
>>> Michael Pasternak
>>> RedHat, ENG-Virtualization R&D
>>>
>
>




From alonbl at redhat.com  Sun Dec 23 09:39:09 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Sun, 23 Dec 2012 04:39:09 -0500 (EST)
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <50D6CEAD.5020600@redhat.com>
Message-ID: <530587269.6577245.1356255549940.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Michael Pasternak" <mpastern at redhat.com>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org, "Itamar Heim" <iheim at redhat.com>
> Sent: Sunday, December 23, 2012 11:28:13 AM
> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
> 
> On 12/23/2012 11:07 AM, Alon Bar-Lev wrote:
> > 
> > 
> > ----- Original Message -----
> >> From: "Michael Pasternak" <mpastern at redhat.com>
> >> To: "Alon Bar-Lev" <alonbl at redhat.com>
> >> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org,
> >> "Itamar Heim" <iheim at redhat.com>
> >> Sent: Sunday, December 23, 2012 8:25:23 AM
> >> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
> >> the new feature discussion/implementation.
> >>
> >> On 12/21/2012 01:57 PM, Alon Bar-Lev wrote:
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Michael Pasternak" <mpastern at redhat.com>
> >>>> To: "Itamar Heim" <iheim at redhat.com>
> >>>> Cc: "Alon Bar-Lev" <alonbl at redhat.com>, "engine-devel"
> >>>> <engine-devel at ovirt.org>, arch at ovirt.org
> >>>> Sent: Friday, December 21, 2012 12:44:51 PM
> >>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
> >>>> the new feature discussion/implementation.
> >>>>
> >>>> On 12/21/2012 12:15 AM, Itamar Heim wrote:
> >>>>> On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
> >>>>>>
> >>>>>>
> >>>>>> ----- Original Message -----
> >>>>>>> From: "Itamar Heim" <iheim at redhat.com>
> >>>>>>> To: "Michael Pasternak" <mpastern at redhat.com>
> >>>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
> >>>>>>> Sent: Thursday, December 20, 2012 11:42:37 PM
> >>>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process
> >>>>>>> for
> >>>>>>> the new feature discussion/implementation.
> >>>>>>>
> >>>>>>> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
> >>>>>>>>
> >>>>>>>> Hi All,
> >>>>>>>>
> >>>>>>>> In many cases OSS maintainers not always can be in the loop
> >>>>>>>> of
> >>>>>>>> different threads what may
> >>>>>>>> cause them missing important decisions being taken,
> >>>>>>>>
> >>>>>>>> As result later on during reviews of the patches they're not
> >>>>>>>> accepting (already implemented)
> >>>>>>>> features, what is causing not once for feature to be
> >>>>>>>> re-designed
> >>>>>>>> and/or delayed, what is wrong
> >>>>>>>> from the development cycle PoV.
> >>>>>>>>
> >>>>>>>> Therefore I'd like to suggest establishing dev-rules for the
> >>>>>>>> new
> >>>>>>>> feature implementation,
> >>>>>>>> what will make entire process much more easer for all of us:
> >>>>>>>>
> >>>>>>>> 1. discuss new feature on the mailing list
> >>>>>>>> (requirements/constraints/etc.)
> >>>>>>>> 2. summarise feature details in feature-doc
> >>>>>>>> 3. send feature-doc to review to:
> >>>>>>>>      3.1 ML (engine-devel)
> >>>>>>>>      3.2 MG (mailing group of maintainers of the relevant
> >>>>>>>>      layers)
> >>>>>>>> 4. after feature-doc is accepted,
> >>>>>>>>      4.1 implement the feature
> >>>>>>>>      4.2 send it to gerrit for review to:
> >>>>>>>>          4.2.1 lead maintainer/s (they will review/delegate
> >>>>>>>>          it)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> NOTE 3.2, 4.2.1 will require defining MGs such as:
> >>>>>>>>         - engine-devel-core
> >>>>>>>>         - engine-devel-ui
> >>>>>>>>         - engine-devel-api
> >>>>>>>>         - engine-devel-sdk
> >>>>>>>>         - engine-devel-cli
> >>>>>>>>         - engine-devel-vdsm ...
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Thoughts?
> >>>>>>>>
> >>>>>>>
> >>>>>>> i thought this is what we have the arch mailing list for,
> >>>>>>> since
> >>>>>>> any
> >>>>>>> feature is going to cut through multiple layers/components,
> >>>>>>> unless
> >>>>>>> they
> >>>>>>> are very specific, they should be sent to arch, and all
> >>>>>>> maintainers
> >>>>>>> should follow arch.
> >>>>>>
> >>>>>> What is missing is upstream bugzilla.
> >>>>>>
> >>>>>> A feature, after initiate stage should be represented with a
> >>>>>> bug.
> >>>>>> The bug should be assigned to the right designated milestone.
> >>>>>> All document references (including versions) should be
> >>>>>> attached
> >>>>>> or
> >>>>>> referred by the bug.
> >>>>>> Dependency between features can be established using bug
> >>>>>> dependencies.
> >>>>>> Status can be acquired from buzilla at any time, progress
> >>>>>> reports
> >>>>>> should be input into bugzilla.
> >>>>>> Contact details for the feature can be acquired too, relevant
> >>>>>> and
> >>>>>> interested parties can be CCed explicitly.
> >>>>>> I guess I can add more benefits.
> >>>>>>
> >>>>>> Mailing list is good for idea initiation, but not for
> >>>>>> lifecycle
> >>>>>> management, nor for people to join at implementation phase and
> >>>>>> understand why, how and when.
> >>>>>>
> >>>>>> Having upstream bugzilla will also help us plan ahead, and
> >>>>>> manage
> >>>>>> the break the project into smaller components, to assign core
> >>>>>> developers for each.
> >>>>>
> >>>>> it still won't help tracking all relevant maintainers from rest
> >>>>> api, ui, engine, vdsm saw the bug. we have multiple components.
> >>>>> arch is mostly to cover cross component issues, such as
> >>>>> features.
> >>>>
> >>>> this is why i'm suggesting creating mailing-groups peer
> >>>> component,
> >>>> feature owner knows what layers are involved and should be
> >>>> responsible to CC
> >>>> relevant MGs in both BZ & thread.
> >>>
> >>> mailing group is something nobody can manage.
> >>
> >> why?, we do not have many layers in the system, creating:
> >> "engine{core, db, network, storage/gluster}, api, ui, sdk, cli,
> >> vdsm,
> >> node"
> >> mailing groups will do the trick.
> > 
> > Because in open source you don't know who is actually out there.
> 
> this is not about of including everyone in mailing-group, but
> maintainers,
> and you always know who maintainers are.

I am sorry, but this is not "open" discussion.
You *DO NOT* know who is interested, there can be other people that are highly valuable for a discussion and you are going to loose them.

Maintainer is not someone who is more important the contributer, he just has commit access.

> 
> > This is very different from proprietary.
> > Also, features can be cross component / cross products / cross
> > projects.
> 
> the one who implementing the feature knows what layers he is
> touching,
> and can trigger relevant mailing-groups (this way feature owner does
> not
> have to be aware of actual layer/s maintainers)

Again, you lose one of the most important open source benefit, allowing everyone to participate and contribute.

If you like, a message subject convention will be more productive, "maintainer" may set trigger for a specific prefix, and there can be a documentation of this convention for features and other uses.

All discussions should be made in public, this is burden but also opportunity.

> > 
> > You can have default CC in bugzilla for the people you do know, but
> > you must enable other people to subscribe.
> 
> i'm not talking about bagzila, but the standard process of
> introducing new
> feature, - e.g discussion cannot take place in the bugzila (it's a
> tool to summarise the feature, etc.),
> but i'd like to male sure all relevant maintainers get involved in
> feature design/discussion
> before it get written and not only when they have to review the code,
> and mailing-groups is a
> right tool for that in my view.
> 
> > 
> >>
> >>> you can consider a group as all people who are CC to a bug.
> >>
> >> CC'ing people manually can't guarantee that all relevant people
> >> will
> >> be included in the email (thread).
> >>
> >>>
> >>>>
> >>>>> (or we could create a feature mailing list).
> >>>>
> >>>> ML/MG peer feature is too much i think.
> >>>>
> >>>>>
> >>>>> but i do agree we could use bugzilla for tracking features per
> >>>>> version, if wiki isn't tracking this correctly.
> >>>>
> >>>> +1
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Michael Pasternak
> >>>> RedHat, ENG-Virtualization R&D
> >>>>
> >>
> >>
> >> --
> >>
> >> Michael Pasternak
> >> RedHat, ENG-Virtualization R&D
> >>
> 
> 
> --
> 
> Michael Pasternak
> RedHat, ENG-Virtualization R&D
> 


From mpastern at redhat.com  Sun Dec 23 09:52:56 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Sun, 23 Dec 2012 11:52:56 +0200
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <50D6D06C.6020003@redhat.com>
References: <2069177993.6577063.1356253643361.JavaMail.root@redhat.com>
	<50D6CEAD.5020600@redhat.com> <50D6D06C.6020003@redhat.com>
Message-ID: <50D6D478.7050006@redhat.com>

On 12/23/2012 11:35 AM, Itamar Heim wrote:
> On 12/23/2012 11:28 AM, Michael Pasternak wrote:
>> On 12/23/2012 11:07 AM, Alon Bar-Lev wrote:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Michael Pasternak" <mpastern at redhat.com>
>>>> To: "Alon Bar-Lev" <alonbl at redhat.com>
>>>> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org, "Itamar Heim" <iheim at redhat.com>
>>>> Sent: Sunday, December 23, 2012 8:25:23 AM
>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
>>>>
>>>> On 12/21/2012 01:57 PM, Alon Bar-Lev wrote:
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Michael Pasternak" <mpastern at redhat.com>
>>>>>> To: "Itamar Heim" <iheim at redhat.com>
>>>>>> Cc: "Alon Bar-Lev" <alonbl at redhat.com>, "engine-devel"
>>>>>> <engine-devel at ovirt.org>, arch at ovirt.org
>>>>>> Sent: Friday, December 21, 2012 12:44:51 PM
>>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
>>>>>> the new feature discussion/implementation.
>>>>>>
>>>>>> On 12/21/2012 12:15 AM, Itamar Heim wrote:
>>>>>>> On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Itamar Heim" <iheim at redhat.com>
>>>>>>>>> To: "Michael Pasternak" <mpastern at redhat.com>
>>>>>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>>>>>>> Sent: Thursday, December 20, 2012 11:42:37 PM
>>>>>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
>>>>>>>>> the new feature discussion/implementation.
>>>>>>>>>
>>>>>>>>> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
>>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> In many cases OSS maintainers not always can be in the loop of
>>>>>>>>>> different threads what may
>>>>>>>>>> cause them missing important decisions being taken,
>>>>>>>>>>
>>>>>>>>>> As result later on during reviews of the patches they're not
>>>>>>>>>> accepting (already implemented)
>>>>>>>>>> features, what is causing not once for feature to be
>>>>>>>>>> re-designed
>>>>>>>>>> and/or delayed, what is wrong
>>>>>>>>>> from the development cycle PoV.
>>>>>>>>>>
>>>>>>>>>> Therefore I'd like to suggest establishing dev-rules for the
>>>>>>>>>> new
>>>>>>>>>> feature implementation,
>>>>>>>>>> what will make entire process much more easer for all of us:
>>>>>>>>>>
>>>>>>>>>> 1. discuss new feature on the mailing list
>>>>>>>>>> (requirements/constraints/etc.)
>>>>>>>>>> 2. summarise feature details in feature-doc
>>>>>>>>>> 3. send feature-doc to review to:
>>>>>>>>>>       3.1 ML (engine-devel)
>>>>>>>>>>       3.2 MG (mailing group of maintainers of the relevant
>>>>>>>>>>       layers)
>>>>>>>>>> 4. after feature-doc is accepted,
>>>>>>>>>>       4.1 implement the feature
>>>>>>>>>>       4.2 send it to gerrit for review to:
>>>>>>>>>>           4.2.1 lead maintainer/s (they will review/delegate
>>>>>>>>>>           it)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> NOTE 3.2, 4.2.1 will require defining MGs such as:
>>>>>>>>>>          - engine-devel-core
>>>>>>>>>>          - engine-devel-ui
>>>>>>>>>>          - engine-devel-api
>>>>>>>>>>          - engine-devel-sdk
>>>>>>>>>>          - engine-devel-cli
>>>>>>>>>>          - engine-devel-vdsm ...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thoughts?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> i thought this is what we have the arch mailing list for, since
>>>>>>>>> any
>>>>>>>>> feature is going to cut through multiple layers/components,
>>>>>>>>> unless
>>>>>>>>> they
>>>>>>>>> are very specific, they should be sent to arch, and all
>>>>>>>>> maintainers
>>>>>>>>> should follow arch.
>>>>>>>>
>>>>>>>> What is missing is upstream bugzilla.
>>>>>>>>
>>>>>>>> A feature, after initiate stage should be represented with a
>>>>>>>> bug.
>>>>>>>> The bug should be assigned to the right designated milestone.
>>>>>>>> All document references (including versions) should be attached
>>>>>>>> or
>>>>>>>> referred by the bug.
>>>>>>>> Dependency between features can be established using bug
>>>>>>>> dependencies.
>>>>>>>> Status can be acquired from buzilla at any time, progress
>>>>>>>> reports
>>>>>>>> should be input into bugzilla.
>>>>>>>> Contact details for the feature can be acquired too, relevant
>>>>>>>> and
>>>>>>>> interested parties can be CCed explicitly.
>>>>>>>> I guess I can add more benefits.
>>>>>>>>
>>>>>>>> Mailing list is good for idea initiation, but not for lifecycle
>>>>>>>> management, nor for people to join at implementation phase and
>>>>>>>> understand why, how and when.
>>>>>>>>
>>>>>>>> Having upstream bugzilla will also help us plan ahead, and
>>>>>>>> manage
>>>>>>>> the break the project into smaller components, to assign core
>>>>>>>> developers for each.
>>>>>>>
>>>>>>> it still won't help tracking all relevant maintainers from rest
>>>>>>> api, ui, engine, vdsm saw the bug. we have multiple components.
>>>>>>> arch is mostly to cover cross component issues, such as features.
>>>>>>
>>>>>> this is why i'm suggesting creating mailing-groups peer component,
>>>>>> feature owner knows what layers are involved and should be
>>>>>> responsible to CC
>>>>>> relevant MGs in both BZ & thread.
>>>>>
>>>>> mailing group is something nobody can manage.
>>>>
>>>> why?, we do not have many layers in the system, creating:
>>>> "engine{core, db, network, storage/gluster}, api, ui, sdk, cli, vdsm,
>>>> node"
>>>> mailing groups will do the trick.
>>>
>>> Because in open source you don't know who is actually out there.
>>
>> this is not about of including everyone in mailing-group, but maintainers,
>> and you always know who maintainers are.
>>
>>> This is very different from proprietary.
>>> Also, features can be cross component / cross products / cross projects.
>>
>> the one who implementing the feature knows what layers he is touching,
>> and can trigger relevant mailing-groups (this way feature owner does not
>> have to be aware of actual layer/s maintainers)
>>
>>>
>>> You can have default CC in bugzilla for the people you do know, but you must enable other people to subscribe.
>>
>> i'm not talking about bagzila, but the standard process of introducing new
>> feature, - e.g discussion cannot take place in the bugzila (it's a tool to summarise the feature, etc.),
>> but i'd like to male sure all relevant maintainers get involved in feature design/discussion
>> before it get written and not only when they have to review the code, and mailing-groups is a
>> right tool for that in my view.
> 
> i really expect all maintainers to follow the arch mailing list for this.

Itamar,

i think it's not feasible, people has a lot of constraints and can easily miss stuff,
getting personal notifications is much more practical and can dramatically reduce
redesign of the features during review processes, what is quite painful for the contributors,

i think this way it will be much more easer to follow feature discussions, and i'm sure
it won't cause maintainers to stop following mailing lists,

just entire process will become much more productive for both sides.

> 
>>
>>>
>>>>
>>>>> you can consider a group as all people who are CC to a bug.
>>>>
>>>> CC'ing people manually can't guarantee that all relevant people will
>>>> be included in the email (thread).
>>>>
>>>>>
>>>>>>
>>>>>>> (or we could create a feature mailing list).
>>>>>>
>>>>>> ML/MG peer feature is too much i think.
>>>>>>
>>>>>>>
>>>>>>> but i do agree we could use bugzilla for tracking features per
>>>>>>> version, if wiki isn't tracking this correctly.
>>>>>>
>>>>>> +1
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>>
>>>>>> Michael Pasternak
>>>>>> RedHat, ENG-Virtualization R&D
>>>>>>
>>>>
>>>>
>>>> -- 
>>>>
>>>> Michael Pasternak
>>>> RedHat, ENG-Virtualization R&D
>>>>
>>
>>
> 
> 


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From mpastern at redhat.com  Sun Dec 23 10:01:25 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Sun, 23 Dec 2012 12:01:25 +0200
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <530587269.6577245.1356255549940.JavaMail.root@redhat.com>
References: <530587269.6577245.1356255549940.JavaMail.root@redhat.com>
Message-ID: <50D6D675.6050808@redhat.com>

On 12/23/2012 11:39 AM, Alon Bar-Lev wrote:
> 
> 
> ----- Original Message -----
>> From: "Michael Pasternak" <mpastern at redhat.com>
>> To: "Alon Bar-Lev" <alonbl at redhat.com>
>> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org, "Itamar Heim" <iheim at redhat.com>
>> Sent: Sunday, December 23, 2012 11:28:13 AM
>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
>>
>> On 12/23/2012 11:07 AM, Alon Bar-Lev wrote:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Michael Pasternak" <mpastern at redhat.com>
>>>> To: "Alon Bar-Lev" <alonbl at redhat.com>
>>>> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org,
>>>> "Itamar Heim" <iheim at redhat.com>
>>>> Sent: Sunday, December 23, 2012 8:25:23 AM
>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
>>>> the new feature discussion/implementation.
>>>>
>>>> On 12/21/2012 01:57 PM, Alon Bar-Lev wrote:
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Michael Pasternak" <mpastern at redhat.com>
>>>>>> To: "Itamar Heim" <iheim at redhat.com>
>>>>>> Cc: "Alon Bar-Lev" <alonbl at redhat.com>, "engine-devel"
>>>>>> <engine-devel at ovirt.org>, arch at ovirt.org
>>>>>> Sent: Friday, December 21, 2012 12:44:51 PM
>>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
>>>>>> the new feature discussion/implementation.
>>>>>>
>>>>>> On 12/21/2012 12:15 AM, Itamar Heim wrote:
>>>>>>> On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Itamar Heim" <iheim at redhat.com>
>>>>>>>>> To: "Michael Pasternak" <mpastern at redhat.com>
>>>>>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>>>>>>> Sent: Thursday, December 20, 2012 11:42:37 PM
>>>>>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process
>>>>>>>>> for
>>>>>>>>> the new feature discussion/implementation.
>>>>>>>>>
>>>>>>>>> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
>>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> In many cases OSS maintainers not always can be in the loop
>>>>>>>>>> of
>>>>>>>>>> different threads what may
>>>>>>>>>> cause them missing important decisions being taken,
>>>>>>>>>>
>>>>>>>>>> As result later on during reviews of the patches they're not
>>>>>>>>>> accepting (already implemented)
>>>>>>>>>> features, what is causing not once for feature to be
>>>>>>>>>> re-designed
>>>>>>>>>> and/or delayed, what is wrong
>>>>>>>>>> from the development cycle PoV.
>>>>>>>>>>
>>>>>>>>>> Therefore I'd like to suggest establishing dev-rules for the
>>>>>>>>>> new
>>>>>>>>>> feature implementation,
>>>>>>>>>> what will make entire process much more easer for all of us:
>>>>>>>>>>
>>>>>>>>>> 1. discuss new feature on the mailing list
>>>>>>>>>> (requirements/constraints/etc.)
>>>>>>>>>> 2. summarise feature details in feature-doc
>>>>>>>>>> 3. send feature-doc to review to:
>>>>>>>>>>      3.1 ML (engine-devel)
>>>>>>>>>>      3.2 MG (mailing group of maintainers of the relevant
>>>>>>>>>>      layers)
>>>>>>>>>> 4. after feature-doc is accepted,
>>>>>>>>>>      4.1 implement the feature
>>>>>>>>>>      4.2 send it to gerrit for review to:
>>>>>>>>>>          4.2.1 lead maintainer/s (they will review/delegate
>>>>>>>>>>          it)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> NOTE 3.2, 4.2.1 will require defining MGs such as:
>>>>>>>>>>         - engine-devel-core
>>>>>>>>>>         - engine-devel-ui
>>>>>>>>>>         - engine-devel-api
>>>>>>>>>>         - engine-devel-sdk
>>>>>>>>>>         - engine-devel-cli
>>>>>>>>>>         - engine-devel-vdsm ...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thoughts?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> i thought this is what we have the arch mailing list for,
>>>>>>>>> since
>>>>>>>>> any
>>>>>>>>> feature is going to cut through multiple layers/components,
>>>>>>>>> unless
>>>>>>>>> they
>>>>>>>>> are very specific, they should be sent to arch, and all
>>>>>>>>> maintainers
>>>>>>>>> should follow arch.
>>>>>>>>
>>>>>>>> What is missing is upstream bugzilla.
>>>>>>>>
>>>>>>>> A feature, after initiate stage should be represented with a
>>>>>>>> bug.
>>>>>>>> The bug should be assigned to the right designated milestone.
>>>>>>>> All document references (including versions) should be
>>>>>>>> attached
>>>>>>>> or
>>>>>>>> referred by the bug.
>>>>>>>> Dependency between features can be established using bug
>>>>>>>> dependencies.
>>>>>>>> Status can be acquired from buzilla at any time, progress
>>>>>>>> reports
>>>>>>>> should be input into bugzilla.
>>>>>>>> Contact details for the feature can be acquired too, relevant
>>>>>>>> and
>>>>>>>> interested parties can be CCed explicitly.
>>>>>>>> I guess I can add more benefits.
>>>>>>>>
>>>>>>>> Mailing list is good for idea initiation, but not for
>>>>>>>> lifecycle
>>>>>>>> management, nor for people to join at implementation phase and
>>>>>>>> understand why, how and when.
>>>>>>>>
>>>>>>>> Having upstream bugzilla will also help us plan ahead, and
>>>>>>>> manage
>>>>>>>> the break the project into smaller components, to assign core
>>>>>>>> developers for each.
>>>>>>>
>>>>>>> it still won't help tracking all relevant maintainers from rest
>>>>>>> api, ui, engine, vdsm saw the bug. we have multiple components.
>>>>>>> arch is mostly to cover cross component issues, such as
>>>>>>> features.
>>>>>>
>>>>>> this is why i'm suggesting creating mailing-groups peer
>>>>>> component,
>>>>>> feature owner knows what layers are involved and should be
>>>>>> responsible to CC
>>>>>> relevant MGs in both BZ & thread.
>>>>>
>>>>> mailing group is something nobody can manage.
>>>>
>>>> why?, we do not have many layers in the system, creating:
>>>> "engine{core, db, network, storage/gluster}, api, ui, sdk, cli,
>>>> vdsm,
>>>> node"
>>>> mailing groups will do the trick.
>>>
>>> Because in open source you don't know who is actually out there.
>>
>> this is not about of including everyone in mailing-group, but
>> maintainers,
>> and you always know who maintainers are.
> 
> I am sorry, but this is not "open" discussion.
> You *DO NOT* know who is interested, there can be other people that are highly valuable for a discussion and you are going to loose them.

how possible you will loose them? adding mailing-group/s of maintainers does *not* mean
excluding the mailing list itself ...

anyone that is interested in discussion can still follow it.

> 
> Maintainer is not someone who is more important the contributer, he just has commit access.

i do agree about maintainers not being "more important" than contributor, but for sure maintainers have more
experience with the projects and planned stuff, so they can drive features to the right direction ...

> 
>>
>>> This is very different from proprietary.
>>> Also, features can be cross component / cross products / cross
>>> projects.
>>
>> the one who implementing the feature knows what layers he is
>> touching,
>> and can trigger relevant mailing-groups (this way feature owner does
>> not
>> have to be aware of actual layer/s maintainers)
> 
> Again, you lose one of the most important open source benefit, allowing everyone to participate and contribute.
> 
> If you like, a message subject convention will be more productive, "maintainer" may set trigger for a specific prefix, and there can be a documentation of this convention for features and other uses.
> 
> All discussions should be made in public, this is burden but also opportunity.

Alon,

i think you missed the point, read my answer ^ above, + entire thread was started,
to make sure maintainers explicitly got involved in to design of the features and
not changing it when the code is written (what for sure will make contributors frustrated)

> 
>>>
>>> You can have default CC in bugzilla for the people you do know, but
>>> you must enable other people to subscribe.
>>
>> i'm not talking about bagzila, but the standard process of
>> introducing new
>> feature, - e.g discussion cannot take place in the bugzila (it's a
>> tool to summarise the feature, etc.),
>> but i'd like to male sure all relevant maintainers get involved in
>> feature design/discussion
>> before it get written and not only when they have to review the code,
>> and mailing-groups is a
>> right tool for that in my view.
>>
>>>
>>>>
>>>>> you can consider a group as all people who are CC to a bug.
>>>>
>>>> CC'ing people manually can't guarantee that all relevant people
>>>> will
>>>> be included in the email (thread).
>>>>
>>>>>
>>>>>>
>>>>>>> (or we could create a feature mailing list).
>>>>>>
>>>>>> ML/MG peer feature is too much i think.
>>>>>>
>>>>>>>
>>>>>>> but i do agree we could use bugzilla for tracking features per
>>>>>>> version, if wiki isn't tracking this correctly.
>>>>>>
>>>>>> +1
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Michael Pasternak
>>>>>> RedHat, ENG-Virtualization R&D
>>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Michael Pasternak
>>>> RedHat, ENG-Virtualization R&D
>>>>
>>
>>
>> --
>>
>> Michael Pasternak
>> RedHat, ENG-Virtualization R&D
>>


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From alonbl at redhat.com  Sun Dec 23 10:16:54 2012
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Sun, 23 Dec 2012 05:16:54 -0500 (EST)
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <50D6D675.6050808@redhat.com>
Message-ID: <1935203280.6577398.1356257814058.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Michael Pasternak" <mpastern at redhat.com>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org, "Itamar Heim" <iheim at redhat.com>
> Sent: Sunday, December 23, 2012 12:01:25 PM
> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
> 
> On 12/23/2012 11:39 AM, Alon Bar-Lev wrote:
> > 
> > 
> > ----- Original Message -----
> >> From: "Michael Pasternak" <mpastern at redhat.com>
> >> To: "Alon Bar-Lev" <alonbl at redhat.com>
> >> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org,
> >> "Itamar Heim" <iheim at redhat.com>
> >> Sent: Sunday, December 23, 2012 11:28:13 AM
> >> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
> >> the new feature discussion/implementation.
> >>
> >> On 12/23/2012 11:07 AM, Alon Bar-Lev wrote:
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Michael Pasternak" <mpastern at redhat.com>
> >>>> To: "Alon Bar-Lev" <alonbl at redhat.com>
> >>>> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org,
> >>>> "Itamar Heim" <iheim at redhat.com>
> >>>> Sent: Sunday, December 23, 2012 8:25:23 AM
> >>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
> >>>> the new feature discussion/implementation.
> >>>>
> >>>> On 12/21/2012 01:57 PM, Alon Bar-Lev wrote:
> >>>>>
> >>>>>
> >>>>> ----- Original Message -----
> >>>>>> From: "Michael Pasternak" <mpastern at redhat.com>
> >>>>>> To: "Itamar Heim" <iheim at redhat.com>
> >>>>>> Cc: "Alon Bar-Lev" <alonbl at redhat.com>, "engine-devel"
> >>>>>> <engine-devel at ovirt.org>, arch at ovirt.org
> >>>>>> Sent: Friday, December 21, 2012 12:44:51 PM
> >>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process
> >>>>>> for
> >>>>>> the new feature discussion/implementation.
> >>>>>>
> >>>>>> On 12/21/2012 12:15 AM, Itamar Heim wrote:
> >>>>>>> On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ----- Original Message -----
> >>>>>>>>> From: "Itamar Heim" <iheim at redhat.com>
> >>>>>>>>> To: "Michael Pasternak" <mpastern at redhat.com>
> >>>>>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
> >>>>>>>>> Sent: Thursday, December 20, 2012 11:42:37 PM
> >>>>>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process
> >>>>>>>>> for
> >>>>>>>>> the new feature discussion/implementation.
> >>>>>>>>>
> >>>>>>>>> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi All,
> >>>>>>>>>>
> >>>>>>>>>> In many cases OSS maintainers not always can be in the
> >>>>>>>>>> loop
> >>>>>>>>>> of
> >>>>>>>>>> different threads what may
> >>>>>>>>>> cause them missing important decisions being taken,
> >>>>>>>>>>
> >>>>>>>>>> As result later on during reviews of the patches they're
> >>>>>>>>>> not
> >>>>>>>>>> accepting (already implemented)
> >>>>>>>>>> features, what is causing not once for feature to be
> >>>>>>>>>> re-designed
> >>>>>>>>>> and/or delayed, what is wrong
> >>>>>>>>>> from the development cycle PoV.
> >>>>>>>>>>
> >>>>>>>>>> Therefore I'd like to suggest establishing dev-rules for
> >>>>>>>>>> the
> >>>>>>>>>> new
> >>>>>>>>>> feature implementation,
> >>>>>>>>>> what will make entire process much more easer for all of
> >>>>>>>>>> us:
> >>>>>>>>>>
> >>>>>>>>>> 1. discuss new feature on the mailing list
> >>>>>>>>>> (requirements/constraints/etc.)
> >>>>>>>>>> 2. summarise feature details in feature-doc
> >>>>>>>>>> 3. send feature-doc to review to:
> >>>>>>>>>>      3.1 ML (engine-devel)
> >>>>>>>>>>      3.2 MG (mailing group of maintainers of the relevant
> >>>>>>>>>>      layers)
> >>>>>>>>>> 4. after feature-doc is accepted,
> >>>>>>>>>>      4.1 implement the feature
> >>>>>>>>>>      4.2 send it to gerrit for review to:
> >>>>>>>>>>          4.2.1 lead maintainer/s (they will
> >>>>>>>>>>          review/delegate
> >>>>>>>>>>          it)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> NOTE 3.2, 4.2.1 will require defining MGs such as:
> >>>>>>>>>>         - engine-devel-core
> >>>>>>>>>>         - engine-devel-ui
> >>>>>>>>>>         - engine-devel-api
> >>>>>>>>>>         - engine-devel-sdk
> >>>>>>>>>>         - engine-devel-cli
> >>>>>>>>>>         - engine-devel-vdsm ...
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Thoughts?
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> i thought this is what we have the arch mailing list for,
> >>>>>>>>> since
> >>>>>>>>> any
> >>>>>>>>> feature is going to cut through multiple layers/components,
> >>>>>>>>> unless
> >>>>>>>>> they
> >>>>>>>>> are very specific, they should be sent to arch, and all
> >>>>>>>>> maintainers
> >>>>>>>>> should follow arch.
> >>>>>>>>
> >>>>>>>> What is missing is upstream bugzilla.
> >>>>>>>>
> >>>>>>>> A feature, after initiate stage should be represented with a
> >>>>>>>> bug.
> >>>>>>>> The bug should be assigned to the right designated
> >>>>>>>> milestone.
> >>>>>>>> All document references (including versions) should be
> >>>>>>>> attached
> >>>>>>>> or
> >>>>>>>> referred by the bug.
> >>>>>>>> Dependency between features can be established using bug
> >>>>>>>> dependencies.
> >>>>>>>> Status can be acquired from buzilla at any time, progress
> >>>>>>>> reports
> >>>>>>>> should be input into bugzilla.
> >>>>>>>> Contact details for the feature can be acquired too,
> >>>>>>>> relevant
> >>>>>>>> and
> >>>>>>>> interested parties can be CCed explicitly.
> >>>>>>>> I guess I can add more benefits.
> >>>>>>>>
> >>>>>>>> Mailing list is good for idea initiation, but not for
> >>>>>>>> lifecycle
> >>>>>>>> management, nor for people to join at implementation phase
> >>>>>>>> and
> >>>>>>>> understand why, how and when.
> >>>>>>>>
> >>>>>>>> Having upstream bugzilla will also help us plan ahead, and
> >>>>>>>> manage
> >>>>>>>> the break the project into smaller components, to assign
> >>>>>>>> core
> >>>>>>>> developers for each.
> >>>>>>>
> >>>>>>> it still won't help tracking all relevant maintainers from
> >>>>>>> rest
> >>>>>>> api, ui, engine, vdsm saw the bug. we have multiple
> >>>>>>> components.
> >>>>>>> arch is mostly to cover cross component issues, such as
> >>>>>>> features.
> >>>>>>
> >>>>>> this is why i'm suggesting creating mailing-groups peer
> >>>>>> component,
> >>>>>> feature owner knows what layers are involved and should be
> >>>>>> responsible to CC
> >>>>>> relevant MGs in both BZ & thread.
> >>>>>
> >>>>> mailing group is something nobody can manage.
> >>>>
> >>>> why?, we do not have many layers in the system, creating:
> >>>> "engine{core, db, network, storage/gluster}, api, ui, sdk, cli,
> >>>> vdsm,
> >>>> node"
> >>>> mailing groups will do the trick.
> >>>
> >>> Because in open source you don't know who is actually out there.
> >>
> >> this is not about of including everyone in mailing-group, but
> >> maintainers,
> >> and you always know who maintainers are.
> > 
> > I am sorry, but this is not "open" discussion.
> > You *DO NOT* know who is interested, there can be other people that
> > are highly valuable for a discussion and you are going to loose
> > them.
> 
> how possible you will loose them? adding mailing-group/s of
> maintainers does *not* mean
> excluding the mailing list itself ...

So why do you need the group?
I don't understand... maybe provide another projects that works this way?

> 
> anyone that is interested in discussion can still follow it.
> 
> > 
> > Maintainer is not someone who is more important the contributer, he
> > just has commit access.
> 
> i do agree about maintainers not being "more important" than
> contributor, but for sure maintainers have more
> experience with the projects and planned stuff, so they can drive
> features to the right direction ...
> 
> > 
> >>
> >>> This is very different from proprietary.
> >>> Also, features can be cross component / cross products / cross
> >>> projects.
> >>
> >> the one who implementing the feature knows what layers he is
> >> touching,
> >> and can trigger relevant mailing-groups (this way feature owner
> >> does
> >> not
> >> have to be aware of actual layer/s maintainers)
> > 
> > Again, you lose one of the most important open source benefit,
> > allowing everyone to participate and contribute.
> > 
> > If you like, a message subject convention will be more productive,
> > "maintainer" may set trigger for a specific prefix, and there can
> > be a documentation of this convention for features and other uses.
> > 
> > All discussions should be made in public, this is burden but also
> > opportunity.
> 
> Alon,
> 
> i think you missed the point, read my answer ^ above, + entire thread
> was started,
> to make sure maintainers explicitly got involved in to design of the
> features and
> not changing it when the code is written (what for sure will make
> contributors frustrated)
> 
> > 
> >>>
> >>> You can have default CC in bugzilla for the people you do know,
> >>> but
> >>> you must enable other people to subscribe.
> >>
> >> i'm not talking about bagzila, but the standard process of
> >> introducing new
> >> feature, - e.g discussion cannot take place in the bugzila (it's a
> >> tool to summarise the feature, etc.),
> >> but i'd like to male sure all relevant maintainers get involved in
> >> feature design/discussion
> >> before it get written and not only when they have to review the
> >> code,
> >> and mailing-groups is a
> >> right tool for that in my view.
> >>
> >>>
> >>>>
> >>>>> you can consider a group as all people who are CC to a bug.
> >>>>
> >>>> CC'ing people manually can't guarantee that all relevant people
> >>>> will
> >>>> be included in the email (thread).
> >>>>
> >>>>>
> >>>>>>
> >>>>>>> (or we could create a feature mailing list).
> >>>>>>
> >>>>>> ML/MG peer feature is too much i think.
> >>>>>>
> >>>>>>>
> >>>>>>> but i do agree we could use bugzilla for tracking features
> >>>>>>> per
> >>>>>>> version, if wiki isn't tracking this correctly.
> >>>>>>
> >>>>>> +1
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Michael Pasternak
> >>>>>> RedHat, ENG-Virtualization R&D
> >>>>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Michael Pasternak
> >>>> RedHat, ENG-Virtualization R&D
> >>>>
> >>
> >>
> >> --
> >>
> >> Michael Pasternak
> >> RedHat, ENG-Virtualization R&D
> >>
> 
> 
> --
> 
> Michael Pasternak
> RedHat, ENG-Virtualization R&D
> 


From mpastern at redhat.com  Sun Dec 23 10:27:25 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Sun, 23 Dec 2012 12:27:25 +0200
Subject: [Engine-devel] [SUGGESTION] Defining a process for the new
 feature discussion/implementation.
In-Reply-To: <1935203280.6577398.1356257814058.JavaMail.root@redhat.com>
References: <1935203280.6577398.1356257814058.JavaMail.root@redhat.com>
Message-ID: <50D6DC8D.6020901@redhat.com>

On 12/23/2012 12:16 PM, Alon Bar-Lev wrote:
> 
> ----- Original Message -----
>> > From: "Michael Pasternak" <mpastern at redhat.com>
>> > To: "Alon Bar-Lev" <alonbl at redhat.com>
>> > Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org, "Itamar Heim" <iheim at redhat.com>
>> > Sent: Sunday, December 23, 2012 12:01:25 PM
>> > Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
>> > 
>> > On 12/23/2012 11:39 AM, Alon Bar-Lev wrote:
>>> > > 
>>> > > 
>>> > > ----- Original Message -----
>>>> > >> From: "Michael Pasternak" <mpastern at redhat.com>
>>>> > >> To: "Alon Bar-Lev" <alonbl at redhat.com>
>>>> > >> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org,
>>>> > >> "Itamar Heim" <iheim at redhat.com>
>>>> > >> Sent: Sunday, December 23, 2012 11:28:13 AM
>>>> > >> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
>>>> > >> the new feature discussion/implementation.
>>>> > >>
>>>> > >> On 12/23/2012 11:07 AM, Alon Bar-Lev wrote:
>>>>> > >>>
>>>>> > >>>
>>>>> > >>> ----- Original Message -----
>>>>>> > >>>> From: "Michael Pasternak" <mpastern at redhat.com>
>>>>>> > >>>> To: "Alon Bar-Lev" <alonbl at redhat.com>
>>>>>> > >>>> Cc: "engine-devel" <engine-devel at ovirt.org>, arch at ovirt.org,
>>>>>> > >>>> "Itamar Heim" <iheim at redhat.com>
>>>>>> > >>>> Sent: Sunday, December 23, 2012 8:25:23 AM
>>>>>> > >>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process for
>>>>>> > >>>> the new feature discussion/implementation.
>>>>>> > >>>>
>>>>>> > >>>> On 12/21/2012 01:57 PM, Alon Bar-Lev wrote:
>>>>>>> > >>>>>
>>>>>>> > >>>>>
>>>>>>> > >>>>> ----- Original Message -----
>>>>>>>> > >>>>>> From: "Michael Pasternak" <mpastern at redhat.com>
>>>>>>>> > >>>>>> To: "Itamar Heim" <iheim at redhat.com>
>>>>>>>> > >>>>>> Cc: "Alon Bar-Lev" <alonbl at redhat.com>, "engine-devel"
>>>>>>>> > >>>>>> <engine-devel at ovirt.org>, arch at ovirt.org
>>>>>>>> > >>>>>> Sent: Friday, December 21, 2012 12:44:51 PM
>>>>>>>> > >>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process
>>>>>>>> > >>>>>> for
>>>>>>>> > >>>>>> the new feature discussion/implementation.
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>> On 12/21/2012 12:15 AM, Itamar Heim wrote:
>>>>>>>>> > >>>>>>> On 12/21/2012 12:12 AM, Alon Bar-Lev wrote:
>>>>>>>>>> > >>>>>>>>
>>>>>>>>>> > >>>>>>>>
>>>>>>>>>> > >>>>>>>> ----- Original Message -----
>>>>>>>>>>> > >>>>>>>>> From: "Itamar Heim" <iheim at redhat.com>
>>>>>>>>>>> > >>>>>>>>> To: "Michael Pasternak" <mpastern at redhat.com>
>>>>>>>>>>> > >>>>>>>>> Cc: "engine-devel" <engine-devel at ovirt.org>
>>>>>>>>>>> > >>>>>>>>> Sent: Thursday, December 20, 2012 11:42:37 PM
>>>>>>>>>>> > >>>>>>>>> Subject: Re: [Engine-devel] [SUGGESTION] Defining a process
>>>>>>>>>>> > >>>>>>>>> for
>>>>>>>>>>> > >>>>>>>>> the new feature discussion/implementation.
>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>> > >>>>>>>>> On 12/19/2012 03:08 PM, Michael Pasternak wrote:
>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>> > >>>>>>>>>> Hi All,
>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>> > >>>>>>>>>> In many cases OSS maintainers not always can be in the
>>>>>>>>>>>> > >>>>>>>>>> loop
>>>>>>>>>>>> > >>>>>>>>>> of
>>>>>>>>>>>> > >>>>>>>>>> different threads what may
>>>>>>>>>>>> > >>>>>>>>>> cause them missing important decisions being taken,
>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>> > >>>>>>>>>> As result later on during reviews of the patches they're
>>>>>>>>>>>> > >>>>>>>>>> not
>>>>>>>>>>>> > >>>>>>>>>> accepting (already implemented)
>>>>>>>>>>>> > >>>>>>>>>> features, what is causing not once for feature to be
>>>>>>>>>>>> > >>>>>>>>>> re-designed
>>>>>>>>>>>> > >>>>>>>>>> and/or delayed, what is wrong
>>>>>>>>>>>> > >>>>>>>>>> from the development cycle PoV.
>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>> > >>>>>>>>>> Therefore I'd like to suggest establishing dev-rules for
>>>>>>>>>>>> > >>>>>>>>>> the
>>>>>>>>>>>> > >>>>>>>>>> new
>>>>>>>>>>>> > >>>>>>>>>> feature implementation,
>>>>>>>>>>>> > >>>>>>>>>> what will make entire process much more easer for all of
>>>>>>>>>>>> > >>>>>>>>>> us:
>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>> > >>>>>>>>>> 1. discuss new feature on the mailing list
>>>>>>>>>>>> > >>>>>>>>>> (requirements/constraints/etc.)
>>>>>>>>>>>> > >>>>>>>>>> 2. summarise feature details in feature-doc
>>>>>>>>>>>> > >>>>>>>>>> 3. send feature-doc to review to:
>>>>>>>>>>>> > >>>>>>>>>>      3.1 ML (engine-devel)
>>>>>>>>>>>> > >>>>>>>>>>      3.2 MG (mailing group of maintainers of the relevant
>>>>>>>>>>>> > >>>>>>>>>>      layers)
>>>>>>>>>>>> > >>>>>>>>>> 4. after feature-doc is accepted,
>>>>>>>>>>>> > >>>>>>>>>>      4.1 implement the feature
>>>>>>>>>>>> > >>>>>>>>>>      4.2 send it to gerrit for review to:
>>>>>>>>>>>> > >>>>>>>>>>          4.2.1 lead maintainer/s (they will
>>>>>>>>>>>> > >>>>>>>>>>          review/delegate
>>>>>>>>>>>> > >>>>>>>>>>          it)
>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>> > >>>>>>>>>> NOTE 3.2, 4.2.1 will require defining MGs such as:
>>>>>>>>>>>> > >>>>>>>>>>         - engine-devel-core
>>>>>>>>>>>> > >>>>>>>>>>         - engine-devel-ui
>>>>>>>>>>>> > >>>>>>>>>>         - engine-devel-api
>>>>>>>>>>>> > >>>>>>>>>>         - engine-devel-sdk
>>>>>>>>>>>> > >>>>>>>>>>         - engine-devel-cli
>>>>>>>>>>>> > >>>>>>>>>>         - engine-devel-vdsm ...
>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>>> > >>>>>>>>>> Thoughts?
>>>>>>>>>>>> > >>>>>>>>>>
>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>> > >>>>>>>>> i thought this is what we have the arch mailing list for,
>>>>>>>>>>> > >>>>>>>>> since
>>>>>>>>>>> > >>>>>>>>> any
>>>>>>>>>>> > >>>>>>>>> feature is going to cut through multiple layers/components,
>>>>>>>>>>> > >>>>>>>>> unless
>>>>>>>>>>> > >>>>>>>>> they
>>>>>>>>>>> > >>>>>>>>> are very specific, they should be sent to arch, and all
>>>>>>>>>>> > >>>>>>>>> maintainers
>>>>>>>>>>> > >>>>>>>>> should follow arch.
>>>>>>>>>> > >>>>>>>>
>>>>>>>>>> > >>>>>>>> What is missing is upstream bugzilla.
>>>>>>>>>> > >>>>>>>>
>>>>>>>>>> > >>>>>>>> A feature, after initiate stage should be represented with a
>>>>>>>>>> > >>>>>>>> bug.
>>>>>>>>>> > >>>>>>>> The bug should be assigned to the right designated
>>>>>>>>>> > >>>>>>>> milestone.
>>>>>>>>>> > >>>>>>>> All document references (including versions) should be
>>>>>>>>>> > >>>>>>>> attached
>>>>>>>>>> > >>>>>>>> or
>>>>>>>>>> > >>>>>>>> referred by the bug.
>>>>>>>>>> > >>>>>>>> Dependency between features can be established using bug
>>>>>>>>>> > >>>>>>>> dependencies.
>>>>>>>>>> > >>>>>>>> Status can be acquired from buzilla at any time, progress
>>>>>>>>>> > >>>>>>>> reports
>>>>>>>>>> > >>>>>>>> should be input into bugzilla.
>>>>>>>>>> > >>>>>>>> Contact details for the feature can be acquired too,
>>>>>>>>>> > >>>>>>>> relevant
>>>>>>>>>> > >>>>>>>> and
>>>>>>>>>> > >>>>>>>> interested parties can be CCed explicitly.
>>>>>>>>>> > >>>>>>>> I guess I can add more benefits.
>>>>>>>>>> > >>>>>>>>
>>>>>>>>>> > >>>>>>>> Mailing list is good for idea initiation, but not for
>>>>>>>>>> > >>>>>>>> lifecycle
>>>>>>>>>> > >>>>>>>> management, nor for people to join at implementation phase
>>>>>>>>>> > >>>>>>>> and
>>>>>>>>>> > >>>>>>>> understand why, how and when.
>>>>>>>>>> > >>>>>>>>
>>>>>>>>>> > >>>>>>>> Having upstream bugzilla will also help us plan ahead, and
>>>>>>>>>> > >>>>>>>> manage
>>>>>>>>>> > >>>>>>>> the break the project into smaller components, to assign
>>>>>>>>>> > >>>>>>>> core
>>>>>>>>>> > >>>>>>>> developers for each.
>>>>>>>>> > >>>>>>>
>>>>>>>>> > >>>>>>> it still won't help tracking all relevant maintainers from
>>>>>>>>> > >>>>>>> rest
>>>>>>>>> > >>>>>>> api, ui, engine, vdsm saw the bug. we have multiple
>>>>>>>>> > >>>>>>> components.
>>>>>>>>> > >>>>>>> arch is mostly to cover cross component issues, such as
>>>>>>>>> > >>>>>>> features.
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>> this is why i'm suggesting creating mailing-groups peer
>>>>>>>> > >>>>>> component,
>>>>>>>> > >>>>>> feature owner knows what layers are involved and should be
>>>>>>>> > >>>>>> responsible to CC
>>>>>>>> > >>>>>> relevant MGs in both BZ & thread.
>>>>>>> > >>>>>
>>>>>>> > >>>>> mailing group is something nobody can manage.
>>>>>> > >>>>
>>>>>> > >>>> why?, we do not have many layers in the system, creating:
>>>>>> > >>>> "engine{core, db, network, storage/gluster}, api, ui, sdk, cli,
>>>>>> > >>>> vdsm,
>>>>>> > >>>> node"
>>>>>> > >>>> mailing groups will do the trick.
>>>>> > >>>
>>>>> > >>> Because in open source you don't know who is actually out there.
>>>> > >>
>>>> > >> this is not about of including everyone in mailing-group, but
>>>> > >> maintainers,
>>>> > >> and you always know who maintainers are.
>>> > > 
>>> > > I am sorry, but this is not "open" discussion.
>>> > > You *DO NOT* know who is interested, there can be other people that
>>> > > are highly valuable for a discussion and you are going to loose
>>> > > them.
>> > 
>> > how possible you will loose them? adding mailing-group/s of
>> > maintainers does *not* mean
>> > excluding the mailing list itself ...
> So why do you need the group?

please see very first email of this tread, you will see there that suggestion
works like this:

1. discuss new feature on the mailing list (requirements/constraints/etc.)
2. summarise feature details in feature-doc
3. send feature-doc to review to:
   3.1 ML (engine-devel)
   3.2 MG (mailing group of maintainers of the relevant layers)
4. after feature-doc is accepted,
   4.1 implement the feature
   4.2 send it to gerrit for review to:
       4.2.1 lead maintainer/s (they will review/delegate it)

(3.1 answer your concerns from the previous email)

-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From shuming at linux.vnet.ibm.com  Sun Dec 23 14:00:10 2012
From: shuming at linux.vnet.ibm.com (Shu Ming)
Date: Sun, 23 Dec 2012 22:00:10 +0800
Subject: [Engine-devel] FW: Querying for and registering unknown disk
 images on a storage domain
In-Reply-To: <D290AD8432118048947689BA3AE8A9B3094E3FDC@SACEXCMBX04-PRD.hq.netapp.com>
References: <D290AD8432118048947689BA3AE8A9B3094E3FDC@SACEXCMBX04-PRD.hq.netapp.com>
Message-ID: <50D70E6A.9060407@linux.vnet.ibm.com>

2012-12-20 23:18, Morrissey, Christopher:
>
> Hi All,
>
> I've been working on a bit of functionality for the engine that will 
> allow a user to query a domain for new disk images 
> (GetUnregisteredImagesQuery) for which the engine was previously 
> unaware and a separate command to register those images 
> (ImportImageCommand). These commands will be exposed through the REST API.
>
> This functionality is needed as we are developing an extension/plugin 
> to oVirt that will allow a NetApp storage controller to handle cloning 
> the actual disks outside of oVirt and need to import them once they 
> are cloned. We'll be using other existing APIs to attach the disk to 
> the necessary VM once the disk is cloned. On the NetApp side, we'll 
> ensure the disk is coalesced before cloning so as to avoid the issues 
> of registering snapshots.
>
  I am just curious about how the third party tool like NetApp to make 
sure the disk of a running VM coalesced before cloning? By an agent in 
the VM to flush file-system cache out to the disk?

> GetUnregisteredImagesQuery will be accessible through the disks 
> resource collection on a storage domain. A "disks" resource collection 
> does not yet exist and will need to be added. To access the 
> unregistered images, a parameter (maybe "unregistered=true") would be 
> passed. So the path to "GET" the unregistered disk images on a domain 
> would be something like 
> /api/storagedomains/f0dbcb33-69d3-4899-9352-8e8a02f01bbd/disks?unregistered=true. 
> This will return a list of disk images that can be each used as input 
> to the ImportImageCommand to get them added to oVirt.
>
> ImportImageCommand will be accessible through "POST"ing a disk to 
> /api/disks?import=true. The disk will be added to the oVirt DB based 
> on the information supplied and afterward would be available to attach 
> to a VM.
>
> When querying for unregistered disk images, the 
> GetUnregisteredImagesQuery command will use the getImagesList() VDSM 
> command. Currently this only reports the GUIDs of all disk images in a 
> domain. I had been using the getVolumesList() and getVolumeInfo() VDSM 
> commands to fill in the information so that valid disk image objects 
> could be registered in oVirt. It seems these two functions are set to 
> be removed since they are too invasive into the internal VDSM 
> workings. The VDSM team will need to either return more information 
> about each disk as part of the getImagesList() function or add a new 
> function getImageInfo() that will give the same information for a 
> given image GUID.
>

Here is the project proposal for floating disk in oVirt.  I think 
unregistered images are also floating disks.
http://www.ovirt.org/Features/DetailedFloatingDisk

> Note that much of this work had originally been submitted under patch 
> http://gerrit.ovirt.org/#/c/9603/. After several reviews it was found 
> to be lacking in its design and was using deprecated APIs that did not 
> yet have replacements. I'm reworking the code now to conform to this 
> design and asking for further input from the VDSM, core, and restapi 
> teams to ensure we can get this done quickly and correctly as it is 
> needed for the 3.2 release.
>
> -Chris
>
> *Chris Morrissey*
>
> Software Engineer
>
> NetApp Inc.
>
> 919.476.4428
>
>
>
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel


-- 
---
?? Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming at cn.ibm.com or shuming at linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20121223/5f2112a5/attachment.html>

From yzaslavs at redhat.com  Sun Dec 23 14:15:12 2012
From: yzaslavs at redhat.com (Yair Zaslavsky)
Date: Sun, 23 Dec 2012 09:15:12 -0500 (EST)
Subject: [Engine-devel] Problems with ovirt gerrit?
In-Reply-To: <230940902.50969961.1356272068467.JavaMail.root@redhat.com>
Message-ID: <215490274.50969979.1356272112582.JavaMail.root@redhat.com>

Hi,
I see that git fetch -v (or git fetch) get stuck.
I saw Maor and Yaniv Bronheim suffer from same issue
Can someone please be of aid and fix this?

Yair



From iheim at redhat.com  Sun Dec 23 16:49:05 2012
From: iheim at redhat.com (Itamar Heim)
Date: Sun, 23 Dec 2012 18:49:05 +0200
Subject: [Engine-devel] Problems with ovirt gerrit?
In-Reply-To: <215490274.50969979.1356272112582.JavaMail.root@redhat.com>
References: <215490274.50969979.1356272112582.JavaMail.root@redhat.com>
Message-ID: <50D73601.1040905@redhat.com>

On 12/23/2012 04:15 PM, Yair Zaslavsky wrote:
> Hi,
> I see that git fetch -v (or git fetch) get stuck.
> I saw Maor and Yaniv Bronheim suffer from same issue
> Can someone please be of aid and fix this?
>
> Yair
>
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>

infra mailing list is good enough, though putting me in cc directly will 
help seeing this earlier (by me. other infra team members can restart 
the service as first first aid).
did this just now, even though i didn't see any issue in gerrit / worked 
ok for me.
is it better now?



From iheim at redhat.com  Sun Dec 23 16:57:52 2012
From: iheim at redhat.com (Itamar Heim)
Date: Sun, 23 Dec 2012 18:57:52 +0200
Subject: [Engine-devel] [vdsm] FW: Querying for and registering unknown
 disk images on a storage domain
In-Reply-To: <50D70E6A.9060407@linux.vnet.ibm.com>
References: <D290AD8432118048947689BA3AE8A9B3094E3FDC@SACEXCMBX04-PRD.hq.netapp.com>
	<50D70E6A.9060407@linux.vnet.ibm.com>
Message-ID: <50D73810.80605@redhat.com>

On 12/23/2012 04:00 PM, Shu Ming wrote:
> 2012-12-20 23:18, Morrissey, Christopher:
>>
>> Hi All,
>>
>> I?ve been working on a bit of functionality for the engine that will
>> allow a user to query a domain for new disk images
>> (GetUnregisteredImagesQuery) for which the engine was previously
>> unaware and a separate command to register those images
>> (ImportImageCommand). These commands will be exposed through the REST API.
>>
>> This functionality is needed as we are developing an extension/plugin
>> to oVirt that will allow a NetApp storage controller to handle cloning
>> the actual disks outside of oVirt and need to import them once they
>> are cloned. We?ll be using other existing APIs to attach the disk to
>> the necessary VM once the disk is cloned. On the NetApp side, we?ll
>> ensure the disk is coalesced before cloning so as to avoid the issues
>> of registering snapshots.
>>
>   I am just curious about how the third party tool like NetApp to make
> sure the disk of a running VM coalesced before cloning? By an agent in
> the VM to flush file-system cache out to the disk?

I'd expect either a livesnapshot before, or doing this on a VM which is 
down.

>
>> GetUnregisteredImagesQuery will be accessible through the disks
>> resource collection on a storage domain. A ?disks? resource collection
>> does not yet exist and will need to be added. To access the
>> unregistered images, a parameter (maybe ?unregistered=true?) would be
>> passed. So the path to ?GET? the unregistered disk images on a domain
>> would be something like
>> /api/storagedomains/f0dbcb33-69d3-4899-9352-8e8a02f01bbd/disks?unregistered=true.
>> This will return a list of disk images that can be each used as input
>> to the ImportImageCommand to get them added to oVirt.
>>
>> ImportImageCommand will be accessible through ?POST?ing a disk to
>> /api/disks?import=true. The disk will be added to the oVirt DB based
>> on the information supplied and afterward would be available to attach
>> to a VM.
>>
>> When querying for unregistered disk images, the
>> GetUnregisteredImagesQuery command will use the getImagesList() VDSM
>> command. Currently this only reports the GUIDs of all disk images in a
>> domain. I had been using the getVolumesList() and getVolumeInfo() VDSM
>> commands to fill in the information so that valid disk image objects
>> could be registered in oVirt. It seems these two functions are set to
>> be removed since they are too invasive into the internal VDSM
>> workings. The VDSM team will need to either return more information
>> about each disk as part of the getImagesList() function or add a new
>> function getImageInfo() that will give the same information for a
>> given image GUID.
>>
>
> Here is the project proposal for floating disk in oVirt.  I think
> unregistered images are also floating disks.
> http://www.ovirt.org/Features/DetailedFloatingDisk

floating disks are disks the engine is aware of, but not associated to 
any VM.
the scan domain feature is to add "orphan" disks - disks on the storage 
the engine isn't aware of to begin with.


From vered at redhat.com  Mon Dec 24 06:39:41 2012
From: vered at redhat.com (Vered Volansky)
Date: Mon, 24 Dec 2012 01:39:41 -0500 (EST)
Subject: [Engine-devel] Read Only Disk Feature
In-Reply-To: <449238800.57765711.1356331077051.JavaMail.root@redhat.com>
Message-ID: <1739535710.57765778.1356331181113.JavaMail.root@redhat.com>

Hi all,

This is the wiki page for the new read-only disks feature in the engine:

http://www.ovirt.org/Features/Read_Only_Disk

Comments are welcome.

Thanks,
Vered


From snmishra at linux.vnet.ibm.com  Tue Dec 25 22:32:36 2012
From: snmishra at linux.vnet.ibm.com (snmishra at linux.vnet.ibm.com)
Date: Tue, 25 Dec 2012 14:32:36 -0800
Subject: [Engine-devel] Patches awaiting review.
Message-ID: <20121225143236.Horde.YVMvZJir309Q2imEC4Dh1QA@imap.linux.ibm.com>


Following patches are waiting for review -

http://gerrit.ovirt.org/#/c/10072/
http://gerrit.ovirt.org/#/c/10085/
http://gerrit.ovirt.org/#/c/10086/
http://gerrit.ovirt.org/#/c/10087/
http://gerrit.ovirt.org/#/c/10042/

Thanks
Sharad Mishra



From snmishra at linux.vnet.ibm.com  Wed Dec 26 18:24:30 2012
From: snmishra at linux.vnet.ibm.com (snmishra at linux.vnet.ibm.com)
Date: Wed, 26 Dec 2012 10:24:30 -0800
Subject: [Engine-devel] lastest source from gerrit fails to complete tests
	successfully.
Message-ID: <20121226102430.Horde.OHdX7pir309Q20De6FSQwZA@imap.linux.ibm.com>

Hi,

    I am seeing following test fail -

Running org.ovirt.engine.api.restapi.types.VmMapperTest
Tests run: 44, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.136  
sec <<< FAILURE!

Results :

Failed tests:    
testRoundtrip(org.ovirt.engine.api.restapi.types.TemplateMapperTest):  
expected:<true> but was:<false>
   testRoundtrip(org.ovirt.engine.api.restapi.types.VmMapperTest):  
expected:<true> but was:<false>

Tests run: 101, Failures: 2, Errors: 0, Skipped: 0


***********

started seeing this error in last 24 hours. Was working yesterday.

Sharad Mishra



From ahadas at redhat.com  Wed Dec 26 18:32:24 2012
From: ahadas at redhat.com (Arik Hadas)
Date: Wed, 26 Dec 2012 13:32:24 -0500 (EST)
Subject: [Engine-devel] lastest source from gerrit fails to complete
 tests	successfully.
In-Reply-To: <20121226102430.Horde.OHdX7pir309Q20De6FSQwZA@imap.linux.ibm.com>
Message-ID: <750784861.13011036.1356546744670.JavaMail.root@redhat.com>

My bad,
will push a fix for the test ASAP..

----- Original Message -----
From: snmishra at linux.vnet.ibm.com
To: engine-devel at ovirt.org
Sent: Wednesday, December 26, 2012 8:24:30 PM
Subject: [Engine-devel] lastest source from gerrit fails to complete tests	successfully.

Hi,

    I am seeing following test fail -

Running org.ovirt.engine.api.restapi.types.VmMapperTest
Tests run: 44, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.136  
sec <<< FAILURE!

Results :

Failed tests:    
testRoundtrip(org.ovirt.engine.api.restapi.types.TemplateMapperTest):  
expected:<true> but was:<false>
   testRoundtrip(org.ovirt.engine.api.restapi.types.VmMapperTest):  
expected:<true> but was:<false>

Tests run: 101, Failures: 2, Errors: 0, Skipped: 0


***********

started seeing this error in last 24 hours. Was working yesterday.

Sharad Mishra

_______________________________________________
Engine-devel mailing list
Engine-devel at ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


From eedri at redhat.com  Thu Dec 27 08:05:30 2012
From: eedri at redhat.com (Eyal Edri)
Date: Thu, 27 Dec 2012 03:05:30 -0500 (EST)
Subject: [Engine-devel] [oVirt Jenkins] ovirt_engine_unit_tests - Build
 # 2971 - Still Unstable!
In-Reply-To: <624640345.5479.1356594801799.JavaMail.jenkins@jenkins.ovirt.org>
Message-ID: <2137910072.801976.1356595530556.JavaMail.root@redhat.com>

fyi,

unit tests are failing from yesterday, please check.

Eyal Edri.
oVirt infra team.

----- Original Message -----
> From: "Jenkins oVirt Server" <jenkins at ovirt.org>
> To: eedri at redhat.com, engine-patches at ovirt.org, oliel at redhat.com, yzaslavs at redhat.com, mkolesni at redhat.com
> Sent: Thursday, December 27, 2012 9:53:20 AM
> Subject: [oVirt Jenkins] ovirt_engine_unit_tests - Build # 2971 - Still Unstable!
> 
> Project: http://jenkins.ovirt.org/job/ovirt_engine_unit_tests/
> Build: http://jenkins.ovirt.org/job/ovirt_engine_unit_tests/2971/
> Build Number: 2971
> Build Status:  Still Unstable
> Triggered By: Started by upstream project "ovirt_engine" build number
> 3,735
> 
> -------------------------------------
> Changes Since Last Success:
> -------------------------------------
> Changes for Build #2969
> 
> Changes for Build #2970
> [emesika] core:fixing GeneralDbDAOTest
> 
> 
> Changes for Build #2971
> [mkolesni] engine: Add network command packages to factory
> 
> [mkolesni] engine: Move DC network commands to their package
> 
> [mkolesni] engine: Moved VM NIC commands to their package
> 
> [mkolesni] engine: Move template NIC commands to their package
> 
> [mkolesni] engine: Extract SetNetworkStatus method
> 
> [mkolesni] engine: Move cluster network commands to their package
> 
> [mkolesni] engine: Move host network commands to their package
> 
> 
> 
> 
> -----------------
> Failed Tests:
> -----------------
> 2 tests failed.
> REGRESSION:
>  org.ovirt.engine.api.restapi.types.VmMapperTest.testRoundtrip
> 
> Error Message:
> expected:<true> but was:<false>
> 
> Stack Trace:
> java.lang.AssertionError: expected:<true> but was:<false>
> 	at org.junit.Assert.fail(Assert.java:93)
> 	at org.junit.Assert.failNotEquals(Assert.java:647)
> 	at org.junit.Assert.assertEquals(Assert.java:128)
> 	at org.junit.Assert.assertEquals(Assert.java:147)
> 	at
> 	org.ovirt.engine.api.restapi.types.VmMapperTest.verify(VmMapperTest.java:109)
> 	at
> 	org.ovirt.engine.api.restapi.types.VmMapperTest.verify(VmMapperTest.java:26)
> 	at
> 	org.ovirt.engine.api.restapi.types.AbstractInvertibleMappingTest.testRoundtrip(AbstractInvertibleMappingTest.java:57)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at
> 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at
> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at
> 	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> 	at
> 	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> 	at
> 	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> 	at
> 	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> 	at
> 	org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
> 	at org.junit.rules.RunRules.evaluate(RunRules.java:18)
> 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> 	at
> 	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> 	at
> 	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> 	at
> 	org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> 	at
> 	org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
> 	at
> 	org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at
> 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at
> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at
> 	org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
> 	at
> 	org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
> 	at
> 	org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
> 	at
> 	org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
> 	at
> 	org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
> 
> 
> FAILED:
>  org.ovirt.engine.api.restapi.types.TemplateMapperTest.testRoundtrip
> 
> Error Message:
> expected:<true> but was:<false>
> 
> Stack Trace:
> java.lang.AssertionError: expected:<true> but was:<false>
> 	at org.junit.Assert.fail(Assert.java:93)
> 	at org.junit.Assert.failNotEquals(Assert.java:647)
> 	at org.junit.Assert.assertEquals(Assert.java:128)
> 	at org.junit.Assert.assertEquals(Assert.java:147)
> 	at
> 	org.ovirt.engine.api.restapi.types.TemplateMapperTest.verify(TemplateMapperTest.java:71)
> 	at
> 	org.ovirt.engine.api.restapi.types.TemplateMapperTest.verify(TemplateMapperTest.java:12)
> 	at
> 	org.ovirt.engine.api.restapi.types.AbstractInvertibleMappingTest.testRoundtrip(AbstractInvertibleMappingTest.java:57)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at
> 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at
> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at
> 	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> 	at
> 	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> 	at
> 	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> 	at
> 	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> 	at
> 	org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
> 	at org.junit.rules.RunRules.evaluate(RunRules.java:18)
> 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> 	at
> 	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> 	at
> 	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> 	at
> 	org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> 	at
> 	org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
> 	at
> 	org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at
> 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at
> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at
> 	org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
> 	at
> 	org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
> 	at
> 	org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
> 	at
> 	org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
> 	at
> 	org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
> 
> 
>  
> 
> ------------------
> Build Log:
> ------------------
> [...truncated 4214 lines...]
> [INFO] [INFO] Copying
> artifact[jar:org.ovirt.engine.core:compat:3.2.0]
> to[lib/engine-compat.jar]
> [INFO] Copying artifact[jar:org.ovirt.engine.core:dal:3.2.0]
> to[lib/engine-dal.jar]
> [INFO] Copying artifact[jar:org.ovirt.engine.core:utils:3.2.0]
> to[lib/engine-utils.jar]
> [INFO] Copying
> artifact[jar:org.ovirt.engine.core:engineencryptutils:3.2.0]
> to[lib/engine-encryptutils.jar]
> 
> [INFO] Copying artifact[jar:org.ovirt.engine.core:vdsbroker:3.2.0]
> to[lib/engine-vdsbroker.jar]
> [INFO] Copying artifact[war:org.ovirt.engine.core:root-war:3.2.0]
> to[root.war] (unpacked)
> [INFO] Copying artifact[war:org.ovirt.engine.ui:rmw-war:3.2.0]
> to[ovirtengineweb.war] (unpacked)
> [INFO] Copying
> artifact[war:org.ovirt.engine.api:restapi-webapp:3.2.0]
> to[restapi.war] (unpacked)
> [INFO] Copying artifact[war:org.ovirt.engine.ui:userportal:3.2.0]
> to[userportal.war] (unpacked)
> [INFO] --- maven-ear-plugin:2.6:ear (default-ear) @ engine-server-ear
> ---
> [INFO] Copying artifact[war:org.ovirt.engine.ui:webadmin:3.2.0]
> to[webadmin.war] (unpacked)
> [INFO] Copying artifact[ejb:org.ovirt.engine.ui:genericapi:3.2.0]
> to[engine-genericapi.jar] (unpacked)
> [INFO] Copying artifact[ejb:org.ovirt.engine.core:scheduler:3.2.0]
> to[engine-scheduler.jar] (unpacked)
> [INFO] Copying artifact[ejb:org.ovirt.engine.core:bll:3.2.0]
> to[engine-bll.jar] (unpacked)
> [INFO] Copying artifact[jar:commons-codec:commons-codec:1.4]
> to[lib/commons-codec-1.4.jar]
> [INFO] Copying
> artifact[jar:org.hibernate:hibernate-validator:4.0.2.GA]
> to[lib/hibernate-validator-4.0.2.GA.jar]
> [INFO] Copying artifact[jar:javax.validation:validation-api:1.0.0.GA]
> to[lib/validation-api-1.0.0.GA.jar]
> [INFO] Copying artifact[jar:org.slf4j:slf4j-api:1.5.6]
> to[lib/slf4j-api-1.5.6.jar]
> [INFO] Copying artifact[jar:javax.xml.bind:jaxb-api:2.1]
> to[lib/jaxb-api-2.1.jar]
> [INFO] Copying artifact[jar:javax.xml.stream:stax-api:1.0-2]
> to[lib/stax-api-1.0-2.jar]
> [INFO] Copying artifact[jar:javax.activation:activation:1.1]
> to[lib/activation-1.1.jar]
> [INFO] Copying artifact[jar:com.sun.xml.bind:jaxb-impl:2.1.3]
> to[lib/jaxb-impl-2.1.3.jar]
> [INFO] Copying
> artifact[jar:org.hibernate:hibernate-annotations:3.4.0.GA]
> to[lib/hibernate-annotations-3.4.0.GA.jar]
> [INFO] Copying artifact[jar:org.hibernate:ejb3-persistence:1.0.2.GA]
> to[lib/ejb3-persistence-1.0.2.GA.jar]
> [INFO] Copying
> artifact[jar:org.hibernate:hibernate-commons-annotations:3.1.0.GA]
> to[lib/hibernate-commons-annotations-3.1.0.GA.jar]
> [INFO] Copying artifact[jar:org.hibernate:hibernate-core:3.3.0.SP1]
> to[lib/hibernate-core-3.3.0.SP1.jar]
> [INFO] Copying artifact[jar:antlr:antlr:2.7.6]
> to[lib/antlr-2.7.6.jar]
> [INFO] Copying artifact[jar:dom4j:dom4j:1.6.1]
> to[lib/dom4j-1.6.1.jar]
> [INFO] Copying artifact[jar:xml-apis:xml-apis:1.0.b2]
> to[lib/xml-apis-1.0.b2.jar]
> [INFO] Copying
> artifact[jar:org.jboss.spec.javax.interceptor:jboss-interceptors-api_1.1_spec:1.0.0.Final]
> to[lib/jboss-interceptors-api_1.1_spec-1.0.0.Final.jar]
> [INFO] Copying
> artifact[jar:org.ovirt.engine.core:engine-tools-common:3.2.0]
> to[lib/engine-tools-common-3.2.0.jar]
> [INFO] Copying
> artifact[jar:commons-beanutils:commons-beanutils:1.8.2]
> to[lib/commons-beanutils-1.8.2.jar]
> [INFO] Copying artifact[jar:org.apache.mina:mina-core:2.0.1]
> to[lib/mina-core-2.0.1.jar]
> [INFO] Copying artifact[jar:org.apache.sshd:sshd-core:0.7.0]
> to[lib/sshd-core-0.7.0.jar]
> [INFO] Copying
> artifact[jar:org.ovirt.otopi:otopi:0.0.0-master-20121216.120940-2]
> to[lib/otopi-0.0.0-master-SNAPSHOT.jar]
> [INFO] Copying
> artifact[jar:org.ovirt.ovirt-host-deploy:ovirt-host-deploy:0.0.0-master-20121216.121130-2]
> to[lib/ovirt-host-deploy-0.0.0-master-SNAPSHOT.jar]
> [INFO] Copying artifact[jar:org.apache.commons:commons-compress:1.3]
> to[lib/commons-compress-1.3.jar]
> [INFO] Copying artifact[jar:commons-lang:commons-lang:2.4]
> to[lib/commons-lang-2.4.jar]
> [INFO] Copying artifact[jar:org.apache.xmlrpc:xmlrpc-client:3.1.3]
> to[lib/xmlrpc-client-3.1.3.jar]
> [INFO] Copying artifact[jar:org.apache.xmlrpc:xmlrpc-common:3.1.3]
> to[lib/xmlrpc-common-3.1.3.jar]
> [INFO] Copying
> artifact[jar:org.apache.ws.commons.util:ws-commons-util:1.0.2]
> to[lib/ws-commons-util-1.0.2.jar]
> [INFO] Copying
> artifact[jar:org.springframework:spring-jdbc:3.1.1.RELEASE]
> to[lib/spring-jdbc-3.1.1.RELEASE.jar]
> [INFO] Copying
> artifact[jar:org.springframework:spring-tx:3.1.1.RELEASE]
> to[lib/spring-tx-3.1.1.RELEASE.jar]
> [INFO] Copying
> artifact[jar:org.springframework.ldap:spring-ldap-core:1.3.1.RELEASE]
> to[lib/spring-ldap-core-1.3.1.RELEASE.jar]
> [INFO] Copying
> artifact[jar:commons-httpclient:commons-httpclient:3.1]
> to[lib/commons-httpclient-3.1.jar]
> [INFO] Copying
> artifact[jar:commons-collections:commons-collections:3.1]
> to[lib/commons-collections-3.1.jar]
> [INFO] Copying artifact[jar:org.quartz-scheduler:quartz:2.1.2]
> to[lib/quartz-2.1.2.jar]
> [INFO] Copying artifact[jar:c3p0:c3p0:0.9.1.1]
> to[lib/c3p0-0.9.1.1.jar]
> [INFO] Copying
> artifact[jar:org.ovirt.engine.core:searchbackend:3.2.0]
> to[lib/searchbackend-3.2.0.jar]
> [INFO] Copying
> artifact[jar:org.springframework:spring-core:3.1.1.RELEASE]
> to[lib/spring-core-3.1.1.RELEASE.jar]
> [INFO] Copying
> artifact[jar:org.springframework:spring-asm:3.1.1.RELEASE]
> to[lib/spring-asm-3.1.1.RELEASE.jar]
> [INFO] Copying
> artifact[jar:org.springframework:spring-beans:3.1.1.RELEASE]
> to[lib/spring-beans-3.1.1.RELEASE.jar]
> [INFO] Copying
> artifact[jar:org.springframework:spring-context:3.1.1.RELEASE]
> to[lib/spring-context-3.1.1.RELEASE.jar]
> [INFO] Copying
> artifact[jar:org.springframework:spring-aop:3.1.1.RELEASE]
> to[lib/spring-aop-3.1.1.RELEASE.jar]
> [INFO] Copying artifact[jar:aopalliance:aopalliance:1.0]
> to[lib/aopalliance-1.0.jar]
> [INFO] Copying
> artifact[jar:org.springframework:spring-expression:3.1.1.RELEASE]
> to[lib/spring-expression-3.1.1.RELEASE.jar]
> [INFO] Copy ear sources to
> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine
> [WARNING] resourcesDir is deprecated. Please use the
> earSourceDirectory property instead.
> [INFO] Copy ear resources to
> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine
> [INFO] Including custom manifest
> file[/home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine/META-INF/MANIFEST.MF]
> [INFO] Building jar:
> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine.ear
> mojoSucceeded
> org.apache.maven.plugins:maven-ear-plugin:2.6(default-ear)
> mojoStarted
> org.apache.maven.plugins:maven-dependency-plugin:2.1(copy-quartz-jar)
> [INFO]
> [INFO] --- maven-dependency-plugin:2.1:copy (copy-quartz-jar) @
> engine-server-ear ---
> [INFO] Configured Artifact: org.quartz-scheduler:quartz:2.1.2:jar
> mojoSucceeded
> org.apache.maven.plugins:maven-dependency-plugin:2.1(copy-quartz-jar)[INFO]
> Copying quartz-2.1.2.jar to
> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/quartz/quartz-2.1.2.jar
> 
> [INFO] Installing
> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine.ear
> to
> /home/jenkins/workspace/ovirt_engine_unit_tests/.repository/org/ovirt/engine/engine-server-ear/3.2.0/engine-server-ear-3.2.0.ear
> mojoStarted
> org.apache.maven.plugins:maven-install-plugin:2.3.1(default-install)
> [INFO]
> [INFO] --- maven-install-plugin:2.3.1:install (default-install) @
> engine-server-ear ---
> [INFO] Installing
> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/pom.xml to
> /home/jenkins/workspace/ovirt_engine_unit_tests/.repository/org/ovirt/engine/engine-server-ear/3.2.0/engine-server-ear-3.2.0.pom
> mojoSucceeded
> org.apache.maven.plugins:maven-install-plugin:2.3.1(default-install)
> projectSucceeded org.ovirt.engine:engine-server-ear:3.2.0
> sessionEnded
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] oVirt Engine Root Project ......................... SUCCESS
> [15:29.251s]
> [INFO] oVirt Build Tools root ............................ SUCCESS
> [26.004s]
> [INFO] oVirt checkstyle .................................. SUCCESS
> [7:05.769s]
> [INFO] oVirt Checkstyle Checks ........................... SUCCESS
> [41.919s]
> [INFO] oVirt Modules - backend ........................... SUCCESS
> [4.686s]
> [INFO] oVirt Manager ..................................... SUCCESS
> [7.793s]
> [INFO] oVirt DB Scripts .................................. SUCCESS
> [13.494s]
> [INFO] oVirt Modules - manager ........................... SUCCESS
> [57.659s]
> [INFO] CSharp Compatibility .............................. SUCCESS
> [1:24.773s]
> [INFO] Encryption Libraries .............................. SUCCESS
> [18.381s]
> [INFO] oVirt Tools ....................................... SUCCESS
> [10.515s]
> [INFO] oVirt Tools Common Library ........................ SUCCESS
> [15.192s]
> [INFO] Common Code ....................................... SUCCESS
> [1:18.121s]
> [INFO] Common utilities .................................. SUCCESS
> [1:51.096s]
> [INFO] Data Access Layer ................................. SUCCESS
> [1:04.478s]
> [INFO] engine beans ...................................... SUCCESS
> [1.877s]
> [INFO] engine scheduler bean ............................. SUCCESS
> [16.919s]
> [INFO] Vds broker ........................................ SUCCESS
> [32.755s]
> [INFO] Search Backend .................................... SUCCESS
> [27.715s]
> [INFO] Backend Logic @Service bean ....................... SUCCESS
> [2:03.757s]
> [INFO] oVirt RESTful API Backend Integration ............. SUCCESS
> [4.298s]
> [INFO] oVirt RESTful API interface ....................... SUCCESS
> [1.720s]
> [INFO] oVirt Engine API Definition ....................... SUCCESS
> [22.018s]
> [INFO] oVirt Engine API Commom Parent POM ................ SUCCESS
> [1.747s]
> [INFO] oVirt Engine API Common JAX-RS .................... SUCCESS
> [21.870s]
> [INFO] oVirt RESTful API Backend Integration Type Mappers  SUCCESS
> [1:01.945s]
> [INFO] oVirt RESTful API Backend Integration JAX-RS Resources
>  SUCCESS [2:10.153s]
> [INFO] oVirt RESTful API Backend Integration Webapp ...... SUCCESS
> [5:14.944s]
> [INFO] oVirt Engine Web Root ............................. SUCCESS
> [1:17.531s]
> [INFO] oVirt Configuration Tool .......................... SUCCESS
> [1:53.084s]
> [INFO] Notifier Service package .......................... SUCCESS
> [1.166s]
> [INFO] Notifier Service .................................. SUCCESS
> [27.529s]
> [INFO] Notifier Service Resources ........................ SUCCESS
> [18.911s]
> [INFO] oVirt Modules - frontend .......................... SUCCESS
> [3.358s]
> [INFO] oVirt APIs ........................................ SUCCESS
> [20.346s]
> [INFO] oVirt generic API ................................. SUCCESS
> [1:16.903s]
> [INFO] oVirt Modules - webadmin .......................... SUCCESS
> [2.160s]
> [INFO] oVirt Modules - ui ................................ SUCCESS
> [6.200s]
> [INFO] Extensions for GWT ................................ SUCCESS
> [21.327s]
> [INFO] UI Utils Compatibility (for UICommon) ............. SUCCESS
> [18.965s]
> [INFO] Frontend for GWT UI Projects ...................... SUCCESS
> [46.618s]
> [INFO] UICommonWeb ....................................... SUCCESS
> [2:11.624s]
> [INFO] oVirt GWT UI common infrastructure ................ SUCCESS
> [1:17.131s]
> [INFO] WebAdmin .......................................... SUCCESS
> [45.318s]
> [INFO] UserPortal ........................................ SUCCESS
> [19.200s]
> [INFO] oVirt WARs ........................................ SUCCESS
> [1.798s]
> [INFO] oVirt Web Application Module ...................... SUCCESS
> [6.787s]
> [INFO] oVirt Server EAR .................................. SUCCESS
> [24.517s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 58:53.824s
> [INFO] Finished at: Thu Dec 27 08:52:59 CET 2012
> [INFO] Final Memory: 213M/612M
> [INFO]
> ------------------------------------------------------------------------
> Projects to build: [MavenProject: org.ovirt.engine:root:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/pom.xml,
> MavenProject: org.ovirt.engine:build-tools-root:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/pom.xml,
> MavenProject: org.ovirt.engine:checkstyles:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/checkstyles/pom.xml,
> MavenProject: org.ovirt.engine:ovirt-checkstyle-extension:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/ovirt-checkstyle-extension/pom.xml,
> MavenProject: org.ovirt.engine.core:backend:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/pom.xml,
> MavenProject: org.ovirt.engine.core:manager:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/pom.xml,
> MavenProject: org.ovirt.engine.core:dbscripts:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/dbscripts/pom.xml,
> MavenProject: org.ovirt.engine.core:manager-modules:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/pom.xml,
> MavenProject: org.ovirt.engine.core:compat:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/pom.xml,
> MavenProject: org.ovirt.engine.core:engineencryptutils:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/engineencryptutils/pom.xml,
> MavenProject: org.ovirt.engine.core:manager-tools:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/pom.xml,
> MavenProject: org.ovirt.engine.core:engine-tools-common:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-tools-common/pom.xml,
> MavenProject: org.ovirt.engine.core:common:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/pom.xml,
> MavenProject: org.ovirt.engine.core:utils:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/utils/pom.xml,
> MavenProject: org.ovirt.engine.core:dal:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/dal/pom.xml,
> MavenProject: org.ovirt.engine.core:beans:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/pom.xml,
> MavenProject: org.ovirt.engine.core:scheduler:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/scheduler/pom.xml,
> MavenProject: org.ovirt.engine.core:vdsbroker:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/vdsbroker/pom.xml,
> MavenProject: org.ovirt.engine.core:searchbackend:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/pom.xml,
> MavenProject: org.ovirt.engine.core:bll:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/bll/pom.xml,
> MavenProject: org.ovirt.engine.api:restapi-parent:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/pom.xml,
> MavenProject: org.ovirt.engine.api:interface:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/pom.xml,
> MavenProject: org.ovirt.engine.api:restapi-definition:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/definition/pom.xml,
> MavenProject: org.ovirt.engine.api:common-parent:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/pom.xml,
> MavenProject: org.ovirt.engine.api:interface-common-jaxrs:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/jaxrs/pom.xml,
> MavenProject: org.ovirt.engine.api:restapi-types:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/types/pom.xml,
> MavenProject: org.ovirt.engine.api:restapi-jaxrs:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/jaxrs/pom.xml,
> MavenProject: org.ovirt.engine.api:restapi-webapp:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/webapp/pom.xml,
> MavenProject: org.ovirt.engine.core:root-war:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/root/pom.xml,
> MavenProject: org.ovirt.engine.core:engine-config:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-config/pom.xml,
> MavenProject: org.ovirt.engine.core:engine-notifier:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/pom.xml,
> MavenProject: org.ovirt.engine.core:engine-notifier-service:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-service/pom.xml,
> MavenProject: org.ovirt.engine.core:engine-notifier-resources:3.2.0
> @
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-resources/pom.xml,
> MavenProject: org.ovirt.engine.ui:frontend-all:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/pom.xml,
> MavenProject: org.ovirt.engine.ui:api:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/pom.xml,
> MavenProject: org.ovirt.engine.ui:genericapi:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/genericapi/pom.xml,
> MavenProject: org.ovirt.engine.ui:webadmin-all:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/pom.xml,
> MavenProject: org.ovirt.engine.ui:webadmin-modules:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/pom.xml,
> MavenProject: org.ovirt.engine.ui:gwt-extension:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/pom.xml,
> MavenProject: org.ovirt.engine.ui:uicompat:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicompat/pom.xml,
> MavenProject: org.ovirt.engine.ui:frontend:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/frontend/pom.xml,
> MavenProject: org.ovirt.engine.ui:uicommonweb:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/pom.xml,
> MavenProject: org.ovirt.engine.ui:gwt-common:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-common/pom.xml,
> MavenProject: org.ovirt.engine.ui:webadmin:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/webadmin/pom.xml,
> MavenProject: org.ovirt.engine.ui:userportal:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/userportal-gwtp/pom.xml,
> MavenProject: org.ovirt.engine.ui:wars:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/pom.xml,
> MavenProject: org.ovirt.engine.ui:rmw-war:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/rmw-war/pom.xml,
> MavenProject: org.ovirt.engine:engine-server-ear:3.2.0 @
> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/pom.xml]
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/target/uicommonweb-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/target/uicommonweb-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/target/uicommonweb-3.2.0-sources.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/root/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/root/target/root-war-3.2.0.war
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/webapp/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/webapp/target/restapi-webapp-3.2.0.war
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-tools-common/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-tools-common/target/engine-tools-common-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-tools-common/target/engine-tools-common-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/target/common-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/target/common-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/target/common-3.2.0-sources.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-common/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-common/target/gwt-common-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-common/target/gwt-common-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/checkstyles/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/checkstyles/target/checkstyles-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/checkstyles/target/checkstyles-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/scheduler/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/scheduler/target/scheduler-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/scheduler/target/scheduler-3.2.0-client.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-config/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-config/target/engine-config-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-config/target/engine-config-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-resources/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-resources/target/engine-notifier-resources-3.2.0.zip
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/genericapi/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/genericapi/target/genericapi-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/genericapi/target/genericapi-3.2.0-client.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/engineencryptutils/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/engineencryptutils/target/engineencryptutils-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/engineencryptutils/target/engineencryptutils-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-service/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-service/target/engine-notifier-service-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-service/target/engine-notifier-service-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/utils/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/utils/target/utils-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/utils/target/utils-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/definition/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/definition/target/restapi-definition-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/definition/target/restapi-definition-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/types/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/types/target/restapi-types-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/types/target/restapi-types-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicompat/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicompat/target/uicompat-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicompat/target/uicompat-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/ovirt-checkstyle-extension/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/ovirt-checkstyle-extension/target/ovirt-checkstyle-extension-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/ovirt-checkstyle-extension/target/ovirt-checkstyle-extension-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/jaxrs/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/jaxrs/target/restapi-jaxrs-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/jaxrs/target/restapi-jaxrs-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/frontend/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/frontend/target/frontend-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/frontend/target/frontend-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/rmw-war/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/rmw-war/target/rmw-war-3.2.0.war
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/dal/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/dal/target/dal-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/dal/target/dal-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine.ear
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/webadmin/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/webadmin/target/webadmin-3.2.0.war
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/target/searchbackend-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/target/searchbackend-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/target/searchbackend-3.2.0-sources.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/target/gwt-extension-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/target/gwt-extension-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/target/gwt-extension-3.2.0-sources.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/target/compat-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/target/compat-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/target/compat-3.2.0-sources.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/jaxrs/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/jaxrs/target/interface-common-jaxrs-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/jaxrs/target/interface-common-jaxrs-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/userportal-gwtp/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/userportal-gwtp/target/userportal-3.2.0.war
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/vdsbroker/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/vdsbroker/target/vdsbroker-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/vdsbroker/target/vdsbroker-3.2.0-tests.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/bll/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/bll/target/bll-3.2.0.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/bll/target/bll-3.2.0-client.jar
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/pom.xml
> [JENKINS] Archiving disabled - not archiving
> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/dbscripts/pom.xml
> Waiting for Jenkins to finish collecting data
> channel stopped
> IRC notifier plugin: Sending notification to: #ovirt-jenkins
> Email was triggered for: Unstable
> Sending email for trigger: Unstable
> 


From eedri at redhat.com  Thu Dec 27 09:15:50 2012
From: eedri at redhat.com (Eyal Edri)
Date: Thu, 27 Dec 2012 04:15:50 -0500 (EST)
Subject: [Engine-devel] [oVirt Jenkins] ovirt_engine_unit_tests - Build
 # 2971 - Still Unstable!
In-Reply-To: <2137910072.801976.1356595530556.JavaMail.root@redhat.com>
Message-ID: <693405074.808253.1356599750763.JavaMail.root@redhat.com>

Arik,

it seems your commit [1] is related to the unit test that is failing,
can you please fix or disable the test? 

[1] http://gerrit.ovirt.org/gitweb?p=ovirt-engine.git&a=commit&h=f673ddb66a0fdaaaf8902fe9dacc6d0e55178eb1

Eyal.

----- Original Message -----
> From: "Eyal Edri" <eedri at redhat.com>
> To: engine-devel at ovirt.org, mkolesni at redhat.com, "Eli Mesika" <emesika at redhat.com>
> Sent: Thursday, December 27, 2012 10:05:30 AM
> Subject: Re: [Engine-devel] [oVirt Jenkins] ovirt_engine_unit_tests - Build # 2971 - Still Unstable!
> 
> fyi,
> 
> unit tests are failing from yesterday, please check.
> 
> Eyal Edri.
> oVirt infra team.
> 
> ----- Original Message -----
> > From: "Jenkins oVirt Server" <jenkins at ovirt.org>
> > To: eedri at redhat.com, engine-patches at ovirt.org, oliel at redhat.com,
> > yzaslavs at redhat.com, mkolesni at redhat.com
> > Sent: Thursday, December 27, 2012 9:53:20 AM
> > Subject: [oVirt Jenkins] ovirt_engine_unit_tests - Build # 2971 -
> > Still Unstable!
> > 
> > Project: http://jenkins.ovirt.org/job/ovirt_engine_unit_tests/
> > Build: http://jenkins.ovirt.org/job/ovirt_engine_unit_tests/2971/
> > Build Number: 2971
> > Build Status:  Still Unstable
> > Triggered By: Started by upstream project "ovirt_engine" build
> > number
> > 3,735
> > 
> > -------------------------------------
> > Changes Since Last Success:
> > -------------------------------------
> > Changes for Build #2969
> > 
> > Changes for Build #2970
> > [emesika] core:fixing GeneralDbDAOTest
> > 
> > 
> > Changes for Build #2971
> > [mkolesni] engine: Add network command packages to factory
> > 
> > [mkolesni] engine: Move DC network commands to their package
> > 
> > [mkolesni] engine: Moved VM NIC commands to their package
> > 
> > [mkolesni] engine: Move template NIC commands to their package
> > 
> > [mkolesni] engine: Extract SetNetworkStatus method
> > 
> > [mkolesni] engine: Move cluster network commands to their package
> > 
> > [mkolesni] engine: Move host network commands to their package
> > 
> > 
> > 
> > 
> > -----------------
> > Failed Tests:
> > -----------------
> > 2 tests failed.
> > REGRESSION:
> >  org.ovirt.engine.api.restapi.types.VmMapperTest.testRoundtrip
> > 
> > Error Message:
> > expected:<true> but was:<false>
> > 
> > Stack Trace:
> > java.lang.AssertionError: expected:<true> but was:<false>
> > 	at org.junit.Assert.fail(Assert.java:93)
> > 	at org.junit.Assert.failNotEquals(Assert.java:647)
> > 	at org.junit.Assert.assertEquals(Assert.java:128)
> > 	at org.junit.Assert.assertEquals(Assert.java:147)
> > 	at
> > 	org.ovirt.engine.api.restapi.types.VmMapperTest.verify(VmMapperTest.java:109)
> > 	at
> > 	org.ovirt.engine.api.restapi.types.VmMapperTest.verify(VmMapperTest.java:26)
> > 	at
> > 	org.ovirt.engine.api.restapi.types.AbstractInvertibleMappingTest.testRoundtrip(AbstractInvertibleMappingTest.java:57)
> > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > 	at
> > 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > 	at
> > 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > 	at java.lang.reflect.Method.invoke(Method.java:601)
> > 	at
> > 	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> > 	at
> > 	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> > 	at
> > 	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> > 	at
> > 	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> > 	at
> > 	org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> > 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
> > 	at org.junit.rules.RunRules.evaluate(RunRules.java:18)
> > 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> > 	at
> > 	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> > 	at
> > 	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> > 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> > 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> > 	at
> > 	org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> > 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> > 	at
> > 	org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> > 	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> > 	at
> > 	org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> > 	at
> > 	org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
> > 	at
> > 	org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
> > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > 	at
> > 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > 	at
> > 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > 	at java.lang.reflect.Method.invoke(Method.java:601)
> > 	at
> > 	org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
> > 	at
> > 	org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
> > 	at
> > 	org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
> > 	at
> > 	org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
> > 	at
> > 	org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
> > 
> > 
> > FAILED:
> >  org.ovirt.engine.api.restapi.types.TemplateMapperTest.testRoundtrip
> > 
> > Error Message:
> > expected:<true> but was:<false>
> > 
> > Stack Trace:
> > java.lang.AssertionError: expected:<true> but was:<false>
> > 	at org.junit.Assert.fail(Assert.java:93)
> > 	at org.junit.Assert.failNotEquals(Assert.java:647)
> > 	at org.junit.Assert.assertEquals(Assert.java:128)
> > 	at org.junit.Assert.assertEquals(Assert.java:147)
> > 	at
> > 	org.ovirt.engine.api.restapi.types.TemplateMapperTest.verify(TemplateMapperTest.java:71)
> > 	at
> > 	org.ovirt.engine.api.restapi.types.TemplateMapperTest.verify(TemplateMapperTest.java:12)
> > 	at
> > 	org.ovirt.engine.api.restapi.types.AbstractInvertibleMappingTest.testRoundtrip(AbstractInvertibleMappingTest.java:57)
> > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > 	at
> > 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > 	at
> > 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > 	at java.lang.reflect.Method.invoke(Method.java:601)
> > 	at
> > 	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> > 	at
> > 	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> > 	at
> > 	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> > 	at
> > 	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> > 	at
> > 	org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> > 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
> > 	at org.junit.rules.RunRules.evaluate(RunRules.java:18)
> > 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> > 	at
> > 	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> > 	at
> > 	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> > 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> > 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> > 	at
> > 	org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> > 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> > 	at
> > 	org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> > 	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> > 	at
> > 	org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> > 	at
> > 	org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
> > 	at
> > 	org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
> > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > 	at
> > 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > 	at
> > 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > 	at java.lang.reflect.Method.invoke(Method.java:601)
> > 	at
> > 	org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
> > 	at
> > 	org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
> > 	at
> > 	org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
> > 	at
> > 	org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
> > 	at
> > 	org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
> > 
> > 
> >  
> > 
> > ------------------
> > Build Log:
> > ------------------
> > [...truncated 4214 lines...]
> > [INFO] [INFO] Copying
> > artifact[jar:org.ovirt.engine.core:compat:3.2.0]
> > to[lib/engine-compat.jar]
> > [INFO] Copying artifact[jar:org.ovirt.engine.core:dal:3.2.0]
> > to[lib/engine-dal.jar]
> > [INFO] Copying artifact[jar:org.ovirt.engine.core:utils:3.2.0]
> > to[lib/engine-utils.jar]
> > [INFO] Copying
> > artifact[jar:org.ovirt.engine.core:engineencryptutils:3.2.0]
> > to[lib/engine-encryptutils.jar]
> > 
> > [INFO] Copying artifact[jar:org.ovirt.engine.core:vdsbroker:3.2.0]
> > to[lib/engine-vdsbroker.jar]
> > [INFO] Copying artifact[war:org.ovirt.engine.core:root-war:3.2.0]
> > to[root.war] (unpacked)
> > [INFO] Copying artifact[war:org.ovirt.engine.ui:rmw-war:3.2.0]
> > to[ovirtengineweb.war] (unpacked)
> > [INFO] Copying
> > artifact[war:org.ovirt.engine.api:restapi-webapp:3.2.0]
> > to[restapi.war] (unpacked)
> > [INFO] Copying artifact[war:org.ovirt.engine.ui:userportal:3.2.0]
> > to[userportal.war] (unpacked)
> > [INFO] --- maven-ear-plugin:2.6:ear (default-ear) @
> > engine-server-ear
> > ---
> > [INFO] Copying artifact[war:org.ovirt.engine.ui:webadmin:3.2.0]
> > to[webadmin.war] (unpacked)
> > [INFO] Copying artifact[ejb:org.ovirt.engine.ui:genericapi:3.2.0]
> > to[engine-genericapi.jar] (unpacked)
> > [INFO] Copying artifact[ejb:org.ovirt.engine.core:scheduler:3.2.0]
> > to[engine-scheduler.jar] (unpacked)
> > [INFO] Copying artifact[ejb:org.ovirt.engine.core:bll:3.2.0]
> > to[engine-bll.jar] (unpacked)
> > [INFO] Copying artifact[jar:commons-codec:commons-codec:1.4]
> > to[lib/commons-codec-1.4.jar]
> > [INFO] Copying
> > artifact[jar:org.hibernate:hibernate-validator:4.0.2.GA]
> > to[lib/hibernate-validator-4.0.2.GA.jar]
> > [INFO] Copying
> > artifact[jar:javax.validation:validation-api:1.0.0.GA]
> > to[lib/validation-api-1.0.0.GA.jar]
> > [INFO] Copying artifact[jar:org.slf4j:slf4j-api:1.5.6]
> > to[lib/slf4j-api-1.5.6.jar]
> > [INFO] Copying artifact[jar:javax.xml.bind:jaxb-api:2.1]
> > to[lib/jaxb-api-2.1.jar]
> > [INFO] Copying artifact[jar:javax.xml.stream:stax-api:1.0-2]
> > to[lib/stax-api-1.0-2.jar]
> > [INFO] Copying artifact[jar:javax.activation:activation:1.1]
> > to[lib/activation-1.1.jar]
> > [INFO] Copying artifact[jar:com.sun.xml.bind:jaxb-impl:2.1.3]
> > to[lib/jaxb-impl-2.1.3.jar]
> > [INFO] Copying
> > artifact[jar:org.hibernate:hibernate-annotations:3.4.0.GA]
> > to[lib/hibernate-annotations-3.4.0.GA.jar]
> > [INFO] Copying
> > artifact[jar:org.hibernate:ejb3-persistence:1.0.2.GA]
> > to[lib/ejb3-persistence-1.0.2.GA.jar]
> > [INFO] Copying
> > artifact[jar:org.hibernate:hibernate-commons-annotations:3.1.0.GA]
> > to[lib/hibernate-commons-annotations-3.1.0.GA.jar]
> > [INFO] Copying artifact[jar:org.hibernate:hibernate-core:3.3.0.SP1]
> > to[lib/hibernate-core-3.3.0.SP1.jar]
> > [INFO] Copying artifact[jar:antlr:antlr:2.7.6]
> > to[lib/antlr-2.7.6.jar]
> > [INFO] Copying artifact[jar:dom4j:dom4j:1.6.1]
> > to[lib/dom4j-1.6.1.jar]
> > [INFO] Copying artifact[jar:xml-apis:xml-apis:1.0.b2]
> > to[lib/xml-apis-1.0.b2.jar]
> > [INFO] Copying
> > artifact[jar:org.jboss.spec.javax.interceptor:jboss-interceptors-api_1.1_spec:1.0.0.Final]
> > to[lib/jboss-interceptors-api_1.1_spec-1.0.0.Final.jar]
> > [INFO] Copying
> > artifact[jar:org.ovirt.engine.core:engine-tools-common:3.2.0]
> > to[lib/engine-tools-common-3.2.0.jar]
> > [INFO] Copying
> > artifact[jar:commons-beanutils:commons-beanutils:1.8.2]
> > to[lib/commons-beanutils-1.8.2.jar]
> > [INFO] Copying artifact[jar:org.apache.mina:mina-core:2.0.1]
> > to[lib/mina-core-2.0.1.jar]
> > [INFO] Copying artifact[jar:org.apache.sshd:sshd-core:0.7.0]
> > to[lib/sshd-core-0.7.0.jar]
> > [INFO] Copying
> > artifact[jar:org.ovirt.otopi:otopi:0.0.0-master-20121216.120940-2]
> > to[lib/otopi-0.0.0-master-SNAPSHOT.jar]
> > [INFO] Copying
> > artifact[jar:org.ovirt.ovirt-host-deploy:ovirt-host-deploy:0.0.0-master-20121216.121130-2]
> > to[lib/ovirt-host-deploy-0.0.0-master-SNAPSHOT.jar]
> > [INFO] Copying
> > artifact[jar:org.apache.commons:commons-compress:1.3]
> > to[lib/commons-compress-1.3.jar]
> > [INFO] Copying artifact[jar:commons-lang:commons-lang:2.4]
> > to[lib/commons-lang-2.4.jar]
> > [INFO] Copying artifact[jar:org.apache.xmlrpc:xmlrpc-client:3.1.3]
> > to[lib/xmlrpc-client-3.1.3.jar]
> > [INFO] Copying artifact[jar:org.apache.xmlrpc:xmlrpc-common:3.1.3]
> > to[lib/xmlrpc-common-3.1.3.jar]
> > [INFO] Copying
> > artifact[jar:org.apache.ws.commons.util:ws-commons-util:1.0.2]
> > to[lib/ws-commons-util-1.0.2.jar]
> > [INFO] Copying
> > artifact[jar:org.springframework:spring-jdbc:3.1.1.RELEASE]
> > to[lib/spring-jdbc-3.1.1.RELEASE.jar]
> > [INFO] Copying
> > artifact[jar:org.springframework:spring-tx:3.1.1.RELEASE]
> > to[lib/spring-tx-3.1.1.RELEASE.jar]
> > [INFO] Copying
> > artifact[jar:org.springframework.ldap:spring-ldap-core:1.3.1.RELEASE]
> > to[lib/spring-ldap-core-1.3.1.RELEASE.jar]
> > [INFO] Copying
> > artifact[jar:commons-httpclient:commons-httpclient:3.1]
> > to[lib/commons-httpclient-3.1.jar]
> > [INFO] Copying
> > artifact[jar:commons-collections:commons-collections:3.1]
> > to[lib/commons-collections-3.1.jar]
> > [INFO] Copying artifact[jar:org.quartz-scheduler:quartz:2.1.2]
> > to[lib/quartz-2.1.2.jar]
> > [INFO] Copying artifact[jar:c3p0:c3p0:0.9.1.1]
> > to[lib/c3p0-0.9.1.1.jar]
> > [INFO] Copying
> > artifact[jar:org.ovirt.engine.core:searchbackend:3.2.0]
> > to[lib/searchbackend-3.2.0.jar]
> > [INFO] Copying
> > artifact[jar:org.springframework:spring-core:3.1.1.RELEASE]
> > to[lib/spring-core-3.1.1.RELEASE.jar]
> > [INFO] Copying
> > artifact[jar:org.springframework:spring-asm:3.1.1.RELEASE]
> > to[lib/spring-asm-3.1.1.RELEASE.jar]
> > [INFO] Copying
> > artifact[jar:org.springframework:spring-beans:3.1.1.RELEASE]
> > to[lib/spring-beans-3.1.1.RELEASE.jar]
> > [INFO] Copying
> > artifact[jar:org.springframework:spring-context:3.1.1.RELEASE]
> > to[lib/spring-context-3.1.1.RELEASE.jar]
> > [INFO] Copying
> > artifact[jar:org.springframework:spring-aop:3.1.1.RELEASE]
> > to[lib/spring-aop-3.1.1.RELEASE.jar]
> > [INFO] Copying artifact[jar:aopalliance:aopalliance:1.0]
> > to[lib/aopalliance-1.0.jar]
> > [INFO] Copying
> > artifact[jar:org.springframework:spring-expression:3.1.1.RELEASE]
> > to[lib/spring-expression-3.1.1.RELEASE.jar]
> > [INFO] Copy ear sources to
> > /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine
> > [WARNING] resourcesDir is deprecated. Please use the
> > earSourceDirectory property instead.
> > [INFO] Copy ear resources to
> > /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine
> > [INFO] Including custom manifest
> > file[/home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine/META-INF/MANIFEST.MF]
> > [INFO] Building jar:
> > /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine.ear
> > mojoSucceeded
> > org.apache.maven.plugins:maven-ear-plugin:2.6(default-ear)
> > mojoStarted
> > org.apache.maven.plugins:maven-dependency-plugin:2.1(copy-quartz-jar)
> > [INFO]
> > [INFO] --- maven-dependency-plugin:2.1:copy (copy-quartz-jar) @
> > engine-server-ear ---
> > [INFO] Configured Artifact: org.quartz-scheduler:quartz:2.1.2:jar
> > mojoSucceeded
> > org.apache.maven.plugins:maven-dependency-plugin:2.1(copy-quartz-jar)[INFO]
> > Copying quartz-2.1.2.jar to
> > /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/quartz/quartz-2.1.2.jar
> > 
> > [INFO] Installing
> > /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine.ear
> > to
> > /home/jenkins/workspace/ovirt_engine_unit_tests/.repository/org/ovirt/engine/engine-server-ear/3.2.0/engine-server-ear-3.2.0.ear
> > mojoStarted
> > org.apache.maven.plugins:maven-install-plugin:2.3.1(default-install)
> > [INFO]
> > [INFO] --- maven-install-plugin:2.3.1:install (default-install) @
> > engine-server-ear ---
> > [INFO] Installing
> > /home/jenkins/workspace/ovirt_engine_unit_tests/ear/pom.xml to
> > /home/jenkins/workspace/ovirt_engine_unit_tests/.repository/org/ovirt/engine/engine-server-ear/3.2.0/engine-server-ear-3.2.0.pom
> > mojoSucceeded
> > org.apache.maven.plugins:maven-install-plugin:2.3.1(default-install)
> > projectSucceeded org.ovirt.engine:engine-server-ear:3.2.0
> > sessionEnded
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] Reactor Summary:
> > [INFO]
> > [INFO] oVirt Engine Root Project ......................... SUCCESS
> > [15:29.251s]
> > [INFO] oVirt Build Tools root ............................ SUCCESS
> > [26.004s]
> > [INFO] oVirt checkstyle .................................. SUCCESS
> > [7:05.769s]
> > [INFO] oVirt Checkstyle Checks ........................... SUCCESS
> > [41.919s]
> > [INFO] oVirt Modules - backend ........................... SUCCESS
> > [4.686s]
> > [INFO] oVirt Manager ..................................... SUCCESS
> > [7.793s]
> > [INFO] oVirt DB Scripts .................................. SUCCESS
> > [13.494s]
> > [INFO] oVirt Modules - manager ........................... SUCCESS
> > [57.659s]
> > [INFO] CSharp Compatibility .............................. SUCCESS
> > [1:24.773s]
> > [INFO] Encryption Libraries .............................. SUCCESS
> > [18.381s]
> > [INFO] oVirt Tools ....................................... SUCCESS
> > [10.515s]
> > [INFO] oVirt Tools Common Library ........................ SUCCESS
> > [15.192s]
> > [INFO] Common Code ....................................... SUCCESS
> > [1:18.121s]
> > [INFO] Common utilities .................................. SUCCESS
> > [1:51.096s]
> > [INFO] Data Access Layer ................................. SUCCESS
> > [1:04.478s]
> > [INFO] engine beans ...................................... SUCCESS
> > [1.877s]
> > [INFO] engine scheduler bean ............................. SUCCESS
> > [16.919s]
> > [INFO] Vds broker ........................................ SUCCESS
> > [32.755s]
> > [INFO] Search Backend .................................... SUCCESS
> > [27.715s]
> > [INFO] Backend Logic @Service bean ....................... SUCCESS
> > [2:03.757s]
> > [INFO] oVirt RESTful API Backend Integration ............. SUCCESS
> > [4.298s]
> > [INFO] oVirt RESTful API interface ....................... SUCCESS
> > [1.720s]
> > [INFO] oVirt Engine API Definition ....................... SUCCESS
> > [22.018s]
> > [INFO] oVirt Engine API Commom Parent POM ................ SUCCESS
> > [1.747s]
> > [INFO] oVirt Engine API Common JAX-RS .................... SUCCESS
> > [21.870s]
> > [INFO] oVirt RESTful API Backend Integration Type Mappers  SUCCESS
> > [1:01.945s]
> > [INFO] oVirt RESTful API Backend Integration JAX-RS Resources
> >  SUCCESS [2:10.153s]
> > [INFO] oVirt RESTful API Backend Integration Webapp ...... SUCCESS
> > [5:14.944s]
> > [INFO] oVirt Engine Web Root ............................. SUCCESS
> > [1:17.531s]
> > [INFO] oVirt Configuration Tool .......................... SUCCESS
> > [1:53.084s]
> > [INFO] Notifier Service package .......................... SUCCESS
> > [1.166s]
> > [INFO] Notifier Service .................................. SUCCESS
> > [27.529s]
> > [INFO] Notifier Service Resources ........................ SUCCESS
> > [18.911s]
> > [INFO] oVirt Modules - frontend .......................... SUCCESS
> > [3.358s]
> > [INFO] oVirt APIs ........................................ SUCCESS
> > [20.346s]
> > [INFO] oVirt generic API ................................. SUCCESS
> > [1:16.903s]
> > [INFO] oVirt Modules - webadmin .......................... SUCCESS
> > [2.160s]
> > [INFO] oVirt Modules - ui ................................ SUCCESS
> > [6.200s]
> > [INFO] Extensions for GWT ................................ SUCCESS
> > [21.327s]
> > [INFO] UI Utils Compatibility (for UICommon) ............. SUCCESS
> > [18.965s]
> > [INFO] Frontend for GWT UI Projects ...................... SUCCESS
> > [46.618s]
> > [INFO] UICommonWeb ....................................... SUCCESS
> > [2:11.624s]
> > [INFO] oVirt GWT UI common infrastructure ................ SUCCESS
> > [1:17.131s]
> > [INFO] WebAdmin .......................................... SUCCESS
> > [45.318s]
> > [INFO] UserPortal ........................................ SUCCESS
> > [19.200s]
> > [INFO] oVirt WARs ........................................ SUCCESS
> > [1.798s]
> > [INFO] oVirt Web Application Module ...................... SUCCESS
> > [6.787s]
> > [INFO] oVirt Server EAR .................................. SUCCESS
> > [24.517s]
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] BUILD SUCCESS
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] Total time: 58:53.824s
> > [INFO] Finished at: Thu Dec 27 08:52:59 CET 2012
> > [INFO] Final Memory: 213M/612M
> > [INFO]
> > ------------------------------------------------------------------------
> > Projects to build: [MavenProject: org.ovirt.engine:root:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/pom.xml,
> > MavenProject: org.ovirt.engine:build-tools-root:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/pom.xml,
> > MavenProject: org.ovirt.engine:checkstyles:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/checkstyles/pom.xml,
> > MavenProject: org.ovirt.engine:ovirt-checkstyle-extension:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/ovirt-checkstyle-extension/pom.xml,
> > MavenProject: org.ovirt.engine.core:backend:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/pom.xml,
> > MavenProject: org.ovirt.engine.core:manager:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/pom.xml,
> > MavenProject: org.ovirt.engine.core:dbscripts:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/dbscripts/pom.xml,
> > MavenProject: org.ovirt.engine.core:manager-modules:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/pom.xml,
> > MavenProject: org.ovirt.engine.core:compat:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/pom.xml,
> > MavenProject: org.ovirt.engine.core:engineencryptutils:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/engineencryptutils/pom.xml,
> > MavenProject: org.ovirt.engine.core:manager-tools:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/pom.xml,
> > MavenProject: org.ovirt.engine.core:engine-tools-common:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-tools-common/pom.xml,
> > MavenProject: org.ovirt.engine.core:common:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/pom.xml,
> > MavenProject: org.ovirt.engine.core:utils:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/utils/pom.xml,
> > MavenProject: org.ovirt.engine.core:dal:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/dal/pom.xml,
> > MavenProject: org.ovirt.engine.core:beans:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/pom.xml,
> > MavenProject: org.ovirt.engine.core:scheduler:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/scheduler/pom.xml,
> > MavenProject: org.ovirt.engine.core:vdsbroker:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/vdsbroker/pom.xml,
> > MavenProject: org.ovirt.engine.core:searchbackend:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/pom.xml,
> > MavenProject: org.ovirt.engine.core:bll:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/bll/pom.xml,
> > MavenProject: org.ovirt.engine.api:restapi-parent:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/pom.xml,
> > MavenProject: org.ovirt.engine.api:interface:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/pom.xml,
> > MavenProject: org.ovirt.engine.api:restapi-definition:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/definition/pom.xml,
> > MavenProject: org.ovirt.engine.api:common-parent:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/pom.xml,
> > MavenProject: org.ovirt.engine.api:interface-common-jaxrs:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/jaxrs/pom.xml,
> > MavenProject: org.ovirt.engine.api:restapi-types:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/types/pom.xml,
> > MavenProject: org.ovirt.engine.api:restapi-jaxrs:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/jaxrs/pom.xml,
> > MavenProject: org.ovirt.engine.api:restapi-webapp:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/webapp/pom.xml,
> > MavenProject: org.ovirt.engine.core:root-war:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/root/pom.xml,
> > MavenProject: org.ovirt.engine.core:engine-config:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-config/pom.xml,
> > MavenProject: org.ovirt.engine.core:engine-notifier:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/pom.xml,
> > MavenProject: org.ovirt.engine.core:engine-notifier-service:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-service/pom.xml,
> > MavenProject: org.ovirt.engine.core:engine-notifier-resources:3.2.0
> > @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-resources/pom.xml,
> > MavenProject: org.ovirt.engine.ui:frontend-all:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/pom.xml,
> > MavenProject: org.ovirt.engine.ui:api:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/pom.xml,
> > MavenProject: org.ovirt.engine.ui:genericapi:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/genericapi/pom.xml,
> > MavenProject: org.ovirt.engine.ui:webadmin-all:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/pom.xml,
> > MavenProject: org.ovirt.engine.ui:webadmin-modules:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/pom.xml,
> > MavenProject: org.ovirt.engine.ui:gwt-extension:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/pom.xml,
> > MavenProject: org.ovirt.engine.ui:uicompat:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicompat/pom.xml,
> > MavenProject: org.ovirt.engine.ui:frontend:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/frontend/pom.xml,
> > MavenProject: org.ovirt.engine.ui:uicommonweb:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/pom.xml,
> > MavenProject: org.ovirt.engine.ui:gwt-common:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-common/pom.xml,
> > MavenProject: org.ovirt.engine.ui:webadmin:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/webadmin/pom.xml,
> > MavenProject: org.ovirt.engine.ui:userportal:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/userportal-gwtp/pom.xml,
> > MavenProject: org.ovirt.engine.ui:wars:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/pom.xml,
> > MavenProject: org.ovirt.engine.ui:rmw-war:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/rmw-war/pom.xml,
> > MavenProject: org.ovirt.engine:engine-server-ear:3.2.0 @
> > /home/jenkins/workspace/ovirt_engine_unit_tests/ear/pom.xml]
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/target/uicommonweb-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/target/uicommonweb-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/target/uicommonweb-3.2.0-sources.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/root/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/root/target/root-war-3.2.0.war
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/webapp/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/webapp/target/restapi-webapp-3.2.0.war
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-tools-common/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-tools-common/target/engine-tools-common-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-tools-common/target/engine-tools-common-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/target/common-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/target/common-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/target/common-3.2.0-sources.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-common/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-common/target/gwt-common-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-common/target/gwt-common-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/checkstyles/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/checkstyles/target/checkstyles-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/checkstyles/target/checkstyles-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/scheduler/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/scheduler/target/scheduler-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/scheduler/target/scheduler-3.2.0-client.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-config/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-config/target/engine-config-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-config/target/engine-config-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-resources/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-resources/target/engine-notifier-resources-3.2.0.zip
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/genericapi/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/genericapi/target/genericapi-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/genericapi/target/genericapi-3.2.0-client.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/engineencryptutils/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/engineencryptutils/target/engineencryptutils-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/engineencryptutils/target/engineencryptutils-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-service/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-service/target/engine-notifier-service-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-service/target/engine-notifier-service-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/utils/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/utils/target/utils-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/utils/target/utils-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/definition/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/definition/target/restapi-definition-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/definition/target/restapi-definition-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/types/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/types/target/restapi-types-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/types/target/restapi-types-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicompat/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicompat/target/uicompat-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicompat/target/uicompat-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/ovirt-checkstyle-extension/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/ovirt-checkstyle-extension/target/ovirt-checkstyle-extension-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/ovirt-checkstyle-extension/target/ovirt-checkstyle-extension-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/jaxrs/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/jaxrs/target/restapi-jaxrs-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/jaxrs/target/restapi-jaxrs-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/frontend/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/frontend/target/frontend-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/frontend/target/frontend-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/rmw-war/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/rmw-war/target/rmw-war-3.2.0.war
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/dal/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/dal/target/dal-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/dal/target/dal-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/ear/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine.ear
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/webadmin/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/webadmin/target/webadmin-3.2.0.war
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/target/searchbackend-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/target/searchbackend-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/target/searchbackend-3.2.0-sources.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/target/gwt-extension-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/target/gwt-extension-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/target/gwt-extension-3.2.0-sources.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/target/compat-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/target/compat-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/target/compat-3.2.0-sources.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/jaxrs/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/jaxrs/target/interface-common-jaxrs-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/jaxrs/target/interface-common-jaxrs-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/userportal-gwtp/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/userportal-gwtp/target/userportal-3.2.0.war
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/vdsbroker/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/vdsbroker/target/vdsbroker-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/vdsbroker/target/vdsbroker-3.2.0-tests.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/bll/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/bll/target/bll-3.2.0.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/bll/target/bll-3.2.0-client.jar
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/pom.xml
> > [JENKINS] Archiving disabled - not archiving
> > /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/dbscripts/pom.xml
> > Waiting for Jenkins to finish collecting data
> > channel stopped
> > IRC notifier plugin: Sending notification to: #ovirt-jenkins
> > Email was triggered for: Unstable
> > Sending email for trigger: Unstable
> > 
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 


From masayag at redhat.com  Thu Dec 27 12:29:33 2012
From: masayag at redhat.com (Moti Asayag)
Date: Thu, 27 Dec 2012 14:29:33 +0200
Subject: [Engine-devel] [oVirt Jenkins] ovirt_engine_unit_tests - Build
 # 2971 - Still Unstable!
In-Reply-To: <693405074.808253.1356599750763.JavaMail.root@redhat.com>
References: <693405074.808253.1356599750763.JavaMail.root@redhat.com>
Message-ID: <50DC3F2D.5060809@redhat.com>

The tests are failing due to this commit:

http://jenkins.ovirt.org/job/ovirt_engine_unit_tests/2977/changes

Ofri, could you have a look at the tests ?

On 12/27/2012 11:15 AM, Eyal Edri wrote:
> Arik,
> 
> it seems your commit [1] is related to the unit test that is failing,
> can you please fix or disable the test? 
> 
> [1] http://gerrit.ovirt.org/gitweb?p=ovirt-engine.git&a=commit&h=f673ddb66a0fdaaaf8902fe9dacc6d0e55178eb1

This was already fixed in build 2975 by commit
http://gerrit.ovirt.org/gitweb?p=ovirt-engine.git&a=commit&h=dc6469624e9ecabd45f6e6d61c92dfc665a84e37

> 
> Eyal.
> 
> ----- Original Message -----
>> From: "Eyal Edri" <eedri at redhat.com>
>> To: engine-devel at ovirt.org, mkolesni at redhat.com, "Eli Mesika" <emesika at redhat.com>
>> Sent: Thursday, December 27, 2012 10:05:30 AM
>> Subject: Re: [Engine-devel] [oVirt Jenkins] ovirt_engine_unit_tests - Build # 2971 - Still Unstable!
>>
>> fyi,
>>
>> unit tests are failing from yesterday, please check.
>>
>> Eyal Edri.
>> oVirt infra team.
>>
>> ----- Original Message -----
>>> From: "Jenkins oVirt Server" <jenkins at ovirt.org>
>>> To: eedri at redhat.com, engine-patches at ovirt.org, oliel at redhat.com,
>>> yzaslavs at redhat.com, mkolesni at redhat.com
>>> Sent: Thursday, December 27, 2012 9:53:20 AM
>>> Subject: [oVirt Jenkins] ovirt_engine_unit_tests - Build # 2971 -
>>> Still Unstable!
>>>
>>> Project: http://jenkins.ovirt.org/job/ovirt_engine_unit_tests/
>>> Build: http://jenkins.ovirt.org/job/ovirt_engine_unit_tests/2971/
>>> Build Number: 2971
>>> Build Status:  Still Unstable
>>> Triggered By: Started by upstream project "ovirt_engine" build
>>> number
>>> 3,735
>>>
>>> -------------------------------------
>>> Changes Since Last Success:
>>> -------------------------------------
>>> Changes for Build #2969
>>>
>>> Changes for Build #2970
>>> [emesika] core:fixing GeneralDbDAOTest
>>>
>>>
>>> Changes for Build #2971
>>> [mkolesni] engine: Add network command packages to factory
>>>
>>> [mkolesni] engine: Move DC network commands to their package
>>>
>>> [mkolesni] engine: Moved VM NIC commands to their package
>>>
>>> [mkolesni] engine: Move template NIC commands to their package
>>>
>>> [mkolesni] engine: Extract SetNetworkStatus method
>>>
>>> [mkolesni] engine: Move cluster network commands to their package
>>>
>>> [mkolesni] engine: Move host network commands to their package
>>>
>>>
>>>
>>>
>>> -----------------
>>> Failed Tests:
>>> -----------------
>>> 2 tests failed.
>>> REGRESSION:
>>>  org.ovirt.engine.api.restapi.types.VmMapperTest.testRoundtrip
>>>
>>> Error Message:
>>> expected:<true> but was:<false>
>>>
>>> Stack Trace:
>>> java.lang.AssertionError: expected:<true> but was:<false>
>>> 	at org.junit.Assert.fail(Assert.java:93)
>>> 	at org.junit.Assert.failNotEquals(Assert.java:647)
>>> 	at org.junit.Assert.assertEquals(Assert.java:128)
>>> 	at org.junit.Assert.assertEquals(Assert.java:147)
>>> 	at
>>> 	org.ovirt.engine.api.restapi.types.VmMapperTest.verify(VmMapperTest.java:109)
>>> 	at
>>> 	org.ovirt.engine.api.restapi.types.VmMapperTest.verify(VmMapperTest.java:26)
>>> 	at
>>> 	org.ovirt.engine.api.restapi.types.AbstractInvertibleMappingTest.testRoundtrip(AbstractInvertibleMappingTest.java:57)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> 	at
>>> 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> 	at
>>> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> 	at java.lang.reflect.Method.invoke(Method.java:601)
>>> 	at
>>> 	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>>> 	at
>>> 	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>>> 	at
>>> 	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>>> 	at
>>> 	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>>> 	at
>>> 	org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>>> 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
>>> 	at org.junit.rules.RunRules.evaluate(RunRules.java:18)
>>> 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>>> 	at
>>> 	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>>> 	at
>>> 	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>>> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>>> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>>> 	at
>>> 	org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>>> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>>> 	at
>>> 	org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>>> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>>> 	at
>>> 	org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
>>> 	at
>>> 	org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
>>> 	at
>>> 	org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> 	at
>>> 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> 	at
>>> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> 	at java.lang.reflect.Method.invoke(Method.java:601)
>>> 	at
>>> 	org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
>>> 	at
>>> 	org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
>>> 	at
>>> 	org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
>>> 	at
>>> 	org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
>>> 	at
>>> 	org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
>>>
>>>
>>> FAILED:
>>>  org.ovirt.engine.api.restapi.types.TemplateMapperTest.testRoundtrip
>>>
>>> Error Message:
>>> expected:<true> but was:<false>
>>>
>>> Stack Trace:
>>> java.lang.AssertionError: expected:<true> but was:<false>
>>> 	at org.junit.Assert.fail(Assert.java:93)
>>> 	at org.junit.Assert.failNotEquals(Assert.java:647)
>>> 	at org.junit.Assert.assertEquals(Assert.java:128)
>>> 	at org.junit.Assert.assertEquals(Assert.java:147)
>>> 	at
>>> 	org.ovirt.engine.api.restapi.types.TemplateMapperTest.verify(TemplateMapperTest.java:71)
>>> 	at
>>> 	org.ovirt.engine.api.restapi.types.TemplateMapperTest.verify(TemplateMapperTest.java:12)
>>> 	at
>>> 	org.ovirt.engine.api.restapi.types.AbstractInvertibleMappingTest.testRoundtrip(AbstractInvertibleMappingTest.java:57)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> 	at
>>> 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> 	at
>>> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> 	at java.lang.reflect.Method.invoke(Method.java:601)
>>> 	at
>>> 	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>>> 	at
>>> 	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>>> 	at
>>> 	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>>> 	at
>>> 	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>>> 	at
>>> 	org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>>> 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
>>> 	at org.junit.rules.RunRules.evaluate(RunRules.java:18)
>>> 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>>> 	at
>>> 	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>>> 	at
>>> 	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>>> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>>> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>>> 	at
>>> 	org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>>> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>>> 	at
>>> 	org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>>> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>>> 	at
>>> 	org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
>>> 	at
>>> 	org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
>>> 	at
>>> 	org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> 	at
>>> 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> 	at
>>> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> 	at java.lang.reflect.Method.invoke(Method.java:601)
>>> 	at
>>> 	org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
>>> 	at
>>> 	org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
>>> 	at
>>> 	org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
>>> 	at
>>> 	org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
>>> 	at
>>> 	org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
>>>
>>>
>>>  
>>>
>>> ------------------
>>> Build Log:
>>> ------------------
>>> [...truncated 4214 lines...]
>>> [INFO] [INFO] Copying
>>> artifact[jar:org.ovirt.engine.core:compat:3.2.0]
>>> to[lib/engine-compat.jar]
>>> [INFO] Copying artifact[jar:org.ovirt.engine.core:dal:3.2.0]
>>> to[lib/engine-dal.jar]
>>> [INFO] Copying artifact[jar:org.ovirt.engine.core:utils:3.2.0]
>>> to[lib/engine-utils.jar]
>>> [INFO] Copying
>>> artifact[jar:org.ovirt.engine.core:engineencryptutils:3.2.0]
>>> to[lib/engine-encryptutils.jar]
>>>
>>> [INFO] Copying artifact[jar:org.ovirt.engine.core:vdsbroker:3.2.0]
>>> to[lib/engine-vdsbroker.jar]
>>> [INFO] Copying artifact[war:org.ovirt.engine.core:root-war:3.2.0]
>>> to[root.war] (unpacked)
>>> [INFO] Copying artifact[war:org.ovirt.engine.ui:rmw-war:3.2.0]
>>> to[ovirtengineweb.war] (unpacked)
>>> [INFO] Copying
>>> artifact[war:org.ovirt.engine.api:restapi-webapp:3.2.0]
>>> to[restapi.war] (unpacked)
>>> [INFO] Copying artifact[war:org.ovirt.engine.ui:userportal:3.2.0]
>>> to[userportal.war] (unpacked)
>>> [INFO] --- maven-ear-plugin:2.6:ear (default-ear) @
>>> engine-server-ear
>>> ---
>>> [INFO] Copying artifact[war:org.ovirt.engine.ui:webadmin:3.2.0]
>>> to[webadmin.war] (unpacked)
>>> [INFO] Copying artifact[ejb:org.ovirt.engine.ui:genericapi:3.2.0]
>>> to[engine-genericapi.jar] (unpacked)
>>> [INFO] Copying artifact[ejb:org.ovirt.engine.core:scheduler:3.2.0]
>>> to[engine-scheduler.jar] (unpacked)
>>> [INFO] Copying artifact[ejb:org.ovirt.engine.core:bll:3.2.0]
>>> to[engine-bll.jar] (unpacked)
>>> [INFO] Copying artifact[jar:commons-codec:commons-codec:1.4]
>>> to[lib/commons-codec-1.4.jar]
>>> [INFO] Copying
>>> artifact[jar:org.hibernate:hibernate-validator:4.0.2.GA]
>>> to[lib/hibernate-validator-4.0.2.GA.jar]
>>> [INFO] Copying
>>> artifact[jar:javax.validation:validation-api:1.0.0.GA]
>>> to[lib/validation-api-1.0.0.GA.jar]
>>> [INFO] Copying artifact[jar:org.slf4j:slf4j-api:1.5.6]
>>> to[lib/slf4j-api-1.5.6.jar]
>>> [INFO] Copying artifact[jar:javax.xml.bind:jaxb-api:2.1]
>>> to[lib/jaxb-api-2.1.jar]
>>> [INFO] Copying artifact[jar:javax.xml.stream:stax-api:1.0-2]
>>> to[lib/stax-api-1.0-2.jar]
>>> [INFO] Copying artifact[jar:javax.activation:activation:1.1]
>>> to[lib/activation-1.1.jar]
>>> [INFO] Copying artifact[jar:com.sun.xml.bind:jaxb-impl:2.1.3]
>>> to[lib/jaxb-impl-2.1.3.jar]
>>> [INFO] Copying
>>> artifact[jar:org.hibernate:hibernate-annotations:3.4.0.GA]
>>> to[lib/hibernate-annotations-3.4.0.GA.jar]
>>> [INFO] Copying
>>> artifact[jar:org.hibernate:ejb3-persistence:1.0.2.GA]
>>> to[lib/ejb3-persistence-1.0.2.GA.jar]
>>> [INFO] Copying
>>> artifact[jar:org.hibernate:hibernate-commons-annotations:3.1.0.GA]
>>> to[lib/hibernate-commons-annotations-3.1.0.GA.jar]
>>> [INFO] Copying artifact[jar:org.hibernate:hibernate-core:3.3.0.SP1]
>>> to[lib/hibernate-core-3.3.0.SP1.jar]
>>> [INFO] Copying artifact[jar:antlr:antlr:2.7.6]
>>> to[lib/antlr-2.7.6.jar]
>>> [INFO] Copying artifact[jar:dom4j:dom4j:1.6.1]
>>> to[lib/dom4j-1.6.1.jar]
>>> [INFO] Copying artifact[jar:xml-apis:xml-apis:1.0.b2]
>>> to[lib/xml-apis-1.0.b2.jar]
>>> [INFO] Copying
>>> artifact[jar:org.jboss.spec.javax.interceptor:jboss-interceptors-api_1.1_spec:1.0.0.Final]
>>> to[lib/jboss-interceptors-api_1.1_spec-1.0.0.Final.jar]
>>> [INFO] Copying
>>> artifact[jar:org.ovirt.engine.core:engine-tools-common:3.2.0]
>>> to[lib/engine-tools-common-3.2.0.jar]
>>> [INFO] Copying
>>> artifact[jar:commons-beanutils:commons-beanutils:1.8.2]
>>> to[lib/commons-beanutils-1.8.2.jar]
>>> [INFO] Copying artifact[jar:org.apache.mina:mina-core:2.0.1]
>>> to[lib/mina-core-2.0.1.jar]
>>> [INFO] Copying artifact[jar:org.apache.sshd:sshd-core:0.7.0]
>>> to[lib/sshd-core-0.7.0.jar]
>>> [INFO] Copying
>>> artifact[jar:org.ovirt.otopi:otopi:0.0.0-master-20121216.120940-2]
>>> to[lib/otopi-0.0.0-master-SNAPSHOT.jar]
>>> [INFO] Copying
>>> artifact[jar:org.ovirt.ovirt-host-deploy:ovirt-host-deploy:0.0.0-master-20121216.121130-2]
>>> to[lib/ovirt-host-deploy-0.0.0-master-SNAPSHOT.jar]
>>> [INFO] Copying
>>> artifact[jar:org.apache.commons:commons-compress:1.3]
>>> to[lib/commons-compress-1.3.jar]
>>> [INFO] Copying artifact[jar:commons-lang:commons-lang:2.4]
>>> to[lib/commons-lang-2.4.jar]
>>> [INFO] Copying artifact[jar:org.apache.xmlrpc:xmlrpc-client:3.1.3]
>>> to[lib/xmlrpc-client-3.1.3.jar]
>>> [INFO] Copying artifact[jar:org.apache.xmlrpc:xmlrpc-common:3.1.3]
>>> to[lib/xmlrpc-common-3.1.3.jar]
>>> [INFO] Copying
>>> artifact[jar:org.apache.ws.commons.util:ws-commons-util:1.0.2]
>>> to[lib/ws-commons-util-1.0.2.jar]
>>> [INFO] Copying
>>> artifact[jar:org.springframework:spring-jdbc:3.1.1.RELEASE]
>>> to[lib/spring-jdbc-3.1.1.RELEASE.jar]
>>> [INFO] Copying
>>> artifact[jar:org.springframework:spring-tx:3.1.1.RELEASE]
>>> to[lib/spring-tx-3.1.1.RELEASE.jar]
>>> [INFO] Copying
>>> artifact[jar:org.springframework.ldap:spring-ldap-core:1.3.1.RELEASE]
>>> to[lib/spring-ldap-core-1.3.1.RELEASE.jar]
>>> [INFO] Copying
>>> artifact[jar:commons-httpclient:commons-httpclient:3.1]
>>> to[lib/commons-httpclient-3.1.jar]
>>> [INFO] Copying
>>> artifact[jar:commons-collections:commons-collections:3.1]
>>> to[lib/commons-collections-3.1.jar]
>>> [INFO] Copying artifact[jar:org.quartz-scheduler:quartz:2.1.2]
>>> to[lib/quartz-2.1.2.jar]
>>> [INFO] Copying artifact[jar:c3p0:c3p0:0.9.1.1]
>>> to[lib/c3p0-0.9.1.1.jar]
>>> [INFO] Copying
>>> artifact[jar:org.ovirt.engine.core:searchbackend:3.2.0]
>>> to[lib/searchbackend-3.2.0.jar]
>>> [INFO] Copying
>>> artifact[jar:org.springframework:spring-core:3.1.1.RELEASE]
>>> to[lib/spring-core-3.1.1.RELEASE.jar]
>>> [INFO] Copying
>>> artifact[jar:org.springframework:spring-asm:3.1.1.RELEASE]
>>> to[lib/spring-asm-3.1.1.RELEASE.jar]
>>> [INFO] Copying
>>> artifact[jar:org.springframework:spring-beans:3.1.1.RELEASE]
>>> to[lib/spring-beans-3.1.1.RELEASE.jar]
>>> [INFO] Copying
>>> artifact[jar:org.springframework:spring-context:3.1.1.RELEASE]
>>> to[lib/spring-context-3.1.1.RELEASE.jar]
>>> [INFO] Copying
>>> artifact[jar:org.springframework:spring-aop:3.1.1.RELEASE]
>>> to[lib/spring-aop-3.1.1.RELEASE.jar]
>>> [INFO] Copying artifact[jar:aopalliance:aopalliance:1.0]
>>> to[lib/aopalliance-1.0.jar]
>>> [INFO] Copying
>>> artifact[jar:org.springframework:spring-expression:3.1.1.RELEASE]
>>> to[lib/spring-expression-3.1.1.RELEASE.jar]
>>> [INFO] Copy ear sources to
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine
>>> [WARNING] resourcesDir is deprecated. Please use the
>>> earSourceDirectory property instead.
>>> [INFO] Copy ear resources to
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine
>>> [INFO] Including custom manifest
>>> file[/home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine/META-INF/MANIFEST.MF]
>>> [INFO] Building jar:
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine.ear
>>> mojoSucceeded
>>> org.apache.maven.plugins:maven-ear-plugin:2.6(default-ear)
>>> mojoStarted
>>> org.apache.maven.plugins:maven-dependency-plugin:2.1(copy-quartz-jar)
>>> [INFO]
>>> [INFO] --- maven-dependency-plugin:2.1:copy (copy-quartz-jar) @
>>> engine-server-ear ---
>>> [INFO] Configured Artifact: org.quartz-scheduler:quartz:2.1.2:jar
>>> mojoSucceeded
>>> org.apache.maven.plugins:maven-dependency-plugin:2.1(copy-quartz-jar)[INFO]
>>> Copying quartz-2.1.2.jar to
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/quartz/quartz-2.1.2.jar
>>>
>>> [INFO] Installing
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine.ear
>>> to
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/.repository/org/ovirt/engine/engine-server-ear/3.2.0/engine-server-ear-3.2.0.ear
>>> mojoStarted
>>> org.apache.maven.plugins:maven-install-plugin:2.3.1(default-install)
>>> [INFO]
>>> [INFO] --- maven-install-plugin:2.3.1:install (default-install) @
>>> engine-server-ear ---
>>> [INFO] Installing
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/pom.xml to
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/.repository/org/ovirt/engine/engine-server-ear/3.2.0/engine-server-ear-3.2.0.pom
>>> mojoSucceeded
>>> org.apache.maven.plugins:maven-install-plugin:2.3.1(default-install)
>>> projectSucceeded org.ovirt.engine:engine-server-ear:3.2.0
>>> sessionEnded
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] oVirt Engine Root Project ......................... SUCCESS
>>> [15:29.251s]
>>> [INFO] oVirt Build Tools root ............................ SUCCESS
>>> [26.004s]
>>> [INFO] oVirt checkstyle .................................. SUCCESS
>>> [7:05.769s]
>>> [INFO] oVirt Checkstyle Checks ........................... SUCCESS
>>> [41.919s]
>>> [INFO] oVirt Modules - backend ........................... SUCCESS
>>> [4.686s]
>>> [INFO] oVirt Manager ..................................... SUCCESS
>>> [7.793s]
>>> [INFO] oVirt DB Scripts .................................. SUCCESS
>>> [13.494s]
>>> [INFO] oVirt Modules - manager ........................... SUCCESS
>>> [57.659s]
>>> [INFO] CSharp Compatibility .............................. SUCCESS
>>> [1:24.773s]
>>> [INFO] Encryption Libraries .............................. SUCCESS
>>> [18.381s]
>>> [INFO] oVirt Tools ....................................... SUCCESS
>>> [10.515s]
>>> [INFO] oVirt Tools Common Library ........................ SUCCESS
>>> [15.192s]
>>> [INFO] Common Code ....................................... SUCCESS
>>> [1:18.121s]
>>> [INFO] Common utilities .................................. SUCCESS
>>> [1:51.096s]
>>> [INFO] Data Access Layer ................................. SUCCESS
>>> [1:04.478s]
>>> [INFO] engine beans ...................................... SUCCESS
>>> [1.877s]
>>> [INFO] engine scheduler bean ............................. SUCCESS
>>> [16.919s]
>>> [INFO] Vds broker ........................................ SUCCESS
>>> [32.755s]
>>> [INFO] Search Backend .................................... SUCCESS
>>> [27.715s]
>>> [INFO] Backend Logic @Service bean ....................... SUCCESS
>>> [2:03.757s]
>>> [INFO] oVirt RESTful API Backend Integration ............. SUCCESS
>>> [4.298s]
>>> [INFO] oVirt RESTful API interface ....................... SUCCESS
>>> [1.720s]
>>> [INFO] oVirt Engine API Definition ....................... SUCCESS
>>> [22.018s]
>>> [INFO] oVirt Engine API Commom Parent POM ................ SUCCESS
>>> [1.747s]
>>> [INFO] oVirt Engine API Common JAX-RS .................... SUCCESS
>>> [21.870s]
>>> [INFO] oVirt RESTful API Backend Integration Type Mappers  SUCCESS
>>> [1:01.945s]
>>> [INFO] oVirt RESTful API Backend Integration JAX-RS Resources
>>>  SUCCESS [2:10.153s]
>>> [INFO] oVirt RESTful API Backend Integration Webapp ...... SUCCESS
>>> [5:14.944s]
>>> [INFO] oVirt Engine Web Root ............................. SUCCESS
>>> [1:17.531s]
>>> [INFO] oVirt Configuration Tool .......................... SUCCESS
>>> [1:53.084s]
>>> [INFO] Notifier Service package .......................... SUCCESS
>>> [1.166s]
>>> [INFO] Notifier Service .................................. SUCCESS
>>> [27.529s]
>>> [INFO] Notifier Service Resources ........................ SUCCESS
>>> [18.911s]
>>> [INFO] oVirt Modules - frontend .......................... SUCCESS
>>> [3.358s]
>>> [INFO] oVirt APIs ........................................ SUCCESS
>>> [20.346s]
>>> [INFO] oVirt generic API ................................. SUCCESS
>>> [1:16.903s]
>>> [INFO] oVirt Modules - webadmin .......................... SUCCESS
>>> [2.160s]
>>> [INFO] oVirt Modules - ui ................................ SUCCESS
>>> [6.200s]
>>> [INFO] Extensions for GWT ................................ SUCCESS
>>> [21.327s]
>>> [INFO] UI Utils Compatibility (for UICommon) ............. SUCCESS
>>> [18.965s]
>>> [INFO] Frontend for GWT UI Projects ...................... SUCCESS
>>> [46.618s]
>>> [INFO] UICommonWeb ....................................... SUCCESS
>>> [2:11.624s]
>>> [INFO] oVirt GWT UI common infrastructure ................ SUCCESS
>>> [1:17.131s]
>>> [INFO] WebAdmin .......................................... SUCCESS
>>> [45.318s]
>>> [INFO] UserPortal ........................................ SUCCESS
>>> [19.200s]
>>> [INFO] oVirt WARs ........................................ SUCCESS
>>> [1.798s]
>>> [INFO] oVirt Web Application Module ...................... SUCCESS
>>> [6.787s]
>>> [INFO] oVirt Server EAR .................................. SUCCESS
>>> [24.517s]
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD SUCCESS
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 58:53.824s
>>> [INFO] Finished at: Thu Dec 27 08:52:59 CET 2012
>>> [INFO] Final Memory: 213M/612M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> Projects to build: [MavenProject: org.ovirt.engine:root:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/pom.xml,
>>> MavenProject: org.ovirt.engine:build-tools-root:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/pom.xml,
>>> MavenProject: org.ovirt.engine:checkstyles:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/checkstyles/pom.xml,
>>> MavenProject: org.ovirt.engine:ovirt-checkstyle-extension:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/ovirt-checkstyle-extension/pom.xml,
>>> MavenProject: org.ovirt.engine.core:backend:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/pom.xml,
>>> MavenProject: org.ovirt.engine.core:manager:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/pom.xml,
>>> MavenProject: org.ovirt.engine.core:dbscripts:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/dbscripts/pom.xml,
>>> MavenProject: org.ovirt.engine.core:manager-modules:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/pom.xml,
>>> MavenProject: org.ovirt.engine.core:compat:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/pom.xml,
>>> MavenProject: org.ovirt.engine.core:engineencryptutils:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/engineencryptutils/pom.xml,
>>> MavenProject: org.ovirt.engine.core:manager-tools:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/pom.xml,
>>> MavenProject: org.ovirt.engine.core:engine-tools-common:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-tools-common/pom.xml,
>>> MavenProject: org.ovirt.engine.core:common:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/pom.xml,
>>> MavenProject: org.ovirt.engine.core:utils:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/utils/pom.xml,
>>> MavenProject: org.ovirt.engine.core:dal:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/dal/pom.xml,
>>> MavenProject: org.ovirt.engine.core:beans:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/pom.xml,
>>> MavenProject: org.ovirt.engine.core:scheduler:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/scheduler/pom.xml,
>>> MavenProject: org.ovirt.engine.core:vdsbroker:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/vdsbroker/pom.xml,
>>> MavenProject: org.ovirt.engine.core:searchbackend:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/pom.xml,
>>> MavenProject: org.ovirt.engine.core:bll:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/bll/pom.xml,
>>> MavenProject: org.ovirt.engine.api:restapi-parent:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/pom.xml,
>>> MavenProject: org.ovirt.engine.api:interface:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/pom.xml,
>>> MavenProject: org.ovirt.engine.api:restapi-definition:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/definition/pom.xml,
>>> MavenProject: org.ovirt.engine.api:common-parent:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/pom.xml,
>>> MavenProject: org.ovirt.engine.api:interface-common-jaxrs:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/jaxrs/pom.xml,
>>> MavenProject: org.ovirt.engine.api:restapi-types:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/types/pom.xml,
>>> MavenProject: org.ovirt.engine.api:restapi-jaxrs:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/jaxrs/pom.xml,
>>> MavenProject: org.ovirt.engine.api:restapi-webapp:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/webapp/pom.xml,
>>> MavenProject: org.ovirt.engine.core:root-war:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/root/pom.xml,
>>> MavenProject: org.ovirt.engine.core:engine-config:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-config/pom.xml,
>>> MavenProject: org.ovirt.engine.core:engine-notifier:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/pom.xml,
>>> MavenProject: org.ovirt.engine.core:engine-notifier-service:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-service/pom.xml,
>>> MavenProject: org.ovirt.engine.core:engine-notifier-resources:3.2.0
>>> @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-resources/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:frontend-all:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:api:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:genericapi:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/genericapi/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:webadmin-all:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:webadmin-modules:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:gwt-extension:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:uicompat:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicompat/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:frontend:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/frontend/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:uicommonweb:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:gwt-common:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-common/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:webadmin:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/webadmin/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:userportal:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/userportal-gwtp/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:wars:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/pom.xml,
>>> MavenProject: org.ovirt.engine.ui:rmw-war:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/rmw-war/pom.xml,
>>> MavenProject: org.ovirt.engine:engine-server-ear:3.2.0 @
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/pom.xml]
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/target/uicommonweb-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/target/uicommonweb-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicommonweb/target/uicommonweb-3.2.0-sources.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/root/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/root/target/root-war-3.2.0.war
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/webapp/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/webapp/target/restapi-webapp-3.2.0.war
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-tools-common/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-tools-common/target/engine-tools-common-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-tools-common/target/engine-tools-common-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/target/common-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/target/common-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/common/target/common-3.2.0-sources.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-common/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-common/target/gwt-common-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-common/target/gwt-common-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/checkstyles/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/checkstyles/target/checkstyles-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/checkstyles/target/checkstyles-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/scheduler/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/scheduler/target/scheduler-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/beans/scheduler/target/scheduler-3.2.0-client.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-config/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-config/target/engine-config-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-config/target/engine-config-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-resources/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-resources/target/engine-notifier-resources-3.2.0.zip
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/genericapi/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/genericapi/target/genericapi-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/genericapi/target/genericapi-3.2.0-client.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/api/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/engineencryptutils/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/engineencryptutils/target/engineencryptutils-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/engineencryptutils/target/engineencryptutils-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-service/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-service/target/engine-notifier-service-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/tools/engine-notifier/engine-notifier-service/target/engine-notifier-service-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/utils/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/utils/target/utils-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/utils/target/utils-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/definition/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/definition/target/restapi-definition-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/definition/target/restapi-definition-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/types/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/types/target/restapi-types-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/types/target/restapi-types-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicompat/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicompat/target/uicompat-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/uicompat/target/uicompat-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/ovirt-checkstyle-extension/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/ovirt-checkstyle-extension/target/ovirt-checkstyle-extension-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/ovirt-checkstyle-extension/target/ovirt-checkstyle-extension-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/jaxrs/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/jaxrs/target/restapi-jaxrs-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/jaxrs/target/restapi-jaxrs-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/frontend/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/frontend/target/frontend-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/frontend/target/frontend-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/rmw-war/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/wars/rmw-war/target/rmw-war-3.2.0.war
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/dal/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/dal/target/dal-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/dal/target/dal-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/ear/target/engine.ear
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/webadmin/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/webadmin/target/webadmin-3.2.0.war
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/target/searchbackend-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/target/searchbackend-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/searchbackend/target/searchbackend-3.2.0-sources.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/target/gwt-extension-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/target/gwt-extension-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/gwt-extension/target/gwt-extension-3.2.0-sources.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/build-tools-root/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/target/compat-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/target/compat-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/compat/target/compat-3.2.0-sources.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/jaxrs/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/jaxrs/target/interface-common-jaxrs-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/restapi/interface/common/jaxrs/target/interface-common-jaxrs-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/userportal-gwtp/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/frontend/webadmin/modules/userportal-gwtp/target/userportal-3.2.0.war
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/vdsbroker/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/vdsbroker/target/vdsbroker-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/vdsbroker/target/vdsbroker-3.2.0-tests.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/bll/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/bll/target/bll-3.2.0.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/modules/bll/target/bll-3.2.0-client.jar
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/pom.xml
>>> [JENKINS] Archiving disabled - not archiving
>>> /home/jenkins/workspace/ovirt_engine_unit_tests/backend/manager/dbscripts/pom.xml
>>> Waiting for Jenkins to finish collecting data
>>> channel stopped
>>> IRC notifier plugin: Sending notification to: #ovirt-jenkins
>>> Email was triggered for: Unstable
>>> Sending email for trigger: Unstable
>>>
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/engine-devel
>>
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 



From mpastern at redhat.com  Sun Dec 30 09:41:46 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Sun, 30 Dec 2012 11:41:46 +0200
Subject: [Engine-devel] ovirt-sdk 3.2.0.6 released
In-Reply-To: <50ACB562.5040705@redhat.com>
References: <50ACB562.5040705@redhat.com>
Message-ID: <50E00C5A.6010502@redhat.com>


For more details see [1].

[1] http://wiki.ovirt.org/Python-sdk-changelog


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From mpastern at redhat.com  Sun Dec 30 09:42:01 2012
From: mpastern at redhat.com (Michael Pasternak)
Date: Sun, 30 Dec 2012 11:42:01 +0200
Subject: [Engine-devel] ovirt-cli 3.2.0.8 released
In-Reply-To: <50A8EA1F.700@redhat.com>
References: <50A8EA1F.700@redhat.com>
Message-ID: <50E00C69.80402@redhat.com>


For more details see [1].

[1] http://wiki.ovirt.org/Cli-changelog

-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From dfediuck at redhat.com  Mon Dec 31 15:05:01 2012
From: dfediuck at redhat.com (Doron Fediuck)
Date: Mon, 31 Dec 2012 10:05:01 -0500 (EST)
Subject: [Engine-devel] Engine local configuration (was: Error on
	starting webadmin)
In-Reply-To: <50CB0CC5.2030806@redhat.com>
Message-ID: <1061251071.59555727.1356966301141.JavaMail.root@redhat.com>


----- Original Message -----
> From: "Juan Hernandez" <jhernand at redhat.com>
> To: "Roy Golan" <rgolan at redhat.com>
> Cc: engine-devel at ovirt.org
> Sent: Friday, December 14, 2012 1:25:57 PM
> Subject: Re: [Engine-devel] Error on starting webadmin
> 
> On 12/13/2012 03:55 PM, Roy Golan wrote:
> > On 12/13/2012 04:49 PM, Vojtech Szocs wrote:
> >>> this is making the contribution process more complex. lets think
> >>> of a
> >>> lighter way to get a developing setup.
> >> I agree, I just wanted to have the local Engine configuration
> >> steps documented for reference.. If there's a better way to do
> >> it, I'm for it.
> >>
> >> Vojtech
> > 
> > having LocalConfig look for "engine.conf.defaults" system property
> > before fallback to System.getenv("ENGINE_DEFAULTS");
> >   +  concatenating
> >   -Dengine.conf.defaults=$HOME/.engine.conf.defaults
> > to  JAVA_OPTS  on standalone.conf
> 
> How is the system property simpler than the environment variable?
> 
> I agree that this makes the development process a bit more complex at
> the moment, but I think that the way to make it simpler is not to
> continue adding things to standalone.conf. I think that we should
> move
> towards a development environment that is closer to the production
> environment, not the other way around. Ideally the developer should
> be
> able to do something like "make install" to have the engine deployed
> to
> a directory structure similar to what he have in the production
> environment. Then you should be able to go to the bin directory
> inside
> that structure and start the engine (and the other tools) using the
> same
> script that we use in production environments. If we achieve this
> goal
> then we have a simple development environment setup and also we have
> all
> developers testing almost the same thing that will go into the
> production environments. At the moment we don't have that, most times
> you are testing something quite different (in terms of directory
> structure, configuration, etc) to what will be installed in
> production
> environments. I am working in that direction.
> 

Hi Guys,
Sorry to resume this discussion, but I find the current situation 
unfriendly to most developers. I understand there's a need for
specific configurations, but it seems to me that this has taken
one step too far for most developers.

Further more, I expect to see such fundamental concepts being
initially discussed here, and not settle with a technical ack,
only to be a part of a thread called: "Error on starting webadmin".
In this context I expect the verified flag to mean "This was
discussed and verified with contributers in the relevant list".

My use case is exactly what Juan describes, but I think the resolution
should be different. ie- contributers should be able to git clone
and once they setup jboss & postgres they should be able to have
something running.

We cannot assume /etc is a location contributers can access. Think
of university students who would like to see how ovirt works using
university's machines. In the past they may have had issues with
JBoss, but that's already handled by profiles. So there's really
no need for oVirt to ask them more than that. I know UI plugins
are using this infra, but remember that the plugins are completely
optional and should not block basic operation.

Some more concerns this issue raises are related to release versions;
What happens when we need to support more than one jboss and/or
release version?

So here are 2 options we should consider in order to allow local
config without additional requirements:

1. Use defaults.
System should boot and start in whichever way we decide without
anything else needed. Local configuration can override these
defaults if it exists, but it should simply work.

2. Use local JBoss profile.
Have everything we need in my-standalone.conf and start running jboss
with a local profile. This conf file may be generated or copied once
when running mvn -Psetup 

We can also inject the relevant environment variables to the global
standalone.conf as we do for debugging, but this cannot be used in
the students use case, unless we have host-level config.

I know many are still in vacation due to the holidays, so we should
wait for a few more days before making a final decision. Still feel
free to ack / nack / suggest your own.

Doron