- Devel - oVirt List Archives

[Engine-devel] CPU Overcommit Feature
by Greg Padgett 20 Dec '12

20 Dec '12

Hi, I've been working on a feature to allow CPU Overcommitment of hosts in a cluster. This first stage allows the engine to consider host cpu threads as cores for the purposes of VM resource allocation. This wiki page has further details, your comments are welcome! http://www.ovirt.org/Features/cpu_overcommit Thanks in advance, Greg

7 18

[Engine-devel] oVirt-Foreman UI plugin
by Oved Ourfalli 20 Dec '12

20 Dec '12

Hey all, I've been working on an oVirt-Foreman UI-plugin recently, using oVirt's new UI-plugin infrastructure. It allows users to see Foreman data on oVirt entities, from inside Webadmin. I wrote a blog post describing this plugin, what can he do, some challenges that I faced and solutions, some screenshots, future work and more. Check it out at: ovedou.blogpost.com I'll be happy to hear your feedback both on the blog and on the plugin itself! If you need help installing the plugin feel free to contact me :-) Cheers, Oved

3 4

[Engine-devel] Proposal to add Shireesh Anjal as maintainer to engine-gluster
by Itamar Heim 20 Dec '12

20 Dec '12

Shireesh has been working on the engine gluster support for the past year, covering patches and leading reviews of the various engine gluster patches. I'd like to propose a new sub-component of the engine for the gluster module, adding shireesh as its maintainer. Thanks, Itamar

5 5

[Engine-devel] LDAP: Add support for simple authentication over SSL
by snmishra＠linux.vnet.ibm.com 19 Dec '12

19 Dec '12

Hi, IBM Tivoli Directory Server (ITDS) supports simple authentication over SSL. What will it take to add this support? I can help with this work item but will need some guidance. Regards Sharad Mishra

2 4

[Engine-devel] [SUGGESTION] Defining a process for the new feature discussion/implementation.
by Michael Pasternak 19 Dec '12

19 Dec '12

Hi All, In many cases OSS maintainers not always can be in the loop of different threads what may cause them missing important decisions being taken, As result later on during reviews of the patches they're not accepting (already implemented) features, what is causing not once for feature to be re-designed and/or delayed, what is wrong from the development cycle PoV. Therefore I'd like to suggest establishing dev-rules for the new feature implementation, what will make entire process much more easer for all of us: 1. discuss new feature on the mailing list (requirements/constraints/etc.) 2. summarise feature details in feature-doc 3. send feature-doc to review to: 3.1 ML (engine-devel) 3.2 MG (mailing group of maintainers of the relevant layers) 4. after feature-doc is accepted, 4.1 implement the feature 4.2 send it to gerrit for review to: 4.2.1 lead maintainer/s (they will review/delegate it) NOTE 3.2, 4.2.1 will require defining MGs such as: - engine-devel-core - engine-devel-ui - engine-devel-api - engine-devel-sdk - engine-devel-cli - engine-devel-vdsm ... Thoughts? -- Michael Pasternak RedHat, ENG-Virtualization R&D

1 0

[Engine-devel] Revisiting Java7
by Allon Mureinik 18 Dec '12

18 Dec '12

Hi guys, Earlier today, Java6 compatibility was broken (http://gerrit.ovirt.org/#/c/9430/) This was picked up on pretty quickly, and easily fixed (http://gerrit.ovirt.org/#/c/9666/) However, I think this is a good opportunity to revisit our policy towards Java 7. Currently, we have an "odd" setup, where we recommend running a compiling /with/ Java 7 [1] but comply to Java 6 language level [2] and JDK [3]. Inevitably, mistakes like the that happened today will happen. I know we're holding back due to GWT issues, but looking forward to oVirt 4.0, is Java 7 on our roadmap? -Allon [1] http://wiki.ovirt.org/Building_oVirt_engine#Installing_OpenJDK [2] maven-compiler-plugin section in ${OVIRT_GIT}/pom.xml [3] http://jenkins.ovirt.org/job/ovirt_engine_animal_sniffer_check/ P.S. If you want to check that you aren't breaking Java/JDK 6 compatibility locally without installing java 6, you can run "mvn animal-sniffer:check". Note that animal sniffer analyzes binaries, so this has to be done /after/ the project was built. Of course, you can do this in a single line "mvn install animal-sniffer:check"

7 10

[Engine-devel] [Draft]Task Management API
by Saggi Mizrahi 18 Dec '12

18 Dec '12

Dan rightly suggested I'd be more specific about what the task system is instead of what the task system isn't. The problem is that I'm not completely sure how it's going to work. It also depends on the events mechanism. This is my current working draft: TaskInfo: id string methodName string kwargs json-object (string keys variant values) *filtered to remove sensitive information getRunningTasks(filter string, filterType enum{glob, regexp}) Returns a list of TaskInfo of all tasks that their id's match the filter That's it, not even stopTask() As explained, I would like to offload handling to the subsystems. In order to make things easier for the clients every subsystem can choose a filed of the object to be of type OperationInfo. This is a generic structure that the user has a generic way to track all tasks on all subsystem with a report interface. The extraData field is for subsystem specific data. This is where the storage subsystem would put, for example, imageState (broken, degraded, optimized) data. OperationInfo: operationDescription string - something out of an agreed enum of strings vaguely describing the operation at hand for example "Copying", "Merging", "Deleting", "Configuring", "Stopped", "Paused", .... They must be known to the client so it can in turn translate it in the UI. The also have to remain relatively vague as they are part of the interface meaning that new values will break old clients so they have to be reusable. stageDescription - Similar to operation description in case you want more granularity, optional. stage (int, int) - (5, 10) means 5 out of 10. 1 out of 1 implies the UI to not display stage widgets. percentage - 0-100, -1 means unknown. lastError - (code, message) the same errors that can return for regular calls extraData - json-object For example creatVM will return once the object is created in VDSM. getVmInfo() would return, amongst other things, the operation info. For the case of preparing for launch it will be: {"Creating", "configuring", (2, 4), 40, (0, ""), {state="preparing for launch"}} In the case of VM paused on EIO: {"Paused", "Paused", (1, 1), -1, (123, "Error writing to disks"), {state="paused"}} Migration is a tricky one, it will be reported as a task while it's in progress but all the information is available on the image operationInfo. In the case of Migration: {"Migration", "Configuring", (1, 3), -1, (0, ""), {status="Migrating"}} For StorageConnection this is somewhat already the case but in simplified version. If you want to ask about any other operation I'd be more then happy to write my suggestion for it. Subsystems have complete freedom about how to set up the API. For Storage you have Fixes() to start\stop operations. Gluster is pretty autonomous once operations have been started. Since operations return as soon as they are registered (persisted) or fail to register, it makes synchronous programming a bit clunky. vdsm.pauseVm(vmId) doesn't return when the VM is paused but when VDSM committed it will try to pause it. This means you will have to poll in order to see if the operation finished. For gluster, as an example, this is the only way we can check that the operation finished. For stuff we have a bit more control over vdsm will fire events using json-rpc notifications sent to the clients. The will be in the form of: {"method": "alert", "params": { "alertName": <subsystem>(.<objectType>)?.<object>.(<subobject>., ...), "operationInfo", OperationInfo} } The user can register to recive events using a glob or a regexp. registering to vdsm.VM.* pop every time any VM has changed stage. This means that whenever the task finishes, fails or gains significance progress and VDSM is there to track it, an event will be sent to the client. This means that the general flow is. # Register operation vmID = best_vm host.VM.pauseVM(vmID) while True: opInfo = None try: event = host.waitForEvent("vdsm.VM.best_vm", timeout=10) opInfo = event.opInfo except VdsmDisconnectionError: host.waitForReconnect() host.vm.getVmInfo(vmID) # Double check that we didn't miss the event continue except Timeout: # This is a long operation, poll to see that we didn't miss any event # but more commonly, update percentage in the UI to show progress. vmInfo = host.vm.getVmInfo(vmID) opInfo = vmInfo.operationInfo if opInfo.stage.number != op.stage.total: # Operation in progress updateUI(opInfo) else: # Operation completed # Check that the state is what we expected it to be. if oInfo.extraData.state == "paused": return SUCCESS else: return opInfo.lastError vdsm.waitForEvent(filterm, timeout) is a client side libvdsm helper operation. Clients that access the raw API need to create thir own client side code to filter out events and manage their distribution. I'm open to also defining server side filters but I'm not sure whether it's worth it or just having it be a boolean (all events or none) is sufficient. This is a very simplified example but the general flow is clear. Even if the connection is lost for 1 second or 4 days, the code still works. Further more, the user can wait for multiple operations in the same thread using: host.waitForEvent("vdsm.VM.(best_vm_ever|not_so_good_vm)") This means that the client can wait for a 100 VMs or all VMs (using wildecards) in a mechanism similar "poll()" with minimal overhead. This also means that if The fact that operations are registered means that even if connections is lost due to VDSM crashing or the network crashing, the manager doesn't need to care one the original command returns as it know the operation registered. This doesn't mean that every operation must retry forever. How persistent every method is can and should change between the different operations. Also, it means that manager that didn't initiate an operation track it in the same way as those that did. This makes clustered managers a lot easier to implement as if one goes down a second one can more or less immediately with minimal extra code.

2 1

[Engine-devel] Managing async tasks
by Adam Litke 17 Dec '12

17 Dec '12

On today's vdsm call we had a lively discussion around how asynchronous operations should be handled in the future. In an effort to include more people in the discussion and to better capture the resulting conversation I would like to continue that discussion here on the mailing list. A lot of ideas were thrown around about how 'tasks' should be handled in the future. There are a lot of ways that it can be done. To determine how we should implement it, it's probably best if we start with a set of requirements. If we can first agree on these, it should be easy to find a solution that meets them. I'll take a stab at identifying a first set of POSSIBLE requirements: - Standardized method for determining the result of an operation This is a big one for me because it directly affects the consumability of the API. If each verb has different semantics for discovering whether it has completed successfully, then the API will be nearly impossible to use easily. Sorry. That's my list :) Hopefully others will be willing to add other requirements for consideration. >From my understanding, task recovery (stop, abort, rollback, etc) will not be generally supported and should not be a requirement. -- Adam Litke <agl(a)us.ibm.com> IBM Linux Technology Center

3 7

Re: [Engine-devel] [vdsm] Host bios information
by Barak Azulay 16 Dec '12

16 Dec '12

----- Original Message ----- > From: "Ayal Baron" <abaron(a)redhat.com> > To: "Saggi Mizrahi" <smizrahi(a)redhat.com> > Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org> > Sent: Friday, December 14, 2012 2:07:04 AM > Subject: Re: [vdsm] Host bios information > > > > ----- Original Message ----- > > > > > > ----- Original Message ----- > > > From: "Ayal Baron" <abaron(a)redhat.com> > > > To: "Saggi Mizrahi" <smizrahi(a)redhat.com> > > > Cc: "VDSM Project Development" > > > <vdsm-devel(a)lists.fedorahosted.org>, > > > "Shu Ming" <shuming(a)linux.vnet.ibm.com> > > > Sent: Thursday, December 13, 2012 11:30:56 AM > > > Subject: Re: [vdsm] Host bios information > > > > > > > > > > > > ----- Original Message ----- > > > > I think that for the new current XML-RPC API it's OK to add it > > > > to > > > > the > > > > getVdsCaps() verb. > > > > For the new API I suggest moving it to it's own API. The > > > > smaller > > > > the > > > > APIs the easier they are to deprecate and support. > > > > I quite doubt the fields in getBiosInfo() will change half as > > > > frequently as whatever getVdsCaps() returns. > > > > I also kind of want to throw away getVdsCaps() and split it to > > > > better > > > > named better encapsulated methods. > > > > > > Ack. I just don't understand why not start right now? > > > Any new patch should improve things at least a little. > > > We know getVdsCaps() is wrong so let's put the bios info (and > > > anything in getVdsCaps that makes sense to put with it if > > > relevant) > > > in a separate call. Adding a call in engine to this new method > > > should be a no brainer, I don't think that is a good reason for > > > not > > > doing things properly in vdsm, even if we're talking about the > > > current API. > > Well, from what I know the current overhead per call is too large > > to > > mandate a lot of calls. > > At least that is what I've been told. If that is not an issue, do > > it > > in the XML-RPC API too. > > if you call it every 2 seconds to each host then perhaps, but > getVdsCaps is called in initvdsonup which is pretty rare (hours, > days or more) so avoiding an extra call there seems wrong to me. I totally agree that in the long run it is correct to add a different API on vdsm, However for the initial 6 bios params: 1. Host Manufacturer - Manufacturer of the host's machine and bios' vendor (e.g LENOVO) 2. Host Version - For each host the manufacturer gives a unique name (e.g. Lenovo T420s) 3. Host Product Name - ID of the product - same for all similar products (e.g 4174BH4) 4. Host UUID - Unique ID for each host (e.g E03DD601-5219-11CB-BB3F-892313086897) 5. Host Family - Type of host's CPU - (e.g Core i5) 6. Host Serial Number - Unique ID for host's chassis (e.g R9M4N4G) As Dan stated below, is o.k. for this specific phase to add them to getVdsCaps So I suggest: 1 Adding the above 6 above params to getVdsCaps 2 Add a new API to vdsm to retrieve host bios information - that will eventually return the entire list (current will retrieve the exact 6 params and will use the same underlying implementation) 3 In the future engine will start using the new API Start using the new API now on the engine side will require much more testing as it might have an influence on the host state machine. Thanks Barak Azulay > > > > > > > > > > > > Also, in the json-rpc base model, calls are not only cheaper, > > > > you > > > > also have batch calls. This means you can send multiple > > > > requests > > > > as > > > > one message and have VDSM send you the responses as one message > > > > once > > > > all tasks completed. This makes splitting aggregated methods to > > > > smaller methods painless and with minimal overhead. > > > > > > > > ----- Original Message ----- > > > > > From: "Shu Ming" <shuming(a)linux.vnet.ibm.com> > > > > > To: "ybronhei" <ybronhei(a)redhat.com> > > > > > Cc: "VDSM Project Development" > > > > > <vdsm-devel(a)lists.fedorahosted.org> > > > > > Sent: Thursday, December 13, 2012 11:04:09 AM > > > > > Subject: Re: [vdsm] Host bios information > > > > > > > > > > After a quick review of the wiki page, it was stated that > > > > > dmidecode > > > > > gave > > > > > too much informations. Only five fields will be displayed in > > > > > the > > > > > hardware tab, "Manufactory", "Version", "Family", "UUID" and > > > > > "serial > > > > > number". For "Family", it is mean the CPU core's family. > > > > > And > > > > > it > > > > > confuses me a bit with the "CPU name" and "CPU type" fields > > > > > in > > > > > general > > > > > tab. I think we should chose the best one to characterizethe > > > > > CPU > > > > > type. > > > > > > > > > > > > > > > ybronhei: > > > > > > Today in the Api we display general information about the > > > > > > host > > > > > > that > > > > > > vdsm export by getCapabilities Api. > > > > > > > > > > > > We decided to add bios information as part of the > > > > > > information > > > > > > that > > > > > > is > > > > > > displayed in UI under host's general sub-tab. > > > > > > > > > > > > To summaries the feature - We'll modify General tab to > > > > > > Software > > > > > > Information and add another tab for Hardware Information > > > > > > which > > > > > > will > > > > > > include all the bios data that we'll decide to gather from > > > > > > the > > > > > > host > > > > > > and display. > > > > > > > > > > > > Following this feature page: > > > > > > http://www.ovirt.org/Features/Design/HostBiosInfo for more > > > > > > details. > > > > > > All the parameters that can be displayed are mentioned in > > > > > > the > > > > > > wiki. > > > > > > > > > > > > I would greatly appreciate your comments and questions. > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > -- > > > > > --- > > > > > 舒明 Shu Ming > > > > > Open Virtualization Engineerning; CSTL, IBM Corp. > > > > > Tel: 86-10-82451626 Tieline: 9051626 E-mail: > > > > > shuming(a)cn.ibm.com > > > > > or > > > > > shuming(a)linux.vnet.ibm.com > > > > > Address: 3/F Ring Building, ZhongGuanCun Software Park, > > > > > Haidian > > > > > District, Beijing 100193, PRC > > > > > > > > > > > > > > > _______________________________________________ > > > > > vdsm-devel mailing list > > > > > vdsm-devel(a)lists.fedorahosted.org > > > > > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel > > > > > > > > > _______________________________________________ > > > > vdsm-devel mailing list > > > > vdsm-devel(a)lists.fedorahosted.org > > > > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel > > > > > > > > > > _______________________________________________ > vdsm-devel mailing list > vdsm-devel(a)lists.fedorahosted.org > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel >

3 2

[Engine-devel] VDSM tasks, the future
by Saggi Mizrahi 16 Dec '12

16 Dec '12

Because I started hinting about how VDSM tasks are going to look going forward I thought it's better I'll just write everything in an email so we can talk about it in context. This is not set in stone and I'm still debating things myself but it's very close to being done. - Everything is asynchronous. The nature of message based communication is that you can't have synchronous operations. This is not really debatable because it's just how TCP\AMQP\<messaging> works. - Task IDs will be decided by the caller. This is how json-rpc works and also makes sense because no the engine can track the task without needing to have a stage where we give it the task ID back. IDs are reusable as long as no one else is using them at the time so they can be used for synchronizing operations between clients (making sure a command is only executed once on a specific host without locking). - Tasks are transient If VDSM restarts it forgets all the task information. There are 2 ways to have persistent tasks: 1. The task creates an object that you can continue work on in VDSM. The new storage does that by the fact that copyImage() returns one the target volume has been created but before the data has been fully copied. From that moment on the stat of the copy can be queried from any host using getImageStatus() and the specific copy operation can be queried with getTaskStatus() on the host performing it. After VDSM crashes, depending on policy, either VDSM will create a new task to continue the copy or someone else will send a command to continue the operation and that will be a new task. 2. VDSM tasks just start other operations track-able not through the task interface. For example Gluster. gluster.startVolumeRebalance() will return once it has been registered with Gluster. glster.getOperationStatuses() will return the state of the operation from any host. Each call is a task in itself. - No task tags. They are silly and the caller can mangle whatever in the task ID if he really wants to tag tasks. - No explicit recovery stage. VDSM will be crash-only, there should be efforts to make everything crash-safe. If that is problematic, in case of networking, VDSM will recover on start without having a task for it. - No clean Task: Tasks can be started by any number of hosts this means that there is no way to own all tasks. There could be cases where VDSM starts tasks on it's own and thus they have no owner at all. The caller needs to continually track the state of VDSM. We will have brodcasted events to mitigate polling. - No revert Impossible to implement safely. - No SPM\HSM tasks SPM\SDM is no longer necessary for all domain types (only for type). What used to be SPM tasks, or tasks that persist and can be restarted on other hosts is talked about in previous bullet points.

4 5