[Engine-devel] Cloud-Init integration
by Greg Padgett
Hi Everyone,
I'd like to propose a feature we've been doing some investigation into,
which is to integrate cloud-init support into oVirt.
Cloud-init is used to help provision new Linux systems by setting the
hostname, ip, ssh keys, timezone, injecting files, and more. It's used by
OpenStack (amongst others) now, and has a lot of features that may be
helpful to our users.
Details are still evolving, but for more info please see the wiki page:
http://www.ovirt.org/Features/Cloud-Init_Integration
All feedback is welcome!
Thanks,
Greg
11 years, 5 months
[Engine-devel] Proposal VDSM <=> Engine Data Statistics Retrieval Optimization
by Vinzenz Feenstra
This is a multi-part message in MIME format.
--------------020305070807050907000005
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Please find the prettier version on the wiki:
http://www.ovirt.org/Proposal_VDSM_-_Engine_Data_Statistics_Retrieval
Proposal VDSM - Engine Data Statistics Retrieval
VDSM <=> Engine data retrieval optimization
Motivation:
Currently the RHEVM engine is polling the a lot of data from VDSM every
15 seconds. This should be optimized and the amount of data requested
should be more specific.
For each VM the data currently contains much more information than
actually needed which blows up the size of the XML content quite big. We
could optimize this by splitting the reply on the getVmStats based on
the request of the engine into sections. For this reason Omer Frenkel
and me have split up the data into parts based on their usage.
This data can and usually does change during the lifetime of the VM.
Rarely Changed:
This data is change not very frequent and it should be enough to update
this only once in a while. Most commonly this data changes after changes
made in the UI or after a migration of the VM to another Host.
*Status* = Running
*acpiEnable* = true
*vmType* = kvm
*guestName* = W864GUESTAGENTT
*displayType* = qxl
*guestOs* = Win 8
*kvmEnable* = true #/*this should be constant and never changed*/
*pauseCode* = NOERR
*monitorResponse* = 0
*session* = Locked # unused
*netIfaces* = [{'name': 'Realtek RTL8139C+ Fast Ethernet NIC', 'inet6': ['fe80::490c:92bb:bbcc:9f87'], 'inet': ['10.34.60.148'], 'hw': '00:1a:4a:22:3c:db'}]
*appsList* = ['RHEV-Tools 3.2.4', 'RHEV-Agent64 3.2.3', 'RHEV-Serial64 3.2.3', 'RHEV-Network64 3.2.2', 'RHEV-Network64 3.2.3', 'RHEV-Block64 3.2.3', 'RHEV-Balloon64 3.2.3', 'RHEV-Balloon64 3.2.2', 'RHEV-Agent64 3.2.2', 'RHEV-USB 3.2.3', 'RHEV-Block64 3.2.2', 'RHEV-Serial64 3.2.2']
*pid* = 11314
*guestIPs* = 10.34.60.148 # duplicated info
*displayIp* = 0
*displayPort* = 5902
*displaySecurePort* = 5903
*username* = user@W864GUESTAGENTT
*clientIp* =
*lastLogin* = 1361976900.67
Often Changed:
This data is changed quite often however it is not necessary to update
this data every 15 seconds. As this is cumulative data and reflects the
current status, and it does not need to be snapshotted every 15 seconds
to retrieve statistics. The data can be retrieved in much more generous
time slices. (e.g. Every 5 minutes)
*network* = {'vnet1': {'macAddr': '00:1a:4a:22:3c:db', 'rxDropped': '0', 'txDropped': '0', 'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '100', 'name': 'vnet1'}}
*disksUsage* = [{'path': 'c:\\', 'total': '64055406592', 'fs': 'NTFS', 'used': '19223846912'}, {'path': 'd:\\', 'total': '3490912256', 'fs': 'UDF', 'used': '3490912256'}]
*timeOffset* = 14422
*elapsedTime* = 68591
*hash* = 2335461227228498964
*statsAge* = 0.09 # unused
Often Changed but unused
This data does not seem to be used in the engine at all. It is *not*
even used in the data warehouse.
*memoryStats* = {'swap_out': '0', 'majflt': '0', 'mem_free': '1466884', 'swap_in': '0', 'pageflt': '0', 'mem_total': '2096736', 'mem_unused': '1466884'}
*balloonInfo* = {'balloon_max': 2097152, 'balloon_cur': 2097152}
*disks* = {'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '1754496', 'imageID': '28abb923-7b89-4638-84f8-1700f0b76482', 'flushLatency': '156549', 'readRate': '0.00', 'truesize': '18855059456', 'writeRate': '952.05'}, 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}
Very frequent uppdates needed by webadmin portal:
This data is mostly needed for the webadmin portal and might be required
to be updated quite often. An exception here is the statsAge field,
which seems to be unused by the Engine. This data could be requested
every 15 seconds to keep things as they are now.
*cpuSys* = 2.32
*cpuUser* = 1.34
*memUsage* = 30
Proposed Solution for VDSM & Engine:
We will introduce new optional parameters to getVmStats, getAllVmStats
and list to allow a finer grained specification of data which should be
included.
*Parameter:* *statsType*=/*<string>*/ (getVmStats, getAllVmStats only)
*Allowed values:*
* full (default to keep backwards compatibility)
* app-list (Just send the application list)
* rare (include everything from rarely changed to very frequent)
* often (include everything from often changed to very frequent)
* frequent (only send the very frequently changed items)
*Parameter:* *clientId*=*<string>* The client id is specified by the
client and should be unique however constantly used.
*Parameter:* *diff*=*<boolean>* In combination with the clientId VDSM
will send only differences to the previous request from the named
clientId. (if diff=true)
Additional Change:
Besides the introduction of the new parameters for list, getVmStats and
getAllVmStats it might make sense to include a hash for the appList into
the rarely changed section of the response which would allow to identify
changes and avoid having to sent the complete appList every so often and
only if the hash known to the client is outdated.
*Note:* The appList (Application List) reported by the guest agent could
be fully implemented on request only, as long as the guest agent
installed supports this. As there seems to be a request to have the
complete list of installed applications on all guests this data could be
quite extensive and a huge list. On the other hand this data is only
rarely visible and therefore it should not be requested all the time and
only on demand.
Improvement of the Guest Agent:
As part of the proposed solution it is necessary to improve the guest
agent as well. For the full application list there should be implemented
a caching system which will be fully reactive and should not poll the
application list for example all the time. The guest can create a
prepared data file containing all data in the JSON format (as used for
the communication with VDSM via VIO) and just have to read that file
from disk and directly sends it to VDSM. However it is quite possible
that this list is to big and it might have to be chunked into pieces.
(Multiple messages, which would have to be supported by VDSM then as
well) The solution for this is to make VDSM request this data and it
will retrieve the data necessary on request only.
--
Regards,
Vinzenz Feenstra | Senior Software Engineer
RedHat Engineering Virtualization R & D
Phone: +420 532 294 625
IRC: vfeenstr or evilissimo
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
--------------020305070807050907000005
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Please find the prettier version on the wiki:
<a class="moz-txt-link-freetext" href="http://www.ovirt.org/Proposal_VDSM_-_Engine_Data_Statistics_Retrieval">http://www.ovirt.org/Proposal_VDSM_-_Engine_Data_Statistics_Retrieval</a><br>
<br>
<h1 id="firstHeading" class="firstHeading page-header"><span
dir="auto">Proposal VDSM - Engine Data Statistics Retrieval</span></h1>
<h2><span class="mw-headline"
id="VDSM_.3C.3D.3E_Engine_data_retrieval_optimization">VDSM
<=> Engine data retrieval optimization </span></h2>
<h3> <span class="mw-headline" id="Motivation:"> Motivation: </span></h3>
<p>Currently the RHEVM engine is polling the a lot of data from VDSM
every 15 seconds. This should be optimized and the amount of data
requested should be more specific.
</p>
<p>For each VM the data currently contains much more information
than actually needed which blows up the size of the XML content
quite big. We could optimize this by splitting the reply on the
getVmStats based on the request of the engine into sections.
For this reason Omer Frenkel and me have split up the data into
parts based on their usage.
</p>
<p>This data can and usually does change during the lifetime of the
VM.
</p>
<h4> <span class="mw-headline" id="Rarely_Changed:"> Rarely
Changed: </span></h4>
<p>This data is change not very frequent and it should be enough to
update this only once in a while.
Most commonly this data changes after changes made in the UI or
after a migration of the VM to another Host.
</p>
<pre> <b>Status</b> = Running
<b>acpiEnable</b> = true
<b>vmType</b> = kvm
<b>guestName</b> = W864GUESTAGENTT
<b>displayType</b> = qxl
<b>guestOs</b> = Win 8
<b>kvmEnable</b> = true # <i><b>this should be constant and never changed</b></i>
<b>pauseCode</b> = NOERR
<b>monitorResponse</b> = 0
<b>session</b> = Locked # unused
<b>netIfaces</b> = [{'name': 'Realtek RTL8139C+ Fast Ethernet NIC', 'inet6': ['fe80::490c:92bb:bbcc:9f87'], 'inet': ['10.34.60.148'], 'hw': '00:1a:4a:22:3c:db'}]
<b>appsList</b> = ['RHEV-Tools 3.2.4', 'RHEV-Agent64 3.2.3', 'RHEV-Serial64 3.2.3', 'RHEV-Network64 3.2.2', 'RHEV-Network64 3.2.3', 'RHEV-Block64 3.2.3', 'RHEV-Balloon64 3.2.3', 'RHEV-Balloon64 3.2.2', 'RHEV-Agent64 3.2.2', 'RHEV-USB 3.2.3', 'RHEV-Block64 3.2.2', 'RHEV-Serial64 3.2.2']
<b>pid</b> = 11314
<b>guestIPs</b> = 10.34.60.148 # duplicated info
<b>displayIp</b> = 0
<b>displayPort</b> = 5902
<b>displaySecurePort</b> = 5903
<b>username</b> = user@W864GUESTAGENTT
<b>clientIp</b> =
<b>lastLogin</b> = 1361976900.67
</pre>
<h4> <span class="mw-headline" id="Often_Changed:"> Often Changed:
</span></h4>
<p>This data is changed quite often however it is not necessary to
update this data every 15 seconds. As this is cumulative data and
reflects the current status, and it does not need to be
snapshotted every 15 seconds to retrieve statistics. The data can
be retrieved in much more generous time slices. (e.g. Every 5
minutes)
</p>
<pre> <b>network</b> = {'vnet1': {'macAddr': '00:1a:4a:22:3c:db', 'rxDropped': '0', 'txDropped': '0', 'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '100', 'name': 'vnet1'}}
<b>disksUsage</b> = [{'path': 'c:\\', 'total': '64055406592', 'fs': 'NTFS', 'used': '19223846912'}, {'path': 'd:\\', 'total': '3490912256', 'fs': 'UDF', 'used': '3490912256'}]
<b>timeOffset</b> = 14422
<b>elapsedTime</b> = 68591
<b>hash</b> = 2335461227228498964
<b>statsAge</b> = 0.09 # unused
</pre>
<h4> <span class="mw-headline" id="Often_Changed_but_unused"> Often
Changed but unused </span></h4>
<p>This data does not seem to be used in the engine at all. It is <b>not</b>
even used in the data warehouse.
</p>
<pre> <b>memoryStats</b> = {'swap_out': '0', 'majflt': '0', 'mem_free': '1466884', 'swap_in': '0', 'pageflt': '0', 'mem_total': '2096736', 'mem_unused': '1466884'}
<b>balloonInfo</b> = {'balloon_max': 2097152, 'balloon_cur': 2097152}
<b>disks</b> = {'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '1754496', 'imageID': '28abb923-7b89-4638-84f8-1700f0b76482', 'flushLatency': '156549', 'readRate': '0.00', 'truesize': '18855059456', 'writeRate': '952.05'}, 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}
</pre>
<h4> <span class="mw-headline"
id="Very_frequent_uppdates_needed_by_webadmin_portal:"> Very
frequent uppdates needed by webadmin portal: </span></h4>
<p>This data is mostly needed for the webadmin portal and might be
required to be updated quite often. An exception here is the
statsAge field, which seems to be unused by the Engine.
This data could be requested every 15 seconds to keep things as
they are now.
</p>
<pre> <b>cpuSys</b> = 2.32
<b>cpuUser</b> = 1.34
<b>memUsage</b> = 30
</pre>
<h2> <span class="mw-headline"
id="Proposed_Solution_for_VDSM_.26_Engine:"> Proposed Solution
for VDSM & Engine: </span></h2>
<p>We will introduce new optional parameters to getVmStats,
getAllVmStats and list to allow a finer grained specification of
data which should be included.
</p>
<p><b>Parameter:</b> <b>statsType</b>=<i><b><string></b></i>
(getVmStats, getAllVmStats only)
<b>Allowed values:</b>
</p>
<ul>
<li> full (default to keep backwards compatibility)
</li>
<li> app-list (Just send the application list)
</li>
<li> rare (include everything from rarely changed to very
frequent)
</li>
<li> often (include everything from often changed to very
frequent)
</li>
<li> frequent (only send the very frequently changed items)
</li>
</ul>
<p><br>
<b>Parameter:</b> <b>clientId</b>=<b><string></b>
The client id is specified by the client and should be unique
however constantly used.
</p>
<p><b>Parameter:</b> <b>diff</b>=<b><boolean></b>
In combination with the clientId VDSM will send only differences
to the previous request from the named clientId. (if diff=true)
</p>
<p><br>
</p>
<h3> <span class="mw-headline" id="Additional_Change:"> Additional
Change: </span></h3>
<p>Besides the introduction of the new parameters for list,
getVmStats and getAllVmStats it might make sense to include a hash
for the appList into the rarely changed section of the response
which would allow to identify changes and avoid having to sent the
complete appList every so often and only if the hash known to the
client is outdated.
</p>
<p><b>Note:</b> The appList (Application List) reported by the guest
agent could be fully implemented on request only, as long as the
guest agent installed supports this. As there seems to be a
request to have the complete list of installed applications on all
guests this data could be quite extensive and a huge list. On the
other hand this data is only rarely visible and therefore it
should not be requested all the time and only on demand.
</p>
<h3> <span class="mw-headline" id="Improvement_of_the_Guest_Agent:">
Improvement of the Guest Agent: </span></h3>
<p>As part of the proposed solution it is necessary to improve the
guest agent as well. For the full application list there should be
implemented a caching system which will be fully reactive and
should not poll the application list for example all the time.
The guest can create a prepared data file containing all data in
the JSON format (as used for the communication with VDSM via VIO)
and just have to read that file from disk and directly sends it to
VDSM.
However it is quite possible that this list is to big and it might
have to be chunked into pieces. (Multiple messages, which would
have to be supported by VDSM then as well)
The solution for this is to make VDSM request this data and it
will retrieve the data necessary on request only.
</p>
<pre class="moz-signature" cols="72">--
Regards,
Vinzenz Feenstra | Senior Software Engineer
RedHat Engineering Virtualization R & D
Phone: +420 532 294 625
IRC: vfeenstr or evilissimo
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com</pre>
</body>
</html>
--------------020305070807050907000005--
11 years, 5 months
[Engine-devel] [ANN] Development environment and Gentoo packaging (preview)
by Alon Bar-Lev
Hello All,
We[1] are working to rewrite the current installer to something more portable and flexible.
Currently two milestones had been reached:
1. Ability to install almost fully functioning ovirt-engine at $HOME for development environment.
2. Porting ovirt-engine to run on different distribution, Gentoo.
We will be glad to receive feedback on either.
Code is located at github[2] for now, at otopi branch.
Instructions for setting up development environment are available[3].
Gentoo overlay for live ebuilds is available[4], the following packages are valid:
- app-emulation/otopi
- app-emulation/ovirt-host-deploy
- app-emulation/ovirt-jboss-bin
- app-emulation/ovirt-engine
Why Gentoo first? because source based distributions demands the highest level of customization, solving the complex issue ease to continue porting to binary based distributions.
Please remember that this is work in progress, and not guarantee to be stable or even work... The installer was re-written from scratch so expect issues at this point.
Any feedback is welcomed, we are focusing first in providing the functionality of the existing installer to be able to replace it entirely before going into new adventures.
Regards,
Alon Bar-Lev.
[1] Alex Lourie, Sandro Bonazzola, Alon Bar-Lev
[2] https://github.com/alonbl/ovirt-engine
[3] https://github.com/alonbl/ovirt-engine/blob/otopi/README.developer
[4] https://github.com/alonbl/ovirt-overlay
11 years, 6 months
[Engine-devel] Migrating engine-setup to otopi
by Alex Lourie
Hi All
Recent development of the otopi [1] framework allows us to migrate the engine-setup,
upgrade and cleanup (and potentially other) utilities to implementation as an otopi
plugin.
Potential benefits of such a move are:
1. Be able to port engine to other distributions.
2. Be able to install engine in a development mode.
3. Be able to customize installation easily.
4. Share installation of components (reports, dwh).
5. Modular implementation, reduce maintenance costs.
6. Code reuse of installer code for multiple purposes (host-deploy,
enigne-setup).
Currently we are in the process of creating a 'setup' plugin for
the otopi, and the progress can be monitored at [2]. The current
roadmap for the feature is as follows:
1. Recreate the configuration utilities as plugins in otopi.
2. Support side-by side installation using both the old and the
new utilities.
3. Switch to the new utility when the confidence that it is on-par
with an old one is high.
Our goal is to have the new utilities ready for 3.3 release (at
least for the step 2 in the roadmap).
We'd like to hear as much feedback as possible, so we could
address it as soon as possible.
Thanks!
[1] http://gerrit.ovirt.org/#/q/project:otopi,n,z
[2] http://www.ovirt.org/Features/Otopi_Infra_Migration
11 years, 6 months
[Engine-devel] Any way to setup ovirt engine 3.2.0 or 3.2.1 running enviroment?
by Chen, Wei D
Hi,
We notice that ovirt-engine's version (rpm package and source code) has updated from 3.2.0 to 3.2.1. As our development process is always based on 3.2.0. Is there any way to re-setup 3.2.0 running environment?
Besides, we also cannot setup 3.2.1 ovrit-engine running environment as its dependent package glusterfs-3.4 cannot be found from ovirt repo or fedora18 repo. How can I fix this issue? Thanks.
Best Regards,
Dave Chen
11 years, 6 months
[Engine-devel] FeatureSupported and compatibility versions
by Shireesh Anjal
Hi all,
The current mechanism in oVirt to check whether a feature is supported
in a particular compatibility version is to use the FeatureSupported
class. e.g.
FeatureSupported.networkLinking(getVm().getVdsGroupCompatibilityVersion())
Checks whether the "network linking" feature is supported for the the
VM's cluster compatibility version. This internally checks whether the
value of the corresponding config (NetworkLinkingSupported) for the
given compatibility version is true/false.
I'm not sure if this is a good idea, since a feature is typically
supported "from" a particular version. E.g. Gluster support was
introduced in 3.1, and it continues to be available in all subsequent
versions. So I see no point in adding configuration for every version
indicating whether the feature is supported in that version or not. I
suggest to use either of the following options:
1) Instead of using a boolean config for each version, use a single
string config that indicates the "supported from" version e.g.
GlusterSupportedFrom = 3.1. There could be rare cases where a feature,
for some reason, is removed in some release. In such cases, we could use
one additional config for the "supported to" version.
2) Continue with the boolean approach, but do not have entries for every
version; rather make use of the "default value" for majority of cases,
and add the explicit version mapping for the minority e.g.
GlusterSupported = true by default, and false in case of 3.0 (only one
config required for 3.0)
Thoughts?
Regards,
Shireesh
11 years, 6 months
[Engine-devel] UI Plugin API improvements
by Vojtech Szocs
Hi guys,
I've just merged some UI Plugin patches that improve existing API functions, as well as add some new API functions. Please read on to learn what's new.
Modal dialog API
================
Function improved: showDialog
New signature:
showDialog(title, dialogToken, contentUrl, width, height [, options])
Example usage:
showDialog('My Dialog', 'my-dialog', 'http://www.foobar.com/', '800px', '600px', {
// Default value = empty array (no buttons)
buttons: [
{
label: 'Do stuff',
onClick: function() {
alert('Bump!');
}
}
],
// Default value = false
resizeEnabled: true,
// Default value = true
closeIconVisible: true,
// Default value = true
closeOnEscKey: true
});
Notable changes:
* modal dialogs now look & feel the same as standard WebAdmin dialogs
* width & height are strings containing CSS units
* the reason why buttons default to empty array is to give plugin authors the choice to provide custom buttons (or similar input elements) via dialog content (iframe), and use HTML5 window.postMessage to call the plugin (coming soon!)
--
New function: setDialogContentUrl
New signature:
setDialogContentUrl(dialogToken, contentUrl)
Example usage:
setDialogContentUrl('my-dialog', 'http://www.example.com/')
--
New function: closeDialog
New signature:
closeDialog(dialogToken)
Example usage:
closeDialog('my-dialog')
Tab API
=======
Functions improved: addMainTab & addSubTab
New signatures:
addMainTab(label, historyToken, contentUrl [, options])
addSubTab(entityTypeName, label, historyToken, contentUrl [, options])
Example usage:
// Tab is left-aligned by default
addMainTab('Foo Tab', 'foo-tab', 'http://www.foo.com/');
// Tab is right-aligned via options object
addSubTab('VirtualMachine', 'Bar Tab', 'bar-tab', 'http://www.bar.com/', {
alignRight: true
});
--
Regards,
Vojtech
11 years, 7 months
[Engine-devel] Open Attestation integration with oVirt engine proposal has submitted patchset5 for your review
by Chen, Wei D
Hi all,
Before submitting this patch set, we has updated our design page, and new feature about VM template has added to this patchset. In patchset a lot of frontend changes has been imported.
Welcome to review our patchset and thanks advance for your suggestion.
Detailed description: http://wiki.ovirt.org/Trusted_compute_pools
In this patch set, follow changes has been introduced:
1. GUI changes to support for creating a trusted VM on a trusted physical host.
2. View/Edit VM changes to enable end user switch between three run on options.
3. Template relevant changes to support end user create a trusted VM template and create trusted VM based on this template afterwards.
4. Bug fixing and code cleanup.
5. wiki design page update.
Best Regards,
Dave Chen
11 years, 7 months
[Engine-devel] Move SQL out of stored procedures
by Juan Hernandez
Hello,
I would like to start a discussion about the subject. I think this is
something we need to do if one day we want to be able to use any
database other than PostgreSQL.
I did an small example of what it takes and how it looks like to have
the SQL code into the DAOs:
http://gerrit.ovirt.org/13347
It isn't rocket science, it isn't an exciting task, it isn't fun, but
something I think we should eventually do.
I appreciate any comment about how and when to do this, including those
saying that instead of this primitive approach we should use this or
that ORM framework.
Regards,
Juan Hernandez
--
Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
3ºD, 28016 Madrid, Spain
Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.
11 years, 7 months
[Engine-devel] Sandro Bonazzola as a tools maintainer
by Keith Robertson
I would like to propose that we add Sandro Bonazzola as a maintainer for
the Ovirt ISO Uploader, Image Uploader and Log Collector. He too has
been enormously helpful with the maintenance of the tools.
Thanks,
Keith Robertson
11 years, 7 months
