<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Please note that the approach described here is outdated, because
xmlpickle was too easy to use</p>
<p>and too generic: that lead to bloated XML and too high risk of
misusing metadata;</p>
<p>the new approach (<a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/74206/">https://gerrit.ovirt.org/#/c/74206/</a>) tries to
balance things making usage</p>
<p>convenient but disallowing arbitrarily nesting<br>
</p>
<br>
<div class="moz-cite-prefix">On 03/18/2017 03:27 PM, Nir Soffer
wrote:<br>
</div>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div dir="ltr" class="gmail_msg">On Wed, Mar 15, 2017 at
2:28 PM Francesco Romani <<a moz-do-not-send="true"
href="mailto:fromani@redhat.com" class="gmail_msg"
target="_blank">fromani@redhat.com</a>> wrote:<br
class="gmail_msg">
</div>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi
everyone,<br class="gmail_msg">
<br class="gmail_msg">
This is both a report of the current state of my Vdsm
patches for Engine<br class="gmail_msg">
XML support, and a proposal to how move forward and solve<br
class="gmail_msg">
the current open issues.<br class="gmail_msg">
<br class="gmail_msg">
TL;DR:<br class="gmail_msg">
1. we can and IMO should reuse the current JSON schema to
describe the<br class="gmail_msg">
structure (layout) and the types of the metadata section.<br
class="gmail_msg">
2. we don't need a priori validation of stuff in the
metadata section.<br class="gmail_msg">
We will just raise in the creation flow if data is
missing, or wrong,<br class="gmail_msg">
according to our schema.<br class="gmail_msg">
2. we will add *few* items to the metadata section, only
thing we can't<br class="gmail_msg">
express clearly-or at all in the libvirt XML. Redundancy
and verbosiness<br class="gmail_msg">
will be thus kept at bay<br class="gmail_msg">
3. I believe [3] is the best tool to do (de)serialize data
to the<br class="gmail_msg">
metadata section. Existing tools fits poorly in our very
specific use case<br class="gmail_msg">
<br class="gmail_msg">
Examples below<br class="gmail_msg">
<br class="gmail_msg">
+++<br class="gmail_msg">
<br class="gmail_msg">
Long(er) discussion:<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
I have working code[1][2] to encode any custom, picklable,
python<br class="gmail_msg">
object in the metadata section.<br class="gmail_msg">
<br class="gmail_msg">
We should decide which module will do the actual
python<=>XML<br class="gmail_msg">
transformation.<br class="gmail_msg">
Please note that this actually also influences how the
data in the<br class="gmail_msg">
medata section look like, so the two things are a bit
coupled.<br class="gmail_msg">
<br class="gmail_msg">
I'm eager to reinvent another wheel, but after<br
class="gmail_msg">
initial evaluation I honestly think that my pyxmlpickle[3]
is the best<br class="gmail_msg">
tool for the job over the current alternatives:
plistlib[4] and<br class="gmail_msg">
xmltodict[5].<br class="gmail_msg">
<br class="gmail_msg">
I added the initial rationale here:<br class="gmail_msg">
<a moz-do-not-send="true"
href="https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG"
rel="noreferrer" class="gmail_msg" target="_blank">https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG</a><br
class="gmail_msg">
<br class="gmail_msg">
I have completed the initial draft of patches to make it
possible to<br class="gmail_msg">
initialize devices from their XML representation [6]. This
is the bare<br class="gmail_msg">
minimum we need to support the Engine XML, and we *need*
this anyway to<br class="gmail_msg">
unlock the cleanup we planned and I outlined in my google
doc.<br class="gmail_msg">
<br class="gmail_msg">
So we are progressing, but I'd like to speed up things.
Those [6]<br class="gmail_msg">
patches are not yet complete, many flows are not covered
or tested; but<br class="gmail_msg">
they are good enough to demonstrate that there *are*
pieces of<br class="gmail_msg">
information wen need to properly initialize the devices,
but we can't<br class="gmail_msg">
easily extract from the XML.<br class="gmail_msg">
<br class="gmail_msg">
First examples that come to my mind are the storage.Drive
UUIDs; there<br class="gmail_msg">
could also be some ambiguity I'm investigating right now
for<br class="gmail_msg">
displayIp/displayNetwork in Graphics devices. In [6] there
are various<br class="gmail_msg">
TODO to mark more of those cases. Most likely, few more
cases will pop<br class="gmail_msg">
out as I cover all the flows we support.<br
class="gmail_msg">
<br class="gmail_msg">
Long story short: it is hard to correctly rebuild the
device conf from<br class="gmail_msg">
the XML. This is why in [6] I added the 'meta' argument to
from_xml_tree<br class="gmail_msg">
classmethod in [7].<br class="gmail_msg">
<br class="gmail_msg">
'meta' is supposed to be the device metadata: extra data
related to a<br class="gmail_msg">
device which doesn't (yet) fit in the libvirt XML
representation.<br class="gmail_msg">
For example, we can store 'displayIp' and 'displayNetwork'
here and be<br class="gmail_msg">
done with that: using both per-device metadata and the XML<br
class="gmail_msg">
representation of one graphic device, we will have
everything we need to<br class="gmail_msg">
properly build one graphics.Graphics device.<br
class="gmail_msg">
This example may (hopefully) be bogus, but I'm keeping it
because it is<br class="gmail_msg">
one case easy to follow.<br class="gmail_msg">
<br class="gmail_msg">
The device metadata is going to be stored in the vm
metadata for the<br class="gmail_msg">
short/mid term future. Even if the per-device metadata
idea/RFE is<br class="gmail_msg">
accepted (no answer yet, but we are working on it), we
will not have in<br class="gmail_msg">
7.4, and unlikely in 7.5.<br class="gmail_msg">
<br class="gmail_msg">
As it stands today, I believe there are two open
questions:<br class="gmail_msg">
<br class="gmail_msg">
1. do we need a schema for the metadata section?<br
class="gmail_msg">
2. how do we bind the metadata to the devices? How do we
know which<br class="gmail_msg">
metadata belongs to which metadata, if we don't have
aliases nor<br class="gmail_msg">
addresses to match? (e.g. very first time the VM is
created!)<br class="gmail_msg">
<br class="gmail_msg">
My current stance is the following<br class="gmail_msg">
1. In general, one schema gives us two benefits: 1.a. we
document how<br class="gmail_msg">
the layout of the data should be, including types; 1.b. we
can validate<br class="gmail_msg">
the data we receive.<br class="gmail_msg">
So yes, we need a schema, but we don't need a *new*
schema. I think we<br class="gmail_msg">
are in good enough shape with the current Vdsm schema: we
can just<br class="gmail_msg">
translate the python object layout to a XML layout.<br
class="gmail_msg">
<br class="gmail_msg">
One example is probably more explicative. Some actual data
may look<br class="gmail_msg">
like, using my pyxmlpickle module:<br class="gmail_msg">
<br class="gmail_msg">
<domain type='kvm' id='5'><br class="gmail_msg">
<name>a0</name><br class="gmail_msg">
<uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid><br
class="gmail_msg">
<metadata xmlns:ovirt-tune="<a moz-do-not-send="true"
href="http://ovirt.org/vm/tune/1.0" rel="noreferrer"
class="gmail_msg" target="_blank">http://ovirt.org/vm/tune/1.0</a>"<br
class="gmail_msg">
</blockquote>
<div><br>
</div>
<div>The url is broken - do we have this?</div>
</div>
</div>
</div>
</blockquote>
<br>
I guess no. This is old code, dating back to 3.6,<i> which I'm
taking without changes.</i><br>
<br>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div> </div>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
xmlns:ovirt-vm="<a moz-do-not-send="true"
href="http://ovirt.org/vm/1.0" rel="noreferrer"
class="gmail_msg" target="_blank">http://ovirt.org/vm/1.0</a>"<br
class="gmail_msg">
xmlns:ovirt-instance="<a moz-do-not-send="true"
href="http://ovirt.org/vm/instance/1.0" rel="noreferrer"
class="gmail_msg" target="_blank">http://ovirt.org/vm/instance/1.0</a>"><br
class="gmail_msg">
<ovirt-tune:qos/> </blockquote>
<div><br>
</div>
<div>What do we keep here? why does it need its own
namespace?</div>
<div> </div>
</div>
</div>
</div>
</blockquote>
<br>
yes, because of how we use the metadata:<br>
<br>
<a class="moz-txt-link-freetext" href="http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainGetMetadata">http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainGetMetadata</a><br>
<a class="moz-txt-link-freetext" href="http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainSetMetadata">http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainSetMetadata</a><br>
<br>
we use type =
<a class="moz-txt-link-freetext" href="http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_METADATA_ELEMENT">http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_METADATA_ELEMENT</a><br>
<br>
-- I don't think we should change that<br>
<br>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-vm:vm/><br class="gmail_msg">
</blockquote>
<div><br>
</div>
<div>
<div>What do we keep here? why does it need its own
namespace?</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
same as above<br>
<br>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div><br class="inbox-inbox-Apple-interchange-newline">
</div>
<div>Can we merge all namespaces into one generic namespace?</div>
</div>
</div>
</div>
</blockquote>
<br>
Probably, but we have the nice property that code can trivially
access one subset of metadata, e.g.<br>
only the qos subtree, only the container subtree.<br>
Furthermore if we don't change that we are backward compatible at
XML level (inbound migrations<br>
care about that).<br>
<br>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div> </div>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-instance:instance><br class="gmail_msg">
</blockquote>
<div class="gmail_msg"><br class="gmail_msg">
</div>
</div>
</div>
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div class="gmail_msg">What is instance?</div>
</div>
</div>
</div>
</blockquote>
<br>
Gone, it duplicated vm without a good reason<br>
<br>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div class="gmail_msg"> </div>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-instance:value type="dict"> </blockquote>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-instance:item key="devices"
type="list"> </blockquote>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-instance:item index="0"
type="dict"></blockquote>
<div><br>
</div>
<div>Isn't index redundant?</div>
</div>
</div>
</div>
</blockquote>
<br>
It is, but now is gone<br>
<br>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div> </div>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-instance:item key="device"<br
class="gmail_msg">
type="str">vnc</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="specParams"
type="dict"><br class="gmail_msg">
<ovirt-instance:item key="displayNetwork"<br
class="gmail_msg">
type="str">ovirtmgmt</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="displayIp"<br
class="gmail_msg">
type="str">192.168.1.53</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="type"<br
class="gmail_msg">
type="str">graphics</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item index="1"
type="dict"><br class="gmail_msg">
<ovirt-instance:item key="device"<br
class="gmail_msg">
type="str">spice</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="specParams"
type="dict"><br class="gmail_msg">
<ovirt-instance:item key="displayNetwork"<br
class="gmail_msg">
type="str">ovirtmgmt</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="displayIp"<br
class="gmail_msg">
type="str">192.168.1.53</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="type"<br
class="gmail_msg">
type="str">graphics</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item index="2"
type="dict"><br class="gmail_msg">
<ovirt-instance:item key="poolID"<br
class="gmail_msg">
type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="imageID"<br
class="gmail_msg">
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="domainID"<br
class="gmail_msg">
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="device"<br
class="gmail_msg">
type="str">disk</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="path"<br
class="gmail_msg">
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="volumeID"<br
class="gmail_msg">
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:item><br class="gmail_msg">
</ovirt-instance:value><br class="gmail_msg">
</ovirt-instance:instance><br class="gmail_msg">
</metadata><br class="gmail_msg">
<!-- omitted for brevity --><br class="gmail_msg">
</domain><br class="gmail_msg">
</blockquote>
<div class="gmail_msg"><br class="gmail_msg">
</div>
</div>
</div>
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div class="gmail_msg">How about a generic json namespace?<br
class="gmail_msg">
</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">json:</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">{</div>
<div class="gmail_msg"> "devices": [</div>
<div class="gmail_msg"> {</div>
<div class="gmail_msg"> "foo": "bar"</div>
<div class="gmail_msg"> }</div>
<div class="gmail_msg"> ]</div>
<div class="gmail_msg">}</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">xml:</div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg"><json:object></div>
<div class="gmail_msg">
<json:key>devices</json:key></div>
<div class="gmail_msg"> <json:list></div>
<div class="gmail_msg"> <json:object></div>
<div class="gmail_msg">
<json:key>foo</json:key></div>
<div class="gmail_msg">
<div class="gmail_msg">
<json:string>bar</json:string></div>
</div>
<div class="gmail_msg"> </json:object></div>
<div class="gmail_msg"> </json:list><br
class="gmail_msg">
</div>
<div class="gmail_msg"></json:object></div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg">We can query and modify this xml
using xpath, like this:</div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">>>>
r.findall("./metadata/json:object/[json:key='devices']/json:list/json:object/",
namespaces={"json": "<a moz-do-not-send="true"
href="http://ovirt.org/json/1.0">http://ovirt.org/json/1.0</a>"})</div>
<div class="gmail_msg">[<Element '{<a
moz-do-not-send="true"
href="http://ovirt.org/json/1.0%7Dkey">http://ovirt.org/json/1.0}key</a>'
at 0x7fd3386a69d0>, <Element '{<a
moz-do-not-send="true"
href="http://ovirt.org/json/1.0%7Dstring">http://ovirt.org/json/1.0}string</a>'
at 0x7fd3386a6a50>]</div>
<div><br>
</div>
</div>
</div>
<div>Not sure this format will be easy to modify, needs
to tinker more</div>
<div>with this.</div>
<div><br>
</div>
</div>
</div>
</div>
<div class="gmail_quote gmail_msg">
<div class="gmail_msg">It will be probably easier to parse
the entire metadata, change it,</div>
<div class="gmail_msg">and serialize it back.</div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg">Maybe using key and type attribute as
you suggest makes it simpler to use,</div>
<div class="gmail_msg">and we since we convert from xml to
python or python to xml, we can</div>
<div class="gmail_msg">have a "py" namespace.</div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg">
<div class="gmail_msg"><py:item></div>
<div class="gmail_msg"> <py:item key="devices"
type="list"></div>
<div class="gmail_msg"> <py:item type="dict"></div>
<div class="gmail_msg"> <py:item key="foo"
type="str">bar</py:item></div>
<div class="gmail_msg"> </py:item><br>
</py:item></div>
<div class="gmail_msg"></py:item></div>
</div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg">With this we can have:</div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg">
<div class="gmail_msg">>>>
r.findall("./metadata/py:item/py:item[@key='devices']/py:item[1]/py:item[@key='foo']",
namespaces={"py": "<a moz-do-not-send="true"
href="http://ovirt.org/py/1.0">http://ovirt.org/py/1.0</a>"})[0].text</div>
<div class="gmail_msg">'bar'</div>
<div><br>
</div>
</div>
<div class="gmail_msg">Something like this can be useful for
others as well.</div>
</div>
</div>
</div>
</blockquote>
<br>
Great suggestions, I like the py namespace, but concerns were raised
about the solution here being too generic, so I limited it, see<br>
<a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/74206/11">https://gerrit.ovirt.org/#/c/74206/11</a><br>
<br>
but I will add my pyxmlpickle module (<a moz-do-not-send="true"
href="https://github.com/fromanirh/pyxmlpickle" rel="noreferrer"
class="gmail_msg" target="_blank">https://github.com/fromanirh/pyxmlpickle</a>)
anyway in the future according to your comments.<br>
<br>
<br>
<pre class="moz-signature" cols="72">--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani</pre>
</body>
</html>