Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

Wednesday, 22 March 2017

This is a multi-part message in MIME format.
--------------15F6CAD0BC7802BD4F39C8F8
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Please note that the approach described here is outdated, because
xmlpickle was too easy to use

and too generic: that lead to bloated XML and too high risk of misusing
metadata;

the new approach (https://gerrit.ovirt.org/#/c/74206/) tries to balance
things making usage

convenient but disallowing arbitrarily nesting

On 03/18/2017 03:27 PM, Nir Soffer wrote:
...
 On Wed, Mar 15, 2017 at 2:28 PM Francesco Romani
<fromani(a)redhat.com
 <mailto:fromani@redhat.com>> wrote:

     Hi everyone,

     This is both a report of the current state of my Vdsm patches for
     Engine
     XML support, and a proposal to how move forward and solve
     the current open issues.

     TL;DR:
     1. we can and IMO should reuse the current JSON schema to describe the
     structure (layout) and the types of the metadata section.
     2. we don't need a priori validation of stuff in the metadata section.
     We will just raise in the creation flow if data is missing, or wrong,
     according to our schema.
     2. we will add *few* items to the metadata section, only thing we
     can't
     express clearly-or at all in the libvirt XML. Redundancy and
     verbosiness
     will be thus kept at bay
     3. I believe [3] is the best tool to do (de)serialize data to the
     metadata section. Existing tools fits poorly in our very specific
     use case

     Examples below

     +++

     Long(er) discussion:

     I have working code[1][2] to encode any custom, picklable, python
     object in the metadata section.

     We should decide which module will do the actual python<=>XML
     transformation.
     Please note that this actually also influences how the data in the
     medata section look like, so the two things are a bit coupled.

     I'm eager to reinvent another wheel, but after
     initial evaluation I honestly think that my pyxmlpickle[3] is the best
     tool for the job over the current alternatives: plistlib[4] and
     xmltodict[5].

     I added the initial rationale here:
     https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG

     I have completed the initial draft of patches to make it possible to
     initialize devices from their XML representation [6]. This is the bare
     minimum we need to support the Engine XML, and we *need* this
     anyway to
     unlock the cleanup we planned and I outlined in my google doc.

     So we are progressing, but I'd like to speed up things. Those [6]
     patches are not yet complete, many flows are not covered or
     tested;  but
     they are good enough to demonstrate that there *are* pieces of
     information wen need to properly initialize the devices, but we can't
     easily extract from the XML.

     First examples that come to my mind are the storage.Drive UUIDs; there
     could also be some ambiguity I'm investigating right now for
     displayIp/displayNetwork in Graphics devices. In [6] there are various
     TODO to mark more of those cases. Most likely, few more cases will pop
     out as I cover all the flows we support.

     Long story short: it is hard to correctly rebuild the device conf from
     the XML. This is why in [6] I added the 'meta' argument to
     from_xml_tree
     classmethod in [7].

     'meta' is supposed to be the device metadata: extra data related to a
     device which doesn't (yet) fit in the libvirt XML representation.
     For example, we can store 'displayIp' and 'displayNetwork' here and
be
     done with that: using both per-device metadata and the XML
     representation of one graphic device, we will have everything we
     need to
     properly build one graphics.Graphics device.
     This example may (hopefully) be bogus, but I'm keeping it because
     it is
     one case easy to follow.

     The device metadata is going to be stored in the vm metadata for the
     short/mid term future. Even if the per-device metadata idea/RFE is
     accepted (no answer yet, but we are working on it), we will not
     have in
     7.4, and unlikely in 7.5.

     As it stands today, I believe there are two open questions:

     1. do we need a schema for the metadata section?
     2. how do we bind the metadata to the devices? How do we know which
     metadata belongs to which metadata, if we don't have aliases nor
     addresses to match? (e.g. very first time the VM is created!)

     My current stance is the following
     1. In general, one schema gives us two benefits: 1.a. we document how
     the layout of the data should be, including types; 1.b. we can
     validate
     the data we receive.
     So yes, we need a schema, but we don't need a *new* schema. I think we
     are in good enough shape with the current Vdsm schema: we can just
     translate the python object layout to a XML layout.

     One example is probably more explicative. Some actual data may look
     like, using my pyxmlpickle module:

     <domain type='kvm' id='5'>
       <name>a0</name>
       <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid>
       <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0"

 The url is broken - do we have this? 
I guess no. This is old code, dating back to 3.6,/which I'm taking
without changes./

...

     xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
     xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0">
         <ovirt-tune:qos/> 

 What do we keep here? why does it need its own namespace?

yes, because of how we use the metadata:

http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainGetMetadata
http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainSetMetadata

we use type =
http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_METADATA_E...

-- I don't think we should change that

...
         <ovirt-vm:vm/>

 What do we keep here? why does it need its own namespace? 
same as above

...

 Can we merge all namespaces into one generic namespace? 
Probably, but we have the nice property that code can trivially access
one subset of metadata, e.g.
only the qos subtree, only the container subtree.
Furthermore if we don't change that we are backward compatible at XML
level (inbound migrations
care about that).

...

         <ovirt-instance:instance>

 What is instance? 
Gone, it duplicated vm without a good reason

...

           <ovirt-instance:value type="dict">  

             <ovirt-instance:item key="devices" type="list"> 

               <ovirt-instance:item index="0" type="dict">

 Isn't index redundant? 
It is, but now is gone

...

                 <ovirt-instance:item key="device"
     type="str">vnc</ovirt-instance:item>
                 <ovirt-instance:item key="specParams"
type="dict">
                   <ovirt-instance:item key="displayNetwork"
     type="str">ovirtmgmt</ovirt-instance:item>
                   <ovirt-instance:item key="displayIp"
     type="str">192.168.1.53</ovirt-instance:item>
                 </ovirt-instance:item>
                 <ovirt-instance:item key="type"
     type="str">graphics</ovirt-instance:item>
               </ovirt-instance:item>
               <ovirt-instance:item index="1" type="dict">
                 <ovirt-instance:item key="device"
     type="str">spice</ovirt-instance:item>
                 <ovirt-instance:item key="specParams"
type="dict">
                   <ovirt-instance:item key="displayNetwork"
     type="str">ovirtmgmt</ovirt-instance:item>
                   <ovirt-instance:item key="displayIp"
     type="str">192.168.1.53</ovirt-instance:item>
                 </ovirt-instance:item>
                 <ovirt-instance:item key="type"
     type="str">graphics</ovirt-instance:item>
               </ovirt-instance:item>
               <ovirt-instance:item index="2" type="dict">
                 <ovirt-instance:item key="poolID"

type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item>
                 <ovirt-instance:item key="imageID"

type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
                 <ovirt-instance:item key="domainID"

type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
                 <ovirt-instance:item key="device"
     type="str">disk</ovirt-instance:item>
                 <ovirt-instance:item key="path"

type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
                 <ovirt-instance:item key="volumeID"

type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
               </ovirt-instance:item>
             </ovirt-instance:item>
           </ovirt-instance:value>
         </ovirt-instance:instance>
       </metadata>
      <!-- omitted for brevity -->
     </domain>

 How about a generic json namespace?

 json:

 {
     "devices": [
        {
            "foo": "bar"
        }
    ]
 }

 xml:

 <json:object>
     <json:key>devices</json:key>
     <json:list>
        <json:object>
            <json:key>foo</json:key>
            <json:string>bar</json:string>
        </json:object>
     </json:list>
 </json:object>

 We can query and modify this xml using xpath, like this:

 >>>

r.findall("./metadata/json:object/[json:key='devices']/json:list/json:object/",
 namespaces={"json": "http://ovirt.org/json/1.0"})
 [<Element '{http://ovirt.org/json/1.0}key
 <http://ovirt.org/json/1.0%7Dkey>' at 0x7fd3386a69d0>, <Element
 '{http://ovirt.org/json/1.0}string
 <http://ovirt.org/json/1.0%7Dstring>' at 0x7fd3386a6a50>]

 Not sure this format will be easy to modify, needs to tinker more
 with this.

 It will be probably easier to parse the entire metadata, change it,
 and serialize it back.

 Maybe using key and type attribute as you suggest makes it simpler to use,
 and we since we convert from xml to python or python to xml, we can
 have a "py" namespace.

 <py:item>
     <py:item key="devices" type="list">
         <py:item type="dict">
             <py:item key="foo" type="str">bar</py:item>
         </py:item>
     </py:item>
 </py:item>

 With this we can have:

 >>>

r.findall("./metadata/py:item/py:item[@key='devices']/py:item[1]/py:item[@key='foo']",
 namespaces={"py": "http://ovirt.org/py/1.0"})[0].text
 'bar'

 Something like this can be useful for others as well. 
Great suggestions, I like the py namespace, but concerns were raised
about the solution here being too generic, so I limited it, see
https://gerrit.ovirt.org/#/c/74206/11

but I will add my pyxmlpickle module
(https://github.com/fromanirh/pyxmlpickle) anyway in the future
according to your comments.

-- 
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani

--------------15F6CAD0BC7802BD4F39C8F8
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p>Please note that the approach described here is outdated, because
      xmlpickle was too easy to use</p>
    <p>and too generic: that lead to bloated XML and too high risk of
      misusing metadata;</p>
    <p>the new approach (<a class="moz-txt-link-freetext"
href="https://gerrit.ovirt.org/#/c/74206/">https://gerrit.ov...>)
tries to
      balance things making usage</p>
    <p>convenient but disallowing arbitrarily nesting<br>
    </p>
    <br>
    <div class="moz-cite-prefix">On 03/18/2017 03:27 PM, Nir Soffer
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div dir="ltr" class="gmail_msg">
          <div class="gmail_quote gmail_msg">
            <div dir="ltr" class="gmail_msg">On Wed, Mar 15,
2017 at
              2:28 PM Francesco Romani &lt;<a moz-do-not-send="true"
                href="mailto:fromani@redhat.com" class="gmail_msg"
                target=&quot;_blank&quot;&gt;fromani(a)redhat.com&lt;/a&gt;&amp;gt;
wrote:<br
                class="gmail_msg">
            </div>
            <blockquote class="gmail_quote gmail_msg" style="margin:0 0
              0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi
              everyone,<br class="gmail_msg">
              <br class="gmail_msg">
              This is both a report of the current state of my Vdsm
              patches for Engine<br class="gmail_msg">
              XML support, and a proposal to how move forward and solve<br
                class="gmail_msg">
              the current open issues.<br class="gmail_msg">
              <br class="gmail_msg">
              TL;DR:<br class="gmail_msg">
              1. we can and IMO should reuse the current JSON schema to
              describe the<br class="gmail_msg">
              structure (layout) and the types of the metadata section.<br
                class="gmail_msg">
              2. we don't need a priori validation of stuff in the
              metadata section.<br class="gmail_msg">
              We will just raise in the creation flow if data is
              missing, or wrong,<br class="gmail_msg">
              according to our schema.<br class="gmail_msg">
              2. we will add *few* items to the metadata section, only
              thing we can't<br class="gmail_msg">
              express clearly-or at all in the libvirt XML. Redundancy
              and verbosiness<br class="gmail_msg">
              will be thus kept at bay<br class="gmail_msg">
              3. I believe [3] is the best tool to do (de)serialize data
              to the<br class="gmail_msg">
              metadata section. Existing tools fits poorly in our very
              specific use case<br class="gmail_msg">
              <br class="gmail_msg">
              Examples below<br class="gmail_msg">
              <br class="gmail_msg">
              +++<br class="gmail_msg">
              <br class="gmail_msg">
              Long(er) discussion:<br class="gmail_msg">
              <br class="gmail_msg">
              <br class="gmail_msg">
              I have working code[1][2] to encode any custom, picklable,
              python<br class="gmail_msg">
              object in the metadata section.<br class="gmail_msg">
              <br class="gmail_msg">
              We should decide which module will do the actual
              python&lt;=&gt;XML<br class="gmail_msg">
              transformation.<br class="gmail_msg">
              Please note that this actually also influences how the
              data in the<br class="gmail_msg">
              medata section look like, so the two things are a bit
              coupled.<br class="gmail_msg">
              <br class="gmail_msg">
              I'm eager to reinvent another wheel, but after<br
                class="gmail_msg">
              initial evaluation I honestly think that my pyxmlpickle[3]
              is the best<br class="gmail_msg">
              tool for the job over the current alternatives:
              plistlib[4] and<br class="gmail_msg">
              xmltodict[5].<br class="gmail_msg">
              <br class="gmail_msg">
              I added the initial rationale here:<br class="gmail_msg">
              <a moz-do-not-send="true"
                href="https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG"
                rel="noreferrer" class="gmail_msg"
target="_blank">https://gerrit.ovirt.org/#/c/73790/4//COMMIT...
                class="gmail_msg">
              <br class="gmail_msg">
              I have completed the initial draft of patches to make it
              possible to<br class="gmail_msg">
              initialize devices from their XML representation [6]. This
              is the bare<br class="gmail_msg">
              minimum we need to support the Engine XML, and we *need*
              this anyway to<br class="gmail_msg">
              unlock the cleanup we planned and I outlined in my google
              doc.<br class="gmail_msg">
              <br class="gmail_msg">
              So we are progressing, but I'd like to speed up things.
              Those [6]<br class="gmail_msg">
              patches are not yet complete, many flows are not covered
              or tested;  but<br class="gmail_msg">
              they are good enough to demonstrate that there *are*
              pieces of<br class="gmail_msg">
              information wen need to properly initialize the devices,
              but we can't<br class="gmail_msg">
              easily extract from the XML.<br class="gmail_msg">
              <br class="gmail_msg">
              First examples that come to my mind are the storage.Drive
              UUIDs; there<br class="gmail_msg">
              could also be some ambiguity I'm investigating right now
              for<br class="gmail_msg">
              displayIp/displayNetwork in Graphics devices. In [6] there
              are various<br class="gmail_msg">
              TODO to mark more of those cases. Most likely, few more
              cases will pop<br class="gmail_msg">
              out as I cover all the flows we support.<br
                class="gmail_msg">
              <br class="gmail_msg">
              Long story short: it is hard to correctly rebuild the
              device conf from<br class="gmail_msg">
              the XML. This is why in [6] I added the 'meta' argument to
              from_xml_tree<br class="gmail_msg">
              classmethod in [7].<br class="gmail_msg">
              <br class="gmail_msg">
              'meta' is supposed to be the device metadata: extra data
              related to a<br class="gmail_msg">
              device which doesn't (yet) fit in the libvirt XML
              representation.<br class="gmail_msg">
              For example, we can store 'displayIp' and 'displayNetwork'
              here and be<br class="gmail_msg">
              done with that: using both per-device metadata and the XML<br
                class="gmail_msg">
              representation of one graphic device, we will have
              everything we need to<br class="gmail_msg">
              properly build one graphics.Graphics device.<br
                class="gmail_msg">
              This example may (hopefully) be bogus, but I'm keeping it
              because it is<br class="gmail_msg">
              one case easy to follow.<br class="gmail_msg">
              <br class="gmail_msg">
              The device metadata is going to be stored in the vm
              metadata for the<br class="gmail_msg">
              short/mid term future. Even if the per-device metadata
              idea/RFE is<br class="gmail_msg">
              accepted (no answer yet, but we are working on it), we
              will not have in<br class="gmail_msg">
              7.4, and unlikely in 7.5.<br class="gmail_msg">
              <br class="gmail_msg">
              As it stands today, I believe there are two open
              questions:<br class="gmail_msg">
              <br class="gmail_msg">
              1. do we need a schema for the metadata section?<br
                class="gmail_msg">
              2. how do we bind the metadata to the devices? How do we
              know which<br class="gmail_msg">
              metadata belongs to which metadata, if we don't have
              aliases nor<br class="gmail_msg">
              addresses to match? (e.g. very first time the VM is
              created!)<br class="gmail_msg">
              <br class="gmail_msg">
              My current stance is the following<br class="gmail_msg">
              1. In general, one schema gives us two benefits: 1.a. we
              document how<br class="gmail_msg">
              the layout of the data should be, including types; 1.b. we
              can validate<br class="gmail_msg">
              the data we receive.<br class="gmail_msg">
              So yes, we need a schema, but we don't need a *new*
              schema. I think we<br class="gmail_msg">
              are in good enough shape with the current Vdsm schema: we
              can just<br class="gmail_msg">
              translate the python object layout to a XML layout.<br
                class="gmail_msg">
              <br class="gmail_msg">
              One example is probably more explicative. Some actual data
              may look<br class="gmail_msg">
              like, using my pyxmlpickle module:<br class="gmail_msg">
              <br class="gmail_msg">
              &lt;domain type='kvm' id='5'&gt;<br
class="gmail_msg">
                &lt;name&gt;a0&lt;/name&gt;<br
class="gmail_msg">

&lt;uuid&gt;ccd945c8-8069-4f31-8471-bbb58e9dd6ea&lt;/uuid&gt;<br
                class="gmail_msg">
                &lt;metadata xmlns:ovirt-tune="<a
moz-do-not-send="true"
                href="http://ovirt.org/vm/tune/1.0" rel="noreferrer"
                class="gmail_msg"
target="_blank">http://ovirt.org/vm/tune/1.0</a>"...
                class="gmail_msg">
            </blockquote>
            <div><br>
            </div>
            <div>The url is broken - do we have this?</div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    I guess no. This is old code, dating back to 3.6,<i> which I'm
      taking without changes.</i><br>
    <br>
    <blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div dir="ltr" class="gmail_msg">
          <div class="gmail_quote gmail_msg">
            <div> </div>
            <blockquote class="gmail_quote gmail_msg" style="margin:0 0
              0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              xmlns:ovirt-vm="<a moz-do-not-send="true"
                href="http://ovirt.org/vm/1.0" rel="noreferrer"
                class="gmail_msg"
target="_blank">http://ovirt.org/vm/1.0</a>"<br
                class="gmail_msg">
              xmlns:ovirt-instance="<a moz-do-not-send="true"
                href="http://ovirt.org/vm/instance/1.0"
rel="noreferrer"
                class="gmail_msg"
target="_blank">http://ovirt.org/vm/instance/1.0</a>&q...
                class="gmail_msg">
                  &lt;ovirt-tune:qos/&gt; </blockquote>
            <div><br>
            </div>
            <div>What do we keep here? why does it need its own
              namespace?</div>
            <div> </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    yes, because of how we use the metadata:<br>
    <br>
<a class="moz-txt-link-freetext"
href="http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainG...
<a class="moz-txt-link-freetext"
href="http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainS...
    <br>
    we use type =
<a class="moz-txt-link-freetext"
href="http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN...
    <br>
    -- I don't think we should change that<br>
    <br>
    <blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div dir="ltr" class="gmail_msg">
          <div class="gmail_quote gmail_msg">
            <blockquote class="gmail_quote gmail_msg" style="margin:0 0
              0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                  &lt;ovirt-vm:vm/&gt;<br class="gmail_msg">
            </blockquote>
            <div><br>
            </div>
            <div>
              <div>What do we keep here? why does it need its own
                namespace?</div>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    same as above<br>
    <br>
    <blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div dir="ltr" class="gmail_msg">
          <div class="gmail_quote gmail_msg">
            <div><br class="inbox-inbox-Apple-interchange-newline">
            </div>
            <div>Can we merge all namespaces into one generic
namespace?</div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    Probably, but we have the nice property that code can trivially
    access one subset of metadata, e.g.<br>
    only the qos subtree, only the container subtree.<br>
    Furthermore if we don't change that we are backward compatible at
    XML level (inbound migrations<br>
    care about that).<br>
    <br>
    <blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div dir="ltr" class="gmail_msg">
          <div class="gmail_quote gmail_msg">
            <div> </div>
            <blockquote class="gmail_quote gmail_msg" style="margin:0 0
              0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                  &lt;ovirt-instance:instance&gt;<br
class="gmail_msg">
            </blockquote>
            <div class="gmail_msg"><br class="gmail_msg">
            </div>
          </div>
        </div>
        <div dir="ltr" class="gmail_msg">
          <div class="gmail_quote gmail_msg">
            <div class="gmail_msg">What is instance?</div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    Gone, it duplicated vm without a good reason<br>
    <br>
    <blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div dir="ltr" class="gmail_msg">
          <div class="gmail_quote gmail_msg">
            <div class="gmail_msg"> </div>
            <blockquote class="gmail_quote gmail_msg" style="margin:0 0
              0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                    &lt;ovirt-instance:value type="dict"&gt;
 </blockquote>
            <blockquote class="gmail_quote gmail_msg" style="margin:0 0
              0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      &lt;ovirt-instance:item key="devices"
              type="list"&gt; </blockquote>
            <blockquote class="gmail_quote gmail_msg" style="margin:0 0
              0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                        &lt;ovirt-instance:item index="0"
              type="dict"&gt;</blockquote>
            <div><br>
            </div>
            <div>Isn't index redundant?</div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    It is, but now is gone<br>
    <br>
    <blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div dir="ltr" class="gmail_msg">
          <div class="gmail_quote gmail_msg">
            <div> </div>
            <blockquote class="gmail_quote gmail_msg" style="margin:0 0
              0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                          &lt;ovirt-instance:item key="device"<br
                class="gmail_msg">
              type="str"&gt;vnc&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                          &lt;ovirt-instance:item key="specParams"
              type="dict"&gt;<br class="gmail_msg">
                            &lt;ovirt-instance:item
key="displayNetwork"<br
                class="gmail_msg">

type="str"&gt;ovirtmgmt&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                            &lt;ovirt-instance:item key="displayIp"<br
                class="gmail_msg">

type="str"&gt;192.168.1.53&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                          &lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                          &lt;ovirt-instance:item key="type"<br
                class="gmail_msg">

type="str"&gt;graphics&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                        &lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                        &lt;ovirt-instance:item index="1"
              type="dict"&gt;<br class="gmail_msg">
                          &lt;ovirt-instance:item key="device"<br
                class="gmail_msg">
              type="str"&gt;spice&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                          &lt;ovirt-instance:item key="specParams"
              type="dict"&gt;<br class="gmail_msg">
                            &lt;ovirt-instance:item
key="displayNetwork"<br
                class="gmail_msg">

type="str"&gt;ovirtmgmt&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                            &lt;ovirt-instance:item key="displayIp"<br
                class="gmail_msg">

type="str"&gt;192.168.1.53&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                          &lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                          &lt;ovirt-instance:item key="type"<br
                class="gmail_msg">

type="str"&gt;graphics&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                        &lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                        &lt;ovirt-instance:item index="2"
              type="dict"&gt;<br class="gmail_msg">
                          &lt;ovirt-instance:item key="poolID"<br
                class="gmail_msg">
type="str"&gt;5890a292-0390-01d2-01ed-00000000029a&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                          &lt;ovirt-instance:item key="imageID"<br
                class="gmail_msg">
type="str"&gt;66441539-f7ac-4946-8a25-75e422f939d4&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                          &lt;ovirt-instance:item key="domainID"<br
                class="gmail_msg">
type="str"&gt;c578566d-bc61-420c-8f1e-8dfa0a18efd5&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                          &lt;ovirt-instance:item key="device"<br
                class="gmail_msg">
              type="str"&gt;disk&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                          &lt;ovirt-instance:item key="path"<br
                class="gmail_msg">
type="str"&gt;/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                          &lt;ovirt-instance:item key="volumeID"<br
                class="gmail_msg">
type="str"&gt;5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc&lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                        &lt;/ovirt-instance:item&gt;<br
                class="gmail_msg">
                      &lt;/ovirt-instance:item&gt;<br
class="gmail_msg">
                    &lt;/ovirt-instance:value&gt;<br
class="gmail_msg">
                  &lt;/ovirt-instance:instance&gt;<br
class="gmail_msg">
                &lt;/metadata&gt;<br class="gmail_msg">
               &lt;!-- omitted for brevity --&gt;<br
class="gmail_msg">
              &lt;/domain&gt;<br class="gmail_msg">
            </blockquote>
            <div class="gmail_msg"><br class="gmail_msg">
            </div>
          </div>
        </div>
        <div dir="ltr" class="gmail_msg">
          <div class="gmail_quote gmail_msg">
            <div class="gmail_msg">How about a generic json
namespace?<br
                class="gmail_msg">
            </div>
            <div class="gmail_msg"><br class="gmail_msg">
            </div>
            <div class="gmail_msg">json:</div>
            <div class="gmail_msg"><br class="gmail_msg">
            </div>
            <div class="gmail_msg">{</div>
            <div class="gmail_msg">    "devices": [</div>
            <div class="gmail_msg">       {</div>
            <div class="gmail_msg">           "foo":
"bar"</div>
            <div class="gmail_msg">       }</div>
            <div class="gmail_msg">   ]</div>
            <div class="gmail_msg">}</div>
            <div class="gmail_msg"><br class="gmail_msg">
            </div>
            <div class="gmail_msg">xml:</div>
            <div class="gmail_msg"><br>
            </div>
            <div
class="gmail_msg">&lt;json:object&gt;</div>
            <div class="gmail_msg">   
              &lt;json:key&gt;devices&lt;/json:key&gt;</div>
            <div class="gmail_msg">   
&lt;json:list&gt;</div>
            <div class="gmail_msg">     
 &lt;json:object&gt;</div>
            <div class="gmail_msg">         
               &lt;json:key&gt;foo&lt;/json:key&gt;</div>
            <div class="gmail_msg">
              <div class="gmail_msg">         
                 &lt;json:string&gt;bar&lt;/json:string&gt;</div>
            </div>
            <div class="gmail_msg">     
 &lt;/json:object&gt;</div>
            <div class="gmail_msg">    &lt;/json:list&gt;<br
                class="gmail_msg">
            </div>
            <div
class="gmail_msg">&lt;/json:object&gt;</div>
            <div class="gmail_msg"><br>
            </div>
            <div class="gmail_msg">We can query and modify this xml
              using xpath, like this:</div>
            <div class="gmail_msg"><br>
            </div>
            <div class="gmail_msg">
              <div class="gmail_msg">
                <div class="gmail_msg">
                  <div class="gmail_msg">
                    <div class="gmail_msg">&gt;&gt;&gt;
r.findall("./metadata/json:object/[json:key='devices']/json:list/json:object/",
                      namespaces={"json": "<a
moz-do-not-send="true"

href="http://ovirt.org/json/1.0">http://ovirt.org/json/1.0&l...
                    <div class="gmail_msg">[&lt;Element '{<a
                        moz-do-not-send="true"

href="http://ovirt.org/json/1.0%7Dkey">http://ovirt.org/json...
                      at 0x7fd3386a69d0&gt;, &lt;Element '{<a
                        moz-do-not-send="true"

href="http://ovirt.org/json/1.0%7Dstring">http://ovirt.org/j...
                      at 0x7fd3386a6a50&gt;]</div>
                    <div><br>
                    </div>
                  </div>
                </div>
                <div>Not sure this format will be easy to modify, needs
                  to tinker more</div>
                <div>with this.</div>
                <div><br>
                </div>
              </div>
            </div>
          </div>
          <div class="gmail_quote gmail_msg">
            <div class="gmail_msg">It will be probably easier to parse
              the entire metadata, change it,</div>
            <div class="gmail_msg">and serialize it back.</div>
            <div class="gmail_msg"><br>
            </div>
            <div class="gmail_msg">Maybe using key and type attribute as
              you suggest makes it simpler to use,</div>
            <div class="gmail_msg">and we since we convert from xml to
              python or python to xml, we can</div>
            <div class="gmail_msg">have a "py"
namespace.</div>
            <div class="gmail_msg"><br>
            </div>
            <div class="gmail_msg">
              <div class="gmail_msg">&lt;py:item&gt;</div>
              <div class="gmail_msg">    &lt;py:item
key="devices"
                type="list"&gt;</div>
              <div class="gmail_msg">        &lt;py:item
type="dict"&gt;</div>
              <div class="gmail_msg">            &lt;py:item
key="foo"
                type="str"&gt;bar&lt;/py:item&gt;</div>
              <div class="gmail_msg">       
&lt;/py:item&gt;<br>
                    &lt;/py:item&gt;</div>
              <div class="gmail_msg">&lt;/py:item&gt;</div>
            </div>
            <div class="gmail_msg"><br>
            </div>
            <div class="gmail_msg">With this we can have:</div>
            <div class="gmail_msg"><br>
            </div>
            <div class="gmail_msg">
              <div class="gmail_msg">&gt;&gt;&gt;
r.findall("./metadata/py:item/py:item[@key='devices']/py:item[1]/py:item[@key='foo']",
                namespaces={"py": "<a moz-do-not-send="true"

href="http://ovirt.org/py/1.0">http://ovirt.org/py/1.0</a...
              <div class="gmail_msg">'bar'</div>
              <div><br>
              </div>
            </div>
            <div class="gmail_msg">Something like this can be useful for
              others as well.</div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    Great suggestions, I like the py namespace, but concerns were raised
    about the solution here being too generic, so I limited it, see<br>
    <a class="moz-txt-link-freetext"
href="https://gerrit.ovirt.org/#/c/74206/11">https://gerrit....
    <br>
    but I will add my pyxmlpickle module (<a moz-do-not-send="true"
      href="https://github.com/fromanirh/pyxmlpickle"
rel="noreferrer"
      class="gmail_msg"
target="_blank">https://github.com/fromanirh/pyxmlpickle</a>)
    anyway in the future according to your comments.<br>
    <br>
    <br>
    <pre class="moz-signature" cols="72">-- 
Francesco Romani
Red Hat Engineering Virtualization R &amp; D
IRC: fromani</pre>
  </body>
</html>

--------------15F6CAD0BC7802BD4F39C8F8--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML