1:28 p.m.
Hi everyone,
This is both a report of the current state of my Vdsm patches for Engine
XML support, and a proposal to how move forward and solve
the current open issues.
TL;DR:
1. we can and IMO should reuse the current JSON schema to describe the
structure (layout) and the types of the metadata section.
2. we don't need a priori validation of stuff in the metadata section.
We will just raise in the creation flow if data is missing, or wrong,
according to our schema.
2. we will add *few* items to the metadata section, only thing we can't
express clearly-or at all in the libvirt XML. Redundancy and verbosiness
will be thus kept at bay
3. I believe [3] is the best tool to do (de)serialize data to the
metadata section. Existing tools fits poorly in our very specific use case
Examples below
+++
Long(er) discussion:
I have working code[1][2] to encode any custom, picklable, python
object in the metadata section.
We should decide which module will do the actual python<=>XML
transformation.
Please note that this actually also influences how the data in the
medata section look like, so the two things are a bit coupled.
I'm eager to reinvent another wheel, but after
initial evaluation I honestly think that my pyxmlpickle[3] is the best
tool for the job over the current alternatives: plistlib[4] and
xmltodict[5].
I added the initial rationale here:
https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
I have completed the initial draft of patches to make it possible to
initialize devices from their XML representation [6]. This is the bare
minimum we need to support the Engine XML, and we *need* this anyway to
unlock the cleanup we planned and I outlined in my google doc.
So we are progressing, but I'd like to speed up things. Those [6]
patches are not yet complete, many flows are not covered or tested; but
they are good enough to demonstrate that there *are* pieces of
information wen need to properly initialize the devices, but we can't
easily extract from the XML.
First examples that come to my mind are the storage.Drive UUIDs; there
could also be some ambiguity I'm investigating right now for
displayIp/displayNetwork in Graphics devices. In [6] there are various
TODO to mark more of those cases. Most likely, few more cases will pop
out as I cover all the flows we support.
Long story short: it is hard to correctly rebuild the device conf from
the XML. This is why in [6] I added the 'meta' argument to from_xml_tree
classmethod in [7].
'meta' is supposed to be the device metadata: extra data related to a
device which doesn't (yet) fit in the libvirt XML representation.
For example, we can store 'displayIp' and 'displayNetwork' here and be
done with that: using both per-device metadata and the XML
representation of one graphic device, we will have everything we need to
properly build one graphics.Graphics device.
This example may (hopefully) be bogus, but I'm keeping it because it is
one case easy to follow.
The device metadata is going to be stored in the vm metadata for the
short/mid term future. Even if the per-device metadata idea/RFE is
accepted (no answer yet, but we are working on it), we will not have in
7.4, and unlikely in 7.5.
As it stands today, I believe there are two open questions:
1. do we need a schema for the metadata section?
2. how do we bind the metadata to the devices? How do we know which
metadata belongs to which metadata, if we don't have aliases nor
addresses to match? (e.g. very first time the VM is created!)
My current stance is the following
1. In general, one schema gives us two benefits: 1.a. we document how
the layout of the data should be, including types; 1.b. we can validate
the data we receive.
So yes, we need a schema, but we don't need a *new* schema. I think we
are in good enough shape with the current Vdsm schema: we can just
translate the python object layout to a XML layout.
One example is probably more explicative. Some actual data may look
like, using my pyxmlpickle module:
<domain type='kvm' id='5'>
<name>a0</name>
<uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid>
<metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0"
xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0">
<ovirt-tune:qos/>
<ovirt-vm:vm/>
<ovirt-instance:instance>
<ovirt-instance:value type="dict">
<ovirt-instance:item key="devices" type="list">
<ovirt-instance:item index="0" type="dict">
<ovirt-instance:item key="device"
type="str">vnc</ovirt-instance:item>
<ovirt-instance:item key="specParams" type="dict">
<ovirt-instance:item key="displayNetwork"
type="str">ovirtmgmt</ovirt-instance:item>
<ovirt-instance:item key="displayIp"
type="str">192.168.1.53</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">graphics</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item index="1" type="dict">
<ovirt-instance:item key="device"
type="str">spice</ovirt-instance:item>
<ovirt-instance:item key="specParams" type="dict">
<ovirt-instance:item key="displayNetwork"
type="str">ovirtmgmt</ovirt-instance:item>
<ovirt-instance:item key="displayIp"
type="str">192.168.1.53</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">graphics</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item index="2" type="dict">
<ovirt-instance:item key="poolID"
type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item>
<ovirt-instance:item key="imageID"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
<ovirt-instance:item key="domainID"
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
<ovirt-instance:item key="device"
type="str">disk</ovirt-instance:item>
<ovirt-instance:item key="path"
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="volumeID"
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:value>
</ovirt-instance:instance>
</metadata>
<!-- omitted for brevity -->
</domain>
Please note that yes, this is still verbose, but we don't want to add
much data here, for most of information the most reliable source will
be the domain XML. We will add here only the extra info we can't really
fetch from that.
2. I don't think we need explicit validation: we could just raise along
the way in the creation flow if we don't find some extra metadata we
need. This will also solve the issue that if we reuse the current schema
and we omit most of data, we will lack quite a lot of elements
marked mandatory.
Once we reached agreement, I will update my
https://docs.google.com/document/d/1eD8KSLwwyo2Sk64MytbmE0wBxxMlpIyEI1GRc...
accordingly.
Final note: while device take the lion's share, we will likely need help
from the metadata section also to store VM extra info, but all the above
discussion also applies here.
+++
[1]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
- uses xmltodict
[2]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
ported the 'virt-metadata3' topic to pyxmlpickle
[3] https://github.com/fromanirh/pyxmlpickle
[4] https://docs.python.org/2/library/plistlib.html
[5] https://github.com/martinblech/xmltodict
[6]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
[7] https://gerrit.ovirt.org/#/c/72880/15/lib/vdsm/virt/vmdevices/core.py
--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
9:45 a.m.
We talked about sending storage device purely on metadata, letting Vdsm
rebuild them and getting the XML like today.
In the other direction, Vdsm will pass through the XML (perhaps only
parts of it, e.g. the devices subtree) like before.
This way we can minimize the changes we are uncertain of, and more
importantly, we can minimize the risky changes.
The following is a realistic example of how the XML could look like if
we send all but the storage devices. It is built using my pyxmlpickle
module (see [3] below).
<domain type='kvm' id='10'>
<name>a0</name>
<uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid>
<metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0"
xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0">
<ovirt-tune:qos/>
<ovirt-vm:vm/>
<ovirt-instance:instance>
<ovirt-instance:value type="dict">
<ovirt-instance:item key="devices" type="list">
<ovirt-instance:item index="0" type="dict">
<ovirt-instance:item key="index"
type="str">2</ovirt-instance:item>
<ovirt-instance:item key="iface"
type="str">ide</ovirt-instance:item>
<ovirt-instance:item key="specParams" type="dict">
<ovirt-instance:item key="path" type="str"/>
</ovirt-instance:item>
<ovirt-instance:item key="readonly"
type="str">true</ovirt-instance:item>
<ovirt-instance:item key="deviceId"
type="str">e59c985c-46c2-4489-b355-a6f374125eb9</ovirt-instance:item>
<ovirt-instance:item key="address" type="dict">
<ovirt-instance:item key="bus"
type="str">1</ovirt-instance:item>
<ovirt-instance:item key="controller"
type="str">0</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">drive</ovirt-instance:item>
<ovirt-instance:item key="target"
type="str">0</ovirt-instance:item>
<ovirt-instance:item key="unit"
type="str">0</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="device"
type="str">cdrom</ovirt-instance:item>
<ovirt-instance:item key="shared"
type="str">false</ovirt-instance:item>
<ovirt-instance:item key="path" type="str"/>
<ovirt-instance:item key="type"
type="str">disk</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item index="1" type="dict">
<ovirt-instance:item key="poolID"
type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item>
<ovirt-instance:item key="volumeInfo" type="dict">
<ovirt-instance:item key="domainID"
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
<ovirt-instance:item key="volType"
type="str">path</ovirt-instance:item>
<ovirt-instance:item key="leaseOffset"
type="int">0</ovirt-instance:item>
<ovirt-instance:item key="path"
type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="volumeID"
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="leasePath"
type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease</ovirt-instance:item>
<ovirt-instance:item key="imageID"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="index"
type="int">0</ovirt-instance:item>
<ovirt-instance:item key="iface"
type="str">virtio</ovirt-instance:item>
<ovirt-instance:item key="apparentsize"
type="str">8589934592</ovirt-instance:item>
<ovirt-instance:item key="imageID"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
<ovirt-instance:item key="readonly"
type="str">false</ovirt-instance:item>
<ovirt-instance:item key="shared"
type="str">false</ovirt-instance:item>
<ovirt-instance:item key="truesize"
type="str">0</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">disk</ovirt-instance:item>
<ovirt-instance:item key="domainID"
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
<ovirt-instance:item key="reqsize"
type="str">0</ovirt-instance:item>
<ovirt-instance:item key="format"
type="str">raw</ovirt-instance:item>
<ovirt-instance:item key="deviceId"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
<ovirt-instance:item key="address" type="dict">
<ovirt-instance:item key="function"
type="str">0x0</ovirt-instance:item>
<ovirt-instance:item key="bus"
type="str">0x00</ovirt-instance:item>
<ovirt-instance:item key="domain"
type="str">0x0000</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">pci</ovirt-instance:item>
<ovirt-instance:item key="slot"
type="str">0x05</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="device"
type="str">disk</ovirt-instance:item>
<ovirt-instance:item key="path"
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="propagateErrors"
type="str">off</ovirt-instance:item>
<ovirt-instance:item key="optional"
type="str">false</ovirt-instance:item>
<ovirt-instance:item key="bootOrder"
type="str">1</ovirt-instance:item>
<ovirt-instance:item key="volumeID"
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="specParams" type="dict"/>
<ovirt-instance:item key="volumeChain" type="list">
<ovirt-instance:item index="0" type="dict">
<ovirt-instance:item key="domainID"
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
<ovirt-instance:item key="volType"
type="str">path</ovirt-instance:item>
<ovirt-instance:item key="leaseOffset"
type="int">0</ovirt-instance:item>
<ovirt-instance:item key="path"
type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="volumeID"
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="leasePath"
type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease</ovirt-instance:item>
<ovirt-instance:item key="imageID"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:value>
</ovirt-instance:instance>
</metadata>
<maxMemory slots='16' unit='KiB'>4294967296</maxMemory>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static' current='2'>16</vcpu>
<cputune>
<shares>1020</shares>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>oVirt</entry>
<entry name='product'>oVirt Node</entry>
<entry name='version'>7-3.1611.el7.centos</entry>
<entry
name='serial'>ccd79775-c888-4789-975a-fde1143dffc9</entry>
<entry name='uuid'>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64'
machine='pc-i440fx-rhel7.2.0'>hvm</type>
<bios useserial='yes'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>Haswell-noTSX</model>
<topology sockets='16' cores='1' threads='1'/>
<numa>
<cell id='0' cpus='0-1' memory='4194304'
unit='KiB'/>
</numa>
</cpu>
<clock offset='variable' adjustment='0' basis='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<controller type='virtio-serial' index='0' ports='16'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
</controller>
<controller type='usb' index='0'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='00:1a:4a:16:01:51'/>
<source bridge='ovirtmgmt'/>
<target dev='vnet0'/>
<model type='virtio'/>
<filterref filter='vdsm-no-mac-spoofing'/>
<link state='up'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
</interface>
<serial type='unix'>
<source mode='bind'
path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='unix'>
<source mode='bind'
path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='unix'>
<source mode='bind'
path='/var/lib/libvirt/qemu/channels/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.com.redhat.rhevm.vdsm'/>
<target type='virtio' name='com.redhat.rhevm.vdsm'
state='disconnected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0'
port='1'/>
</channel>
<channel type='unix'>
<source mode='bind'
path='/var/lib/libvirt/qemu/channels/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0'
state='disconnected'/>
<alias name='channel1'/>
<address type='virtio-serial' controller='0' bus='0'
port='2'/>
</channel>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'
state='disconnected'/>
<alias name='channel2'/>
<address type='virtio-serial' controller='0' bus='0'
port='3'/>
</channel>
<input type='mouse' bus='ps2'>
<alias name='input0'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input1'/>
</input>
<graphics type='vnc' port='5900' autoport='yes'
listen='192.168.1.53' keymap='en-us'
passwdValidTo='1970-01-01T00:00:01'>
<listen type='network' address='192.168.1.53'
network='vdsm-ovirtmgmt'/>
</graphics>
<graphics type='spice' tlsPort='5901' autoport='yes'
listen='192.168.1.53' keymap='en-us' defaultMode='secure'
passwdValidTo='1970-01-01T00:00:01'>
<listen type='network' address='192.168.1.53'
network='vdsm-ovirtmgmt'/>
<channel name='main' mode='secure'/>
<channel name='display' mode='secure'/>
<channel name='inputs' mode='secure'/>
<channel name='cursor' mode='secure'/>
<channel name='playback' mode='secure'/>
<channel name='record' mode='secure'/>
<channel name='smartcard' mode='secure'/>
<channel name='usbredir' mode='secure'/>
</graphics>
<sound model='ich6'>
<alias name='sound0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x06'
function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='8192'
vgamem='16384' heads='1'
primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x07'
function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='selinux' relabel='yes'>
<label>system_u:system_r:svirt_t:s0:c207,c629</label>
<imagelabel>system_u:object_r:svirt_image_t:s0:c207,c629</imagelabel>
</seclabel>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+107:+107</label>
<imagelabel>+107:+107</imagelabel>
</seclabel>
</domain>
On 03/15/2017 01:28 PM, Francesco Romani wrote:
Hi everyone,
This is both a report of the current state of my Vdsm patches for Engine
XML support, and a proposal to how move forward and solve
the current open issues.
TL;DR:
1. we can and IMO should reuse the current JSON schema to describe the
structure (layout) and the types of the metadata section.
2. we don't need a priori validation of stuff in the metadata section.
We will just raise in the creation flow if data is missing, or wrong,
according to our schema.
2. we will add *few* items to the metadata section, only thing we can't
express clearly-or at all in the libvirt XML. Redundancy and verbosiness
will be thus kept at bay
3. I believe [3] is the best tool to do (de)serialize data to the
metadata section. Existing tools fits poorly in our very specific use case
Examples below
+++
Long(er) discussion:
I have working code[1][2] to encode any custom, picklable, python
object in the metadata section.
We should decide which module will do the actual python<=>XML
transformation.
Please note that this actually also influences how the data in the
medata section look like, so the two things are a bit coupled.
I'm eager to reinvent another wheel, but after
initial evaluation I honestly think that my pyxmlpickle[3] is the best
tool for the job over the current alternatives: plistlib[4] and
xmltodict[5].
I added the initial rationale here:
https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
I have completed the initial draft of patches to make it possible to
initialize devices from their XML representation [6]. This is the bare
minimum we need to support the Engine XML, and we *need* this anyway to
unlock the cleanup we planned and I outlined in my google doc.
So we are progressing, but I'd like to speed up things. Those [6]
patches are not yet complete, many flows are not covered or tested; but
they are good enough to demonstrate that there *are* pieces of
information wen need to properly initialize the devices, but we can't
easily extract from the XML.
First examples that come to my mind are the storage.Drive UUIDs; there
could also be some ambiguity I'm investigating right now for
displayIp/displayNetwork in Graphics devices. In [6] there are various
TODO to mark more of those cases. Most likely, few more cases will pop
out as I cover all the flows we support.
Long story short: it is hard to correctly rebuild the device conf from
the XML. This is why in [6] I added the 'meta' argument to from_xml_tree
classmethod in [7].
'meta' is supposed to be the device metadata: extra data related to a
device which doesn't (yet) fit in the libvirt XML representation.
For example, we can store 'displayIp' and 'displayNetwork' here and be
done with that: using both per-device metadata and the XML
representation of one graphic device, we will have everything we need to
properly build one graphics.Graphics device.
This example may (hopefully) be bogus, but I'm keeping it because it is
one case easy to follow.
The device metadata is going to be stored in the vm metadata for the
short/mid term future. Even if the per-device metadata idea/RFE is
accepted (no answer yet, but we are working on it), we will not have in
7.4, and unlikely in 7.5.
As it stands today, I believe there are two open questions:
1. do we need a schema for the metadata section?
2. how do we bind the metadata to the devices? How do we know which
metadata belongs to which metadata, if we don't have aliases nor
addresses to match? (e.g. very first time the VM is created!)
My current stance is the following
1. In general, one schema gives us two benefits: 1.a. we document how
the layout of the data should be, including types; 1.b. we can validate
the data we receive.
So yes, we need a schema, but we don't need a *new* schema. I think we
are in good enough shape with the current Vdsm schema: we can just
translate the python object layout to a XML layout.
One example is probably more explicative. Some actual data may look
like, using my pyxmlpickle module:
<domain type='kvm' id='5'>
<name>a0</name>
<uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid>
<metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0"
xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0">
<ovirt-tune:qos/>
<ovirt-vm:vm/>
<ovirt-instance:instance>
<ovirt-instance:value type="dict">
<ovirt-instance:item key="devices" type="list">
<ovirt-instance:item index="0" type="dict">
<ovirt-instance:item key="device"
type="str">vnc</ovirt-instance:item>
<ovirt-instance:item key="specParams" type="dict">
<ovirt-instance:item key="displayNetwork"
type="str">ovirtmgmt</ovirt-instance:item>
<ovirt-instance:item key="displayIp"
type="str">192.168.1.53</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">graphics</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item index="1" type="dict">
<ovirt-instance:item key="device"
type="str">spice</ovirt-instance:item>
<ovirt-instance:item key="specParams" type="dict">
<ovirt-instance:item key="displayNetwork"
type="str">ovirtmgmt</ovirt-instance:item>
<ovirt-instance:item key="displayIp"
type="str">192.168.1.53</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">graphics</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item index="2" type="dict">
<ovirt-instance:item key="poolID"
type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item>
<ovirt-instance:item key="imageID"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
<ovirt-instance:item key="domainID"
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
<ovirt-instance:item key="device"
type="str">disk</ovirt-instance:item>
<ovirt-instance:item key="path"
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="volumeID"
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:value>
</ovirt-instance:instance>
</metadata>
<!-- omitted for brevity -->
</domain>
Please note that yes, this is still verbose, but we don't want to add
much data here, for most of information the most reliable source will
be the domain XML. We will add here only the extra info we can't really
fetch from that.
2. I don't think we need explicit validation: we could just raise along
the way in the creation flow if we don't find some extra metadata we
need. This will also solve the issue that if we reuse the current schema
and we omit most of data, we will lack quite a lot of elements
marked mandatory.
Once we reached agreement, I will update my
https://docs.google.com/document/d/1eD8KSLwwyo2Sk64MytbmE0wBxxMlpIyEI1GRc...
accordingly.
Final note: while device take the lion's share, we will likely need help
from the metadata section also to store VM extra info, but all the above
discussion also applies here.
+++
[1]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
- uses xmltodict
[2]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
ported the 'virt-metadata3' topic to pyxmlpickle
[3] https://github.com/fromanirh/pyxmlpickle
[4] https://docs.python.org/2/library/plistlib.html
[5] https://github.com/martinblech/xmltodict
[6]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
[7] https://gerrit.ovirt.org/#/c/72880/15/lib/vdsm/virt/vmdevices/core.py
--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
11:47 a.m.
On 16 Mar 2017, at 09:45, Francesco Romani <fromani(a)redhat.com>
wrote:
We talked about sending storage device purely on metadata, letting Vdsm
rebuild them and getting the XML like today.
In the other direction, Vdsm will pass through the XML (perhaps only
parts of it, e.g. the devices subtree) like before.
This way we can minimize the changes we are uncertain of, and more
importantly, we can minimize the risky changes.
The following is a realistic example of how the XML could look like if
we send all but the storage devices. It is built using my pyxmlpickle
module (see [3] below).
That’s quite verbose. How much work would it need to actually minimize it and turn it into
something more simple.
Most such stuff should go away and I believe it would be beneficial to make it difficult
to use to discourage using metadata as a generic junkyard
<domain type='kvm' id='10'>
<name>a0</name>
<uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid>
<metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0"
xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0">
<ovirt-tune:qos/>
<ovirt-vm:vm/>
<ovirt-instance:instance>
<ovirt-instance:value type="dict">
<ovirt-instance:item key="devices" type="list">
<ovirt-instance:item index="0" type="dict">
<ovirt-instance:item key="index"
type="str">2</ovirt-instance:item>
<ovirt-instance:item key="iface"
type="str">ide</ovirt-instance:item>
<ovirt-instance:item key="specParams" type="dict">
<ovirt-instance:item key="path" type="str"/>
</ovirt-instance:item>
<ovirt-instance:item key="readonly"
type="str">true</ovirt-instance:item>
<ovirt-instance:item key="deviceId"
type="str">e59c985c-46c2-4489-b355-a6f374125eb9</ovirt-instance:item>
<ovirt-instance:item key="address" type="dict">
<ovirt-instance:item key="bus"
type="str">1</ovirt-instance:item>
<ovirt-instance:item key="controller"
type="str">0</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">drive</ovirt-instance:item>
<ovirt-instance:item key="target"
type="str">0</ovirt-instance:item>
<ovirt-instance:item key="unit"
type="str">0</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="device"
type="str">cdrom</ovirt-instance:item>
<ovirt-instance:item key="shared"
type="str">false</ovirt-instance:item>
<ovirt-instance:item key="path" type="str"/>
<ovirt-instance:item key="type"
type="str">disk</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item index="1" type="dict">
<ovirt-instance:item key="poolID"
type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item>
<ovirt-instance:item key="volumeInfo" type="dict">
<ovirt-instance:item key="domainID"
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
<ovirt-instance:item key="volType"
type="str">path</ovirt-instance:item>
<ovirt-instance:item key="leaseOffset"
type="int">0</ovirt-instance:item>
<ovirt-instance:item key="path"
type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="volumeID"
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="leasePath"
type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease</ovirt-instance:item>
<ovirt-instance:item key="imageID"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="index"
type="int">0</ovirt-instance:item>
<ovirt-instance:item key="iface"
type="str">virtio</ovirt-instance:item>
<ovirt-instance:item key="apparentsize"
type="str">8589934592</ovirt-instance:item>
<ovirt-instance:item key="imageID"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
<ovirt-instance:item key="readonly"
type="str">false</ovirt-instance:item>
<ovirt-instance:item key="shared"
type="str">false</ovirt-instance:item>
<ovirt-instance:item key="truesize"
type="str">0</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">disk</ovirt-instance:item>
<ovirt-instance:item key="domainID"
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
<ovirt-instance:item key="reqsize"
type="str">0</ovirt-instance:item>
<ovirt-instance:item key="format"
type="str">raw</ovirt-instance:item>
<ovirt-instance:item key="deviceId"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
<ovirt-instance:item key="address" type="dict">
<ovirt-instance:item key="function"
type="str">0x0</ovirt-instance:item>
<ovirt-instance:item key="bus"
type="str">0x00</ovirt-instance:item>
<ovirt-instance:item key="domain"
type="str">0x0000</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">pci</ovirt-instance:item>
<ovirt-instance:item key="slot"
type="str">0x05</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="device"
type="str">disk</ovirt-instance:item>
<ovirt-instance:item key="path"
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="propagateErrors"
type="str">off</ovirt-instance:item>
<ovirt-instance:item key="optional"
type="str">false</ovirt-instance:item>
<ovirt-instance:item key="bootOrder"
type="str">1</ovirt-instance:item>
<ovirt-instance:item key="volumeID"
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="specParams" type="dict"/>
<ovirt-instance:item key="volumeChain" type="list">
<ovirt-instance:item index="0" type="dict">
<ovirt-instance:item key="domainID"
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
<ovirt-instance:item key="volType"
type="str">path</ovirt-instance:item>
<ovirt-instance:item key="leaseOffset"
type="int">0</ovirt-instance:item>
<ovirt-instance:item key="path"
type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="volumeID"
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="leasePath"
type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease</ovirt-instance:item>
<ovirt-instance:item key="imageID"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:value>
</ovirt-instance:instance>
</metadata>
<maxMemory slots='16' unit='KiB'>4294967296</maxMemory>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static' current='2'>16</vcpu>
<cputune>
<shares>1020</shares>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>oVirt</entry>
<entry name='product'>oVirt Node</entry>
<entry name='version'>7-3.1611.el7.centos</entry>
<entry
name='serial'>ccd79775-c888-4789-975a-fde1143dffc9</entry>
<entry name='uuid'>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64'
machine='pc-i440fx-rhel7.2.0'>hvm</type>
<bios useserial='yes'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>Haswell-noTSX</model>
<topology sockets='16' cores='1' threads='1'/>
<numa>
<cell id='0' cpus='0-1' memory='4194304'
unit='KiB'/>
</numa>
</cpu>
<clock offset='variable' adjustment='0' basis='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<controller type='virtio-serial' index='0' ports='16'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
</controller>
<controller type='usb' index='0'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='00:1a:4a:16:01:51'/>
<source bridge='ovirtmgmt'/>
<target dev='vnet0'/>
<model type='virtio'/>
<filterref filter='vdsm-no-mac-spoofing'/>
<link state='up'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
</interface>
<serial type='unix'>
<source mode='bind'
path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='unix'>
<source mode='bind'
path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='unix'>
<source mode='bind'
path='/var/lib/libvirt/qemu/channels/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.com.redhat.rhevm.vdsm'/>
<target type='virtio' name='com.redhat.rhevm.vdsm'
state='disconnected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0'
port='1'/>
</channel>
<channel type='unix'>
<source mode='bind'
path='/var/lib/libvirt/qemu/channels/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0'
state='disconnected'/>
<alias name='channel1'/>
<address type='virtio-serial' controller='0' bus='0'
port='2'/>
</channel>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'
state='disconnected'/>
<alias name='channel2'/>
<address type='virtio-serial' controller='0' bus='0'
port='3'/>
</channel>
<input type='mouse' bus='ps2'>
<alias name='input0'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input1'/>
</input>
<graphics type='vnc' port='5900' autoport='yes'
listen='192.168.1.53' keymap='en-us'
passwdValidTo='1970-01-01T00:00:01'>
<listen type='network' address='192.168.1.53'
network='vdsm-ovirtmgmt'/>
</graphics>
<graphics type='spice' tlsPort='5901' autoport='yes'
listen='192.168.1.53' keymap='en-us' defaultMode='secure'
passwdValidTo='1970-01-01T00:00:01'>
<listen type='network' address='192.168.1.53'
network='vdsm-ovirtmgmt'/>
<channel name='main' mode='secure'/>
<channel name='display' mode='secure'/>
<channel name='inputs' mode='secure'/>
<channel name='cursor' mode='secure'/>
<channel name='playback' mode='secure'/>
<channel name='record' mode='secure'/>
<channel name='smartcard' mode='secure'/>
<channel name='usbredir' mode='secure'/>
</graphics>
<sound model='ich6'>
<alias name='sound0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x06'
function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='8192'
vgamem='16384' heads='1'
primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x07'
function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='selinux' relabel='yes'>
<label>system_u:system_r:svirt_t:s0:c207,c629</label>
<imagelabel>system_u:object_r:svirt_image_t:s0:c207,c629</imagelabel>
</seclabel>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+107:+107</label>
<imagelabel>+107:+107</imagelabel>
</seclabel>
</domain>
On 03/15/2017 01:28 PM, Francesco Romani wrote:
> Hi everyone,
>
> This is both a report of the current state of my Vdsm patches for Engine
> XML support, and a proposal to how move forward and solve
> the current open issues.
>
> TL;DR:
> 1. we can and IMO should reuse the current JSON schema to describe the
> structure (layout) and the types of the metadata section.
> 2. we don't need a priori validation of stuff in the metadata section.
> We will just raise in the creation flow if data is missing, or wrong,
> according to our schema.
> 2. we will add *few* items to the metadata section, only thing we can't
> express clearly-or at all in the libvirt XML. Redundancy and verbosiness
> will be thus kept at bay
> 3. I believe [3] is the best tool to do (de)serialize data to the
> metadata section. Existing tools fits poorly in our very specific use case
>
> Examples below
>
> +++
>
> Long(er) discussion:
>
>
> I have working code[1][2] to encode any custom, picklable, python
> object in the metadata section.
>
> We should decide which module will do the actual python<=>XML
> transformation.
> Please note that this actually also influences how the data in the
> medata section look like, so the two things are a bit coupled.
>
> I'm eager to reinvent another wheel, but after
> initial evaluation I honestly think that my pyxmlpickle[3] is the best
> tool for the job over the current alternatives: plistlib[4] and
> xmltodict[5].
>
> I added the initial rationale here:
> https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
>
> I have completed the initial draft of patches to make it possible to
> initialize devices from their XML representation [6]. This is the bare
> minimum we need to support the Engine XML, and we *need* this anyway to
> unlock the cleanup we planned and I outlined in my google doc.
>
> So we are progressing, but I'd like to speed up things. Those [6]
> patches are not yet complete, many flows are not covered or tested; but
> they are good enough to demonstrate that there *are* pieces of
> information wen need to properly initialize the devices, but we can't
> easily extract from the XML.
>
> First examples that come to my mind are the storage.Drive UUIDs; there
> could also be some ambiguity I'm investigating right now for
> displayIp/displayNetwork in Graphics devices. In [6] there are various
> TODO to mark more of those cases. Most likely, few more cases will pop
> out as I cover all the flows we support.
>
> Long story short: it is hard to correctly rebuild the device conf from
> the XML. This is why in [6] I added the 'meta' argument to from_xml_tree
> classmethod in [7].
>
> 'meta' is supposed to be the device metadata: extra data related to a
> device which doesn't (yet) fit in the libvirt XML representation.
> For example, we can store 'displayIp' and 'displayNetwork' here and
be
> done with that: using both per-device metadata and the XML
> representation of one graphic device, we will have everything we need to
> properly build one graphics.Graphics device.
> This example may (hopefully) be bogus, but I'm keeping it because it is
> one case easy to follow.
>
> The device metadata is going to be stored in the vm metadata for the
> short/mid term future. Even if the per-device metadata idea/RFE is
> accepted (no answer yet, but we are working on it), we will not have in
> 7.4, and unlikely in 7.5.
>
> As it stands today, I believe there are two open questions:
>
> 1. do we need a schema for the metadata section?
> 2. how do we bind the metadata to the devices? How do we know which
> metadata belongs to which metadata, if we don't have aliases nor
> addresses to match? (e.g. very first time the VM is created!)
>
> My current stance is the following
> 1. In general, one schema gives us two benefits: 1.a. we document how
> the layout of the data should be, including types; 1.b. we can validate
> the data we receive.
> So yes, we need a schema, but we don't need a *new* schema. I think we
> are in good enough shape with the current Vdsm schema: we can just
> translate the python object layout to a XML layout.
>
> One example is probably more explicative. Some actual data may look
> like, using my pyxmlpickle module:
>
> <domain type='kvm' id='5'>
> <name>a0</name>
> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid>
> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0"
> xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
> xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0">
> <ovirt-tune:qos/>
> <ovirt-vm:vm/>
> <ovirt-instance:instance>
> <ovirt-instance:value type="dict">
> <ovirt-instance:item key="devices" type="list">
> <ovirt-instance:item index="0" type="dict">
> <ovirt-instance:item key="device"
> type="str">vnc</ovirt-instance:item>
> <ovirt-instance:item key="specParams"
type="dict">
> <ovirt-instance:item key="displayNetwork"
> type="str">ovirtmgmt</ovirt-instance:item>
> <ovirt-instance:item key="displayIp"
> type="str">192.168.1.53</ovirt-instance:item>
> </ovirt-instance:item>
> <ovirt-instance:item key="type"
> type="str">graphics</ovirt-instance:item>
> </ovirt-instance:item>
> <ovirt-instance:item index="1" type="dict">
> <ovirt-instance:item key="device"
> type="str">spice</ovirt-instance:item>
> <ovirt-instance:item key="specParams"
type="dict">
> <ovirt-instance:item key="displayNetwork"
> type="str">ovirtmgmt</ovirt-instance:item>
> <ovirt-instance:item key="displayIp"
> type="str">192.168.1.53</ovirt-instance:item>
> </ovirt-instance:item>
> <ovirt-instance:item key="type"
> type="str">graphics</ovirt-instance:item>
> </ovirt-instance:item>
> <ovirt-instance:item index="2" type="dict">
> <ovirt-instance:item key="poolID"
>
type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item>
> <ovirt-instance:item key="imageID"
>
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
> <ovirt-instance:item key="domainID"
>
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
> <ovirt-instance:item key="device"
> type="str">disk</ovirt-instance:item>
> <ovirt-instance:item key="path"
>
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
> <ovirt-instance:item key="volumeID"
>
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
> </ovirt-instance:item>
> </ovirt-instance:item>
> </ovirt-instance:value>
> </ovirt-instance:instance>
> </metadata>
> <!-- omitted for brevity -->
> </domain>
>
>
> Please note that yes, this is still verbose, but we don't want to add
> much data here, for most of information the most reliable source will
> be the domain XML. We will add here only the extra info we can't really
> fetch from that.
>
> 2. I don't think we need explicit validation: we could just raise along
> the way in the creation flow if we don't find some extra metadata we
> need. This will also solve the issue that if we reuse the current schema
> and we omit most of data, we will lack quite a lot of elements
> marked mandatory.
>
> Once we reached agreement, I will update my
>
https://docs.google.com/document/d/1eD8KSLwwyo2Sk64MytbmE0wBxxMlpIyEI1GRc...
> accordingly.
>
> Final note: while device take the lion's share, we will likely need help
> from the metadata section also to store VM extra info, but all the above
> discussion also applies here.
>
> +++
>
> [1]
>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
> - uses xmltodict
> [2]
>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
> ported the 'virt-metadata3' topic to pyxmlpickle
> [3] https://github.com/fromanirh/pyxmlpickle
> [4] https://docs.python.org/2/library/plistlib.html
> [5] https://github.com/martinblech/xmltodict
> [6]
>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
> [7] https://gerrit.ovirt.org/#/c/72880/15/lib/vdsm/virt/vmdevices/core.py
>
--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
1:26 p.m.
On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
> On 16 Mar 2017, at 09:45, Francesco Romani
<fromani(a)redhat.com> wrote:
>
> We talked about sending storage device purely on metadata, letting Vdsm
> rebuild them and getting the XML like today.
>
> In the other direction, Vdsm will pass through the XML (perhaps only
> parts of it, e.g. the devices subtree) like before.
>
> This way we can minimize the changes we are uncertain of, and more
> importantly, we can minimize the risky changes.
>
>
> The following is a realistic example of how the XML could look like if
> we send all but the storage devices. It is built using my pyxmlpickle
> module (see [3] below).
That’s quite verbose. How much work would it need to actually minimize it and turn it
into something more simple.
Most such stuff should go away and I believe it would be beneficial to make it difficult
to use to discourage using metadata as a generic junkyard
It is verbose because it is generic - indeed perhaps too generic.
I can try something else based on a concept from Martin Polednik. Will
follow up soon.
Bests,
>
> <domain type='kvm' id='10'>
> <name>a0</name>
> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid>
> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0"
> xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
> xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0">
> <ovirt-tune:qos/>
> <ovirt-vm:vm/>
> <ovirt-instance:instance>
> <ovirt-instance:value type="dict">
> <ovirt-instance:item key="devices" type="list">
> <ovirt-instance:item index="0" type="dict">
> <ovirt-instance:item key="index"
> type="str">2</ovirt-instance:item>
> <ovirt-instance:item key="iface"
> type="str">ide</ovirt-instance:item>
> <ovirt-instance:item key="specParams"
type="dict">
> <ovirt-instance:item key="path" type="str"/>
> </ovirt-instance:item>
> <ovirt-instance:item key="readonly"
> type="str">true</ovirt-instance:item>
> <ovirt-instance:item key="deviceId"
>
type="str">e59c985c-46c2-4489-b355-a6f374125eb9</ovirt-instance:item>
> <ovirt-instance:item key="address" type="dict">
> <ovirt-instance:item key="bus"
> type="str">1</ovirt-instance:item>
> <ovirt-instance:item key="controller"
> type="str">0</ovirt-instance:item>
> <ovirt-instance:item key="type"
> type="str">drive</ovirt-instance:item>
> <ovirt-instance:item key="target"
> type="str">0</ovirt-instance:item>
> <ovirt-instance:item key="unit"
> type="str">0</ovirt-instance:item>
> </ovirt-instance:item>
> <ovirt-instance:item key="device"
> type="str">cdrom</ovirt-instance:item>
> <ovirt-instance:item key="shared"
> type="str">false</ovirt-instance:item>
> <ovirt-instance:item key="path" type="str"/>
> <ovirt-instance:item key="type"
> type="str">disk</ovirt-instance:item>
> </ovirt-instance:item>
> <ovirt-instance:item index="1" type="dict">
> <ovirt-instance:item key="poolID"
>
type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item>
> <ovirt-instance:item key="volumeInfo"
type="dict">
> <ovirt-instance:item key="domainID"
>
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
> <ovirt-instance:item key="volType"
> type="str">path</ovirt-instance:item>
> <ovirt-instance:item key="leaseOffset"
> type="int">0</ovirt-instance:item>
> <ovirt-instance:item key="path"
>
type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
> <ovirt-instance:item key="volumeID"
>
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
> <ovirt-instance:item key="leasePath"
>
type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease</ovirt-instance:item>
> <ovirt-instance:item key="imageID"
>
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
> </ovirt-instance:item>
> <ovirt-instance:item key="index"
> type="int">0</ovirt-instance:item>
> <ovirt-instance:item key="iface"
> type="str">virtio</ovirt-instance:item>
> <ovirt-instance:item key="apparentsize"
> type="str">8589934592</ovirt-instance:item>
> <ovirt-instance:item key="imageID"
>
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
> <ovirt-instance:item key="readonly"
> type="str">false</ovirt-instance:item>
> <ovirt-instance:item key="shared"
> type="str">false</ovirt-instance:item>
> <ovirt-instance:item key="truesize"
> type="str">0</ovirt-instance:item>
> <ovirt-instance:item key="type"
> type="str">disk</ovirt-instance:item>
> <ovirt-instance:item key="domainID"
>
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
> <ovirt-instance:item key="reqsize"
> type="str">0</ovirt-instance:item>
> <ovirt-instance:item key="format"
> type="str">raw</ovirt-instance:item>
> <ovirt-instance:item key="deviceId"
>
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
> <ovirt-instance:item key="address" type="dict">
> <ovirt-instance:item key="function"
> type="str">0x0</ovirt-instance:item>
> <ovirt-instance:item key="bus"
> type="str">0x00</ovirt-instance:item>
> <ovirt-instance:item key="domain"
> type="str">0x0000</ovirt-instance:item>
> <ovirt-instance:item key="type"
> type="str">pci</ovirt-instance:item>
> <ovirt-instance:item key="slot"
> type="str">0x05</ovirt-instance:item>
> </ovirt-instance:item>
> <ovirt-instance:item key="device"
> type="str">disk</ovirt-instance:item>
> <ovirt-instance:item key="path"
>
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
> <ovirt-instance:item key="propagateErrors"
> type="str">off</ovirt-instance:item>
> <ovirt-instance:item key="optional"
> type="str">false</ovirt-instance:item>
> <ovirt-instance:item key="bootOrder"
> type="str">1</ovirt-instance:item>
> <ovirt-instance:item key="volumeID"
>
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
> <ovirt-instance:item key="specParams"
type="dict"/>
> <ovirt-instance:item key="volumeChain"
type="list">
> <ovirt-instance:item index="0" type="dict">
> <ovirt-instance:item key="domainID"
>
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
> <ovirt-instance:item key="volType"
> type="str">path</ovirt-instance:item>
> <ovirt-instance:item key="leaseOffset"
> type="int">0</ovirt-instance:item>
> <ovirt-instance:item key="path"
>
type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
> <ovirt-instance:item key="volumeID"
>
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
> <ovirt-instance:item key="leasePath"
>
type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease</ovirt-instance:item>
> <ovirt-instance:item key="imageID"
>
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
> </ovirt-instance:item>
> </ovirt-instance:item>
> </ovirt-instance:item>
> </ovirt-instance:item>
> </ovirt-instance:value>
> </ovirt-instance:instance>
> </metadata>
> <maxMemory slots='16' unit='KiB'>4294967296</maxMemory>
> <memory unit='KiB'>4194304</memory>
> <currentMemory unit='KiB'>4194304</currentMemory>
> <vcpu placement='static' current='2'>16</vcpu>
> <cputune>
> <shares>1020</shares>
> </cputune>
> <resource>
> <partition>/machine</partition>
> </resource>
> <sysinfo type='smbios'>
> <system>
> <entry name='manufacturer'>oVirt</entry>
> <entry name='product'>oVirt Node</entry>
> <entry name='version'>7-3.1611.el7.centos</entry>
> <entry
name='serial'>ccd79775-c888-4789-975a-fde1143dffc9</entry>
> <entry
name='uuid'>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</entry>
> </system>
> </sysinfo>
> <os>
> <type arch='x86_64'
machine='pc-i440fx-rhel7.2.0'>hvm</type>
> <bios useserial='yes'/>
> <smbios mode='sysinfo'/>
> </os>
> <features>
> <acpi/>
> </features>
> <cpu mode='custom' match='exact'>
> <model fallback='allow'>Haswell-noTSX</model>
> <topology sockets='16' cores='1' threads='1'/>
> <numa>
> <cell id='0' cpus='0-1' memory='4194304'
unit='KiB'/>
> </numa>
> </cpu>
> <clock offset='variable' adjustment='0' basis='utc'>
> <timer name='rtc' tickpolicy='catchup'/>
> <timer name='pit' tickpolicy='delay'/>
> <timer name='hpet' present='no'/>
> </clock>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>destroy</on_crash>
> <devices>
> <emulator>/usr/libexec/qemu-kvm</emulator>
> <controller type='virtio-serial' index='0'
ports='16'>
> <alias name='virtio-serial0'/>
> <address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
> function='0x0'/>
> </controller>
> <controller type='usb' index='0'>
> <alias name='usb'/>
> <address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
> function='0x2'/>
> </controller>
> <controller type='pci' index='0' model='pci-root'>
> <alias name='pci.0'/>
> </controller>
> <controller type='ide' index='0'>
> <alias name='ide'/>
> <address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
> function='0x1'/>
> </controller>
> <interface type='bridge'>
> <mac address='00:1a:4a:16:01:51'/>
> <source bridge='ovirtmgmt'/>
> <target dev='vnet0'/>
> <model type='virtio'/>
> <filterref filter='vdsm-no-mac-spoofing'/>
> <link state='up'/>
> <alias name='net0'/>
> <address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
> function='0x0'/>
> </interface>
> <serial type='unix'>
> <source mode='bind'
>
path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/>
> <target port='0'/>
> <alias name='serial0'/>
> </serial>
> <console type='unix'>
> <source mode='bind'
>
path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/>
> <target type='serial' port='0'/>
> <alias name='serial0'/>
> </console>
> <channel type='unix'>
> <source mode='bind'
>
path='/var/lib/libvirt/qemu/channels/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.com.redhat.rhevm.vdsm'/>
> <target type='virtio' name='com.redhat.rhevm.vdsm'
> state='disconnected'/>
> <alias name='channel0'/>
> <address type='virtio-serial' controller='0' bus='0'
port='1'/>
> </channel>
> <channel type='unix'>
> <source mode='bind'
>
path='/var/lib/libvirt/qemu/channels/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.org.qemu.guest_agent.0'/>
> <target type='virtio' name='org.qemu.guest_agent.0'
> state='disconnected'/>
> <alias name='channel1'/>
> <address type='virtio-serial' controller='0' bus='0'
port='2'/>
> </channel>
> <channel type='spicevmc'>
> <target type='virtio' name='com.redhat.spice.0'
state='disconnected'/>
> <alias name='channel2'/>
> <address type='virtio-serial' controller='0' bus='0'
port='3'/>
> </channel>
> <input type='mouse' bus='ps2'>
> <alias name='input0'/>
> </input>
> <input type='keyboard' bus='ps2'>
> <alias name='input1'/>
> </input>
> <graphics type='vnc' port='5900' autoport='yes'
> listen='192.168.1.53' keymap='en-us'
passwdValidTo='1970-01-01T00:00:01'>
> <listen type='network' address='192.168.1.53'
> network='vdsm-ovirtmgmt'/>
> </graphics>
> <graphics type='spice' tlsPort='5901' autoport='yes'
> listen='192.168.1.53' keymap='en-us' defaultMode='secure'
> passwdValidTo='1970-01-01T00:00:01'>
> <listen type='network' address='192.168.1.53'
> network='vdsm-ovirtmgmt'/>
> <channel name='main' mode='secure'/>
> <channel name='display' mode='secure'/>
> <channel name='inputs' mode='secure'/>
> <channel name='cursor' mode='secure'/>
> <channel name='playback' mode='secure'/>
> <channel name='record' mode='secure'/>
> <channel name='smartcard' mode='secure'/>
> <channel name='usbredir' mode='secure'/>
> </graphics>
> <sound model='ich6'>
> <alias name='sound0'/>
> <address type='pci' domain='0x0000' bus='0x00'
slot='0x06'
> function='0x0'/>
> </sound>
> <video>
> <model type='qxl' ram='65536' vram='8192'
vgamem='16384' heads='1'
> primary='yes'/>
> <alias name='video0'/>
> <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
> function='0x0'/>
> </video>
> <memballoon model='virtio'>
> <alias name='balloon0'/>
> <address type='pci' domain='0x0000' bus='0x00'
slot='0x07'
> function='0x0'/>
> </memballoon>
> </devices>
> <seclabel type='dynamic' model='selinux'
relabel='yes'>
> <label>system_u:system_r:svirt_t:s0:c207,c629</label>
> <imagelabel>system_u:object_r:svirt_image_t:s0:c207,c629</imagelabel>
> </seclabel>
> <seclabel type='dynamic' model='dac' relabel='yes'>
> <label>+107:+107</label>
> <imagelabel>+107:+107</imagelabel>
> </seclabel>
> </domain>
>
>
> On 03/15/2017 01:28 PM, Francesco Romani wrote:
>> Hi everyone,
>>
>> This is both a report of the current state of my Vdsm patches for Engine
>> XML support, and a proposal to how move forward and solve
>> the current open issues.
>>
>> TL;DR:
>> 1. we can and IMO should reuse the current JSON schema to describe the
>> structure (layout) and the types of the metadata section.
>> 2. we don't need a priori validation of stuff in the metadata section.
>> We will just raise in the creation flow if data is missing, or wrong,
>> according to our schema.
>> 2. we will add *few* items to the metadata section, only thing we can't
>> express clearly-or at all in the libvirt XML. Redundancy and verbosiness
>> will be thus kept at bay
>> 3. I believe [3] is the best tool to do (de)serialize data to the
>> metadata section. Existing tools fits poorly in our very specific use case
>>
>> Examples below
>>
>> +++
>>
>> Long(er) discussion:
>>
>>
>> I have working code[1][2] to encode any custom, picklable, python
>> object in the metadata section.
>>
>> We should decide which module will do the actual python<=>XML
>> transformation.
>> Please note that this actually also influences how the data in the
>> medata section look like, so the two things are a bit coupled.
>>
>> I'm eager to reinvent another wheel, but after
>> initial evaluation I honestly think that my pyxmlpickle[3] is the best
>> tool for the job over the current alternatives: plistlib[4] and
>> xmltodict[5].
>>
>> I added the initial rationale here:
>> https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
>>
>> I have completed the initial draft of patches to make it possible to
>> initialize devices from their XML representation [6]. This is the bare
>> minimum we need to support the Engine XML, and we *need* this anyway to
>> unlock the cleanup we planned and I outlined in my google doc.
>>
>> So we are progressing, but I'd like to speed up things. Those [6]
>> patches are not yet complete, many flows are not covered or tested; but
>> they are good enough to demonstrate that there *are* pieces of
>> information wen need to properly initialize the devices, but we can't
>> easily extract from the XML.
>>
>> First examples that come to my mind are the storage.Drive UUIDs; there
>> could also be some ambiguity I'm investigating right now for
>> displayIp/displayNetwork in Graphics devices. In [6] there are various
>> TODO to mark more of those cases. Most likely, few more cases will pop
>> out as I cover all the flows we support.
>>
>> Long story short: it is hard to correctly rebuild the device conf from
>> the XML. This is why in [6] I added the 'meta' argument to from_xml_tree
>> classmethod in [7].
>>
>> 'meta' is supposed to be the device metadata: extra data related to a
>> device which doesn't (yet) fit in the libvirt XML representation.
>> For example, we can store 'displayIp' and 'displayNetwork' here
and be
>> done with that: using both per-device metadata and the XML
>> representation of one graphic device, we will have everything we need to
>> properly build one graphics.Graphics device.
>> This example may (hopefully) be bogus, but I'm keeping it because it is
>> one case easy to follow.
>>
>> The device metadata is going to be stored in the vm metadata for the
>> short/mid term future. Even if the per-device metadata idea/RFE is
>> accepted (no answer yet, but we are working on it), we will not have in
>> 7.4, and unlikely in 7.5.
>>
>> As it stands today, I believe there are two open questions:
>>
>> 1. do we need a schema for the metadata section?
>> 2. how do we bind the metadata to the devices? How do we know which
>> metadata belongs to which metadata, if we don't have aliases nor
>> addresses to match? (e.g. very first time the VM is created!)
>>
>> My current stance is the following
>> 1. In general, one schema gives us two benefits: 1.a. we document how
>> the layout of the data should be, including types; 1.b. we can validate
>> the data we receive.
>> So yes, we need a schema, but we don't need a *new* schema. I think we
>> are in good enough shape with the current Vdsm schema: we can just
>> translate the python object layout to a XML layout.
>>
>> One example is probably more explicative. Some actual data may look
>> like, using my pyxmlpickle module:
>>
>> <domain type='kvm' id='5'>
>> <name>a0</name>
>> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid>
>> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0"
>> xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
>> xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0">
>> <ovirt-tune:qos/>
>> <ovirt-vm:vm/>
>> <ovirt-instance:instance>
>> <ovirt-instance:value type="dict">
>> <ovirt-instance:item key="devices" type="list">
>> <ovirt-instance:item index="0" type="dict">
>> <ovirt-instance:item key="device"
>> type="str">vnc</ovirt-instance:item>
>> <ovirt-instance:item key="specParams"
type="dict">
>> <ovirt-instance:item key="displayNetwork"
>> type="str">ovirtmgmt</ovirt-instance:item>
>> <ovirt-instance:item key="displayIp"
>> type="str">192.168.1.53</ovirt-instance:item>
>> </ovirt-instance:item>
>> <ovirt-instance:item key="type"
>> type="str">graphics</ovirt-instance:item>
>> </ovirt-instance:item>
>> <ovirt-instance:item index="1" type="dict">
>> <ovirt-instance:item key="device"
>> type="str">spice</ovirt-instance:item>
>> <ovirt-instance:item key="specParams"
type="dict">
>> <ovirt-instance:item key="displayNetwork"
>> type="str">ovirtmgmt</ovirt-instance:item>
>> <ovirt-instance:item key="displayIp"
>> type="str">192.168.1.53</ovirt-instance:item>
>> </ovirt-instance:item>
>> <ovirt-instance:item key="type"
>> type="str">graphics</ovirt-instance:item>
>> </ovirt-instance:item>
>> <ovirt-instance:item index="2" type="dict">
>> <ovirt-instance:item key="poolID"
>>
type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item>
>> <ovirt-instance:item key="imageID"
>>
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
>> <ovirt-instance:item key="domainID"
>>
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
>> <ovirt-instance:item key="device"
>> type="str">disk</ovirt-instance:item>
>> <ovirt-instance:item key="path"
>>
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
>> <ovirt-instance:item key="volumeID"
>>
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
>> </ovirt-instance:item>
>> </ovirt-instance:item>
>> </ovirt-instance:value>
>> </ovirt-instance:instance>
>> </metadata>
>> <!-- omitted for brevity -->
>> </domain>
>>
>>
>> Please note that yes, this is still verbose, but we don't want to add
>> much data here, for most of information the most reliable source will
>> be the domain XML. We will add here only the extra info we can't really
>> fetch from that.
>>
>> 2. I don't think we need explicit validation: we could just raise along
>> the way in the creation flow if we don't find some extra metadata we
>> need. This will also solve the issue that if we reuse the current schema
>> and we omit most of data, we will lack quite a lot of elements
>> marked mandatory.
>>
>> Once we reached agreement, I will update my
>>
https://docs.google.com/document/d/1eD8KSLwwyo2Sk64MytbmE0wBxxMlpIyEI1GRc...
>> accordingly.
>>
>> Final note: while device take the lion's share, we will likely need help
>> from the metadata section also to store VM extra info, but all the above
>> discussion also applies here.
>>
>> +++
>>
>> [1]
>>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
>> - uses xmltodict
>> [2]
>>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
>> ported the 'virt-metadata3' topic to pyxmlpickle
>> [3] https://github.com/fromanirh/pyxmlpickle
>> [4] https://docs.python.org/2/library/plistlib.html
>> [5] https://github.com/martinblech/xmltodict
>> [6]
>>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
>> [7] https://gerrit.ovirt.org/#/c/72880/15/lib/vdsm/virt/vmdevices/core.py
>>
> --
> Francesco Romani
> Red Hat Engineering Virtualization R & D
> IRC: fromani
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
>
--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
8:03 p.m.
On 03/16/2017 01:26 PM, Francesco Romani wrote:
On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
>> On 16 Mar 2017, at 09:45, Francesco Romani <fromani(a)redhat.com> wrote:
>>
>> We talked about sending storage device purely on metadata, letting Vdsm
>> rebuild them and getting the XML like today.
>>
>> In the other direction, Vdsm will pass through the XML (perhaps only
>> parts of it, e.g. the devices subtree) like before.
>>
>> This way we can minimize the changes we are uncertain of, and more
>> importantly, we can minimize the risky changes.
>>
>>
>> The following is a realistic example of how the XML could look like if
>> we send all but the storage devices. It is built using my pyxmlpickle
>> module (see [3] below).
> That’s quite verbose. How much work would it need to actually minimize it and turn it
into something more simple.
> Most such stuff should go away and I believe it would be beneficial to make it
difficult to use to discourage using metadata as a generic junkyard
It is verbose because it is generic - indeed perhaps too generic.
I can try something else based on a concept from Martin Polednik. Will
follow up soon.
Early preview:
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
still plenty of TODOs, I expect to be reviewable material worst case
monday morning.
--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
3:57 p.m.
On 03/16/2017 08:03 PM, Francesco Romani wrote:
On 03/16/2017 01:26 PM, Francesco Romani wrote:
> On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
>>> On 16 Mar 2017, at 09:45, Francesco Romani <fromani(a)redhat.com> wrote:
>>>
>>> We talked about sending storage device purely on metadata, letting Vdsm
>>> rebuild them and getting the XML like today.
>>>
>>> In the other direction, Vdsm will pass through the XML (perhaps only
>>> parts of it, e.g. the devices subtree) like before.
>>>
>>> This way we can minimize the changes we are uncertain of, and more
>>> importantly, we can minimize the risky changes.
>>>
>>>
>>> The following is a realistic example of how the XML could look like if
>>> we send all but the storage devices. It is built using my pyxmlpickle
>>> module (see [3] below).
>> That’s quite verbose. How much work would it need to actually minimize it and
turn it into something more simple.
>> Most such stuff should go away and I believe it would be beneficial to make it
difficult to use to discourage using metadata as a generic junkyard
> It is verbose because it is generic - indeed perhaps too generic.
> I can try something else based on a concept from Martin Polednik. Will
> follow up soon.
Early preview:
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
still plenty of TODOs, I expect to be reviewable material worst case
monday morning.
This is how typical XML could look like:
<metadata>
<ovirt-tune:qos />
<ovirt-vm:vm />
<devices>
<ovirt-instance:graphics>
<specParams>
<fileTransferEnable>true</fileTransferEnable>
<copyPasteEnable>true</copyPasteEnable>
<displayIp>192.168.1.51</displayIp>
<keyMap>en-us</keyMap>
<spiceSslCipherSuite>DEFAULT</spiceSslCipherSuite>
<spiceSecureChannels>smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir</spiceSecureChannels>
<displayNetwork>ovirtmgmt</displayNetwork>
</specParams>
</ovirt-instance:graphics>
<ovirt-instance:disk>
<specParams>
<path />
</specParams>
</ovirt-instance:disk>
<ovirt-instance:disk>
<specParams />
<domainID>c578566d-bc61-420c-8f1e-8dfa0a18efd5</domainID>
<volumeID>5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</volumeID>
<poolID>5890a292-0390-01d2-01ed-00000000029a</poolID>
<imageID>66441539-f7ac-4946-8a25-75e422f939d4</imageID>
</ovirt-instance:disk>
</devices>
</metadata>
still working on this
--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
1:14 p.m.
On Fri, Mar 17, 2017 at 4:58 PM Francesco Romani <fromani(a)redhat.com> wrote:
On 03/16/2017 08:03 PM, Francesco Romani wrote:
> On 03/16/2017 01:26 PM, Francesco Romani wrote:
>> On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
>>>> On 16 Mar 2017, at 09:45, Francesco Romani <fromani(a)redhat.com>
wrote:
>>>>
>>>> We talked about sending storage device purely on metadata, letting
Vdsm
>>>> rebuild them and getting the XML like today.
>>>>
>>>> In the other direction, Vdsm will pass through the XML (perhaps only
>>>> parts of it, e.g. the devices subtree) like before.
>>>>
>>>> This way we can minimize the changes we are uncertain of, and more
>>>> importantly, we can minimize the risky changes.
>>>>
>>>>
>>>> The following is a realistic example of how the XML could look like
if
>>>> we send all but the storage devices. It is built using my pyxmlpickle
>>>> module (see [3] below).
>>> That’s quite verbose. How much work would it need to actually minimize
it and turn it into something more simple.
>>> Most such stuff should go away and I believe it would be beneficial to
make it difficult to use to discourage using metadata as a generic junkyard
>> It is verbose because it is generic - indeed perhaps too generic.
>> I can try something else based on a concept from Martin Polednik. Will
>> follow up soon.
> Early preview:
>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
>
> still plenty of TODOs, I expect to be reviewable material worst case
> monday morning.
This is how typical XML could look like:
<metadata>
<ovirt-tune:qos />
<ovirt-vm:vm />
<devices>
Why do we need this nesting?
<ovirt-instance:graphics>
What is ovirt-instance?
ovirt-tune and ovirt-vm make sense, I can guess what you mean, I don't have
any
idea what is ovirt-instance.
<specParams>
<fileTransferEnable>true</fileTransferEnable>
<copyPasteEnable>true</copyPasteEnable>
<displayIp>192.168.1.51</displayIp>
<keyMap>en-us</keyMap>
<spiceSslCipherSuite>DEFAULT</spiceSslCipherSuite>
<spiceSecureChannels>smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir</spiceSecureChannels>
<displayNetwork>ovirtmgmt</displayNetwork>
</specParams>
</ovirt-instance:graphics>
<ovirt-instance:disk>
<specParams>
Why do we need this nesting?
<path />
</specParams>
</ovirt-instance:disk>
<ovirt-instance:disk>
<specParams />
<domainID>c578566d-bc61-420c-8f1e-8dfa0a18efd5</domainID>
We are using sd_id, img_id and vol_id now for these in new storage code.
<volumeID>5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</volumeID>
<poolID>5890a292-0390-01d2-01ed-00000000029a</poolID>
<imageID>66441539-f7ac-4946-8a25-75e422f939d4</imageID>
We need more info about disks, like diskType, shared, discard, etc.
Can you show a complete vm xml with metadata?
</ovirt-instance:disk>
</devices>
</metadata>
still working on this
--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
4:52 p.m.
This is a multi-part message in MIME format.
--------------346863F9A850F6C853C7142A
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
On 03/18/2017 01:14 PM, Nir Soffer wrote:
On Fri, Mar 17, 2017 at 4:58 PM Francesco Romani <fromani(a)redhat.com
<mailto:fromani@redhat.com>> wrote:
On 03/16/2017 08:03 PM, Francesco Romani wrote:
> On 03/16/2017 01:26 PM, Francesco Romani wrote:
>> On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
>>>> On 16 Mar 2017, at 09:45, Francesco Romani
<fromani(a)redhat.com <mailto:fromani@redhat.com>> wrote:
>>>>
>>>> We talked about sending storage device purely on metadata,
letting Vdsm
>>>> rebuild them and getting the XML like today.
>>>>
>>>> In the other direction, Vdsm will pass through the XML
(perhaps only
>>>> parts of it, e.g. the devices subtree) like before.
>>>>
>>>> This way we can minimize the changes we are uncertain of, and
more
>>>> importantly, we can minimize the risky changes.
>>>>
>>>>
>>>> The following is a realistic example of how the XML could
look like if
>>>> we send all but the storage devices. It is built using my
pyxmlpickle
>>>> module (see [3] below).
>>> That’s quite verbose. How much work would it need to actually
minimize it and turn it into something more simple.
>>> Most such stuff should go away and I believe it would be
beneficial to make it difficult to use to discourage using
metadata as a generic junkyard
>> It is verbose because it is generic - indeed perhaps too generic.
>> I can try something else based on a concept from Martin
Polednik. Will
>> follow up soon.
> Early preview:
>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
>
> still plenty of TODOs, I expect to be reviewable material worst case
> monday morning.
This is how typical XML could look like:
<metadata>
<ovirt-tune:qos />
<ovirt-vm:vm />
<devices>
Why do we need this nesting?
We use libvirt metadata for elements; so each direct child of metadata
is a separate metadata grop:
ovirt-tune:qos is one
ovirt-vm:vm is one
and so forth. I don't want to mess up with the existing elements. So we
could be backward compatible at XML level (read: 4.1 XML works without
changes to 4.2)
<ovirt-instance:graphics>
What is ovirt-instance?
Gone, merged with ovirt-vm
ovirt-vm will hold both per-vm metadata and per-device metadata.
<specParams>
<fileTransferEnable>true</fileTransferEnable>
<copyPasteEnable>true</copyPasteEnable>
<displayIp>192.168.1.51</displayIp>
<keyMap>en-us</keyMap>
<spiceSslCipherSuite>DEFAULT</spiceSslCipherSuite>
<spiceSecureChannels>smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir</spiceSecureChannels>
<displayNetwork>ovirtmgmt</displayNetwork>
</specParams>
</ovirt-instance:graphics>
<ovirt-instance:disk>
<specParams>
Why do we need this nesting?
*Here* we had this nesting because the example took vm.conf and brutally
translated to XML. Is a worst case scenario.
Why is it relevant? There it was initial discussion about how to deal
with complex device; to minimize changes, we could
marshal the existing vm.conf into the device metadata, then unmarshal on
Vdsm side and just use it to rebuild the devices
with the very same code we have today (yes, this means
sneaking/embedding vm.conf into the XML)
Should we go that way, it could look like the above.
Let's talk in general now. There are three main use cases requiring nesting:
1. per-vm metadta. Attach key/value pairs. We need one level of nesting
to avoid to mess up with other data. So it could look like
<metadata>
<ovirt-vm:vm>
<foo>1</foo>
<bar>2</bar>
</ovirt-vm:vm>
</metadata>
this is simple and nice and I think is not bothering anyone (hopefully :))
2. per-device metadata: it has to fit into vm section, and we could
possibly have more than one device with metadata, so the simplest format
is something like
<metadata>
<ovirt-vm:vm>
<foo>1</foo>
<bar>2</bar>
<devices>
<device>
<fast>true</fast>
<cheap>false</cheap>
</device>
</devices>
</ovirt-vm:vm>
</metadata>
This is the minimal nesting level we need. We could gather the
per-device metadata in a dict and feed device with it, like with a new
"meta" argument to device constructor, much like "custom" and
"specParams"
Would that look good?
3. QoS. we need to support the current layout for obvious backward
compatibility questions. We could postpone this and use existing code
for some more time, but ultimately this should handled by metadata
module, just because it is supposed to be
the process-wide metadata gateway.
<path />
</specParams>
</ovirt-instance:disk>
<ovirt-instance:disk>
<specParams />
<domainID>c578566d-bc61-420c-8f1e-8dfa0a18efd5</domainID>
We are using sd_id, img_id and vol_id now for these in new storage code.
Noted
<volumeID>5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</volumeID>
<poolID>5890a292-0390-01d2-01ed-00000000029a</poolID>
<imageID>66441539-f7ac-4946-8a25-75e422f939d4</imageID>
We need more info about disks, like diskType, shared, discard, etc.
Can you show a complete vm xml with metadata?
Will post as soon as possible.
--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
--------------346863F9A850F6C853C7142A
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 03/18/2017 01:14 PM, Nir Soffer wrote:<br>
<blockquote
cite="mid:CAMRbyyu0i7vE6Qz7Xr2vz=3BmBVH6CtATavuGhL06==mO_K2bw@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<br>
<div class="gmail_quote">
<div dir="ltr">On Fri, Mar 17, 2017 at 4:58 PM Francesco
Romani <<a moz-do-not-send="true"
href="mailto:fromani@redhat.com">fromani@redhat.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">On
03/16/2017 08:03 PM, Francesco Romani wrote:<br
class="gmail_msg">
> On 03/16/2017 01:26 PM, Francesco Romani wrote:<br
class="gmail_msg">
>> On 03/16/2017 11:47 AM, Michal Skrivanek wrote:<br
class="gmail_msg">
>>>> On 16 Mar 2017, at 09:45, Francesco Romani
<<a moz-do-not-send="true"
href="mailto:fromani@redhat.com" class="gmail_msg"
target="_blank">fromani(a)redhat.com</a>&gt;
wrote:<br
class="gmail_msg">
>>>><br class="gmail_msg">
>>>> We talked about sending storage device
purely on metadata, letting Vdsm<br class="gmail_msg">
>>>> rebuild them and getting the XML like
today.<br class="gmail_msg">
>>>><br class="gmail_msg">
>>>> In the other direction, Vdsm will pass
through the XML (perhaps only<br class="gmail_msg">
>>>> parts of it, e.g. the devices subtree) like
before.<br class="gmail_msg">
>>>><br class="gmail_msg">
>>>> This way we can minimize the changes we are
uncertain of, and more<br class="gmail_msg">
>>>> importantly, we can minimize the risky
changes.<br class="gmail_msg">
>>>><br class="gmail_msg">
>>>><br class="gmail_msg">
>>>> The following is a realistic example of
how the XML could look like if<br class="gmail_msg">
>>>> we send all but the storage devices. It is
built using my pyxmlpickle<br class="gmail_msg">
>>>> module (see [3] below).<br
class="gmail_msg">
>>> That’s quite verbose. How much work would it
need to actually minimize it and turn it into something more
simple.<br class="gmail_msg">
>>> Most such stuff should go away and I believe it
would be beneficial to make it difficult to use to
discourage using metadata as a generic junkyard<br
class="gmail_msg">
>> It is verbose because it is generic - indeed
perhaps too generic.<br class="gmail_msg">
>> I can try something else based on a concept from
Martin Polednik. Will<br class="gmail_msg">
>> follow up soon.<br class="gmail_msg">
> Early preview:<br class="gmail_msg">
> <a moz-do-not-send="true"
href="https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:m...
rel="noreferrer" class="gmail_msg"
target="_blank">https://gerrit.ovirt.org/#/q/status:open+pro...
class="gmail_msg">
><br class="gmail_msg">
> still plenty of TODOs, I expect to be reviewable
material worst case<br class="gmail_msg">
> monday morning.<br class="gmail_msg">
<br class="gmail_msg">
This is how typical XML could look like:<br
class="gmail_msg">
<br class="gmail_msg">
<metadata><br class="gmail_msg">
<ovirt-tune:qos /><br
class="gmail_msg">
<ovirt-vm:vm /><br class="gmail_msg">
<devices><br class="gmail_msg">
</blockquote>
<div><br>
</div>
<div>Why do we need this nesting?</div>
</div>
</div>
</blockquote>
<br>
We use libvirt metadata for elements; so each direct child of
metadata is a separate metadata grop:<br>
<br>
ovirt-tune:qos is one<br>
ovirt-vm:vm is one<br>
<br>
and so forth. I don't want to mess up with the existing elements.
So we could be backward compatible at XML level (read: 4.1 XML works
without changes to 4.2)<br>
<br>
<br>
<blockquote
cite="mid:CAMRbyyu0i7vE6Qz7Xr2vz=3BmBVH6CtATavuGhL06==mO_K2bw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-instance:graphics><br
class="gmail_msg">
</blockquote>
<div><br>
</div>
<div>What is ovirt-instance?</div>
</div>
</div>
</blockquote>
<br>
Gone, merged with ovirt-vm<br>
<br>
ovirt-vm will hold both per-vm metadata and per-device metadata.<br>
<br>
<blockquote
cite="mid:CAMRbyyu0i7vE6Qz7Xr2vz=3BmBVH6CtATavuGhL06==mO_K2bw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<specParams><br
class="gmail_msg">
<fileTransferEnable>true</fileTransferEnable><br
class="gmail_msg">
<copyPasteEnable>true</copyPasteEnable><br
class="gmail_msg">
<displayIp>192.168.1.51</displayIp><br
class="gmail_msg">
<keyMap>en-us</keyMap><br
class="gmail_msg">
<spiceSslCipherSuite>DEFAULT</spiceSslCipherSuite><br
class="gmail_msg">
<br class="gmail_msg">
<spiceSecureChannels>smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir</spiceSecureChannels><br
class="gmail_msg">
<displayNetwork>ovirtmgmt</displayNetwork><br
class="gmail_msg">
</specParams><br
class="gmail_msg">
</ovirt-instance:graphics><br
class="gmail_msg">
<ovirt-instance:disk><br
class="gmail_msg">
<specParams><br
class="gmail_msg">
</blockquote>
<div><br>
</div>
<div>Why do we need this nesting?</div>
</div>
</div>
</blockquote>
<br>
*Here* we had this nesting because the example took vm.conf and
brutally translated to XML. Is a worst case scenario.<br>
<br>
Why is it relevant? There it was initial discussion about how to
deal with complex device; to minimize changes, we could<br>
marshal the existing vm.conf into the device metadata, then
unmarshal on Vdsm side and just use it to rebuild the devices<br>
with the very same code we have today (yes, this means
sneaking/embedding vm.conf into the XML)<br>
<br>
Should we go that way, it could look like the above.<br>
<br>
<br>
Let's talk in general now. There are three main use cases requiring
nesting:<br>
<br>
1. per-vm metadta. Attach key/value pairs. We need one level of
nesting to avoid to mess up with other data. So it could look like<br>
<br>
<metadata><br>
<ovirt-vm:vm><br>
<foo>1</foo><br>
<bar>2</bar><br>
</ovirt-vm:vm><br>
</metadata><br>
<br>
this is simple and nice and I think is not bothering anyone
(hopefully :))<br>
<br>
2. per-device metadata: it has to fit into vm section, and we could
possibly have more than one device with metadata, so the simplest
format is something like<br>
<br>
<metadata><br>
<ovirt-vm:vm><br>
<foo>1</foo><br>
<bar>2</bar><br>
<devices><br>
<device><br>
<fast>true</fast><br>
<cheap>false</cheap><br>
</device><br>
</devices><br>
</ovirt-vm:vm><br>
</metadata><br>
<br>
This is the minimal nesting level we need. We could gather the
per-device metadata in a dict and feed device with it, like with a
new "meta" argument to device constructor, much like "custom" and
"specParams"<br>
<br>
Would that look good?<br>
<br>
<br>
3. QoS. we need to support the current layout for obvious backward
compatibility questions. We could postpone this and use existing
code for some more time, but ultimately this should handled by
metadata module, just because it is supposed to be<br>
the process-wide metadata gateway.<br>
<br>
<blockquote
cite="mid:CAMRbyyu0i7vE6Qz7Xr2vz=3BmBVH6CtATavuGhL06==mO_K2bw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
</div>
</div>
</blockquote>
<blockquote
cite="mid:CAMRbyyu0i7vE6Qz7Xr2vz=3BmBVH6CtATavuGhL06==mO_K2bw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<path /><br
class="gmail_msg">
</specParams><br
class="gmail_msg">
</ovirt-instance:disk><br
class="gmail_msg">
<ovirt-instance:disk><br
class="gmail_msg">
<specParams /><br
class="gmail_msg">
<domainID>c578566d-bc61-420c-8f1e-8dfa0a18efd5</domainID><br
class="gmail_msg">
</blockquote>
<div><br>
</div>
<div>We are using sd_id, img_id and vol_id now for these in
new storage code.</div>
</div>
</div>
</blockquote>
<br>
Noted<br>
<br>
<blockquote
cite="mid:CAMRbyyu0i7vE6Qz7Xr2vz=3BmBVH6CtATavuGhL06==mO_K2bw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<volumeID>5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</volumeID><br
class="gmail_msg">
<poolID>5890a292-0390-01d2-01ed-00000000029a</poolID><br
class="gmail_msg">
<imageID>66441539-f7ac-4946-8a25-75e422f939d4</imageID><br
class="gmail_msg">
</blockquote>
<div><br>
</div>
<div>We need more info about disks, like diskType, shared,
discard, etc.</div>
<div><br>
</div>
<div>Can you show a complete vm xml with metadata?</div>
</div>
</div>
</blockquote>
<br>
Will post as soon as possible.<br>
<br>
<pre class="moz-signature" cols="72">--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani</pre>
</body>
</html>
--------------346863F9A850F6C853C7142A--
12:10 p.m.
New subject: Engine XML: metadata and devices from XML
On 22/03/17 16:52 +0100, Francesco Romani wrote:
On 03/18/2017 01:14 PM, Nir Soffer wrote:
>
>
> On Fri, Mar 17, 2017 at 4:58 PM Francesco Romani <fromani(a)redhat.com
> <mailto:fromani@redhat.com>> wrote:
>
> On 03/16/2017 08:03 PM, Francesco Romani wrote:
> > On 03/16/2017 01:26 PM, Francesco Romani wrote:
> >> On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
> >>>> On 16 Mar 2017, at 09:45, Francesco Romani
> <fromani(a)redhat.com <mailto:fromani@redhat.com>> wrote:
> >>>>
> >>>> We talked about sending storage device purely on metadata,
> letting Vdsm
> >>>> rebuild them and getting the XML like today.
> >>>>
> >>>> In the other direction, Vdsm will pass through the XML
> (perhaps only
> >>>> parts of it, e.g. the devices subtree) like before.
> >>>>
> >>>> This way we can minimize the changes we are uncertain of, and
> more
> >>>> importantly, we can minimize the risky changes.
> >>>>
> >>>>
> >>>> The following is a realistic example of how the XML could
> look like if
> >>>> we send all but the storage devices. It is built using my
> pyxmlpickle
> >>>> module (see [3] below).
> >>> That’s quite verbose. How much work would it need to actually
> minimize it and turn it into something more simple.
> >>> Most such stuff should go away and I believe it would be
> beneficial to make it difficult to use to discourage using
> metadata as a generic junkyard
> >> It is verbose because it is generic - indeed perhaps too generic.
> >> I can try something else based on a concept from Martin
> Polednik. Will
> >> follow up soon.
> > Early preview:
> >
>
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
> >
> > still plenty of TODOs, I expect to be reviewable material worst case
> > monday morning.
>
> This is how typical XML could look like:
>
> <metadata>
> <ovirt-tune:qos />
> <ovirt-vm:vm />
> <devices>
>
>
> Why do we need this nesting?
We use libvirt metadata for elements; so each direct child of metadata
is a separate metadata grop:
ovirt-tune:qos is one
ovirt-vm:vm is one
and so forth. I don't want to mess up with the existing elements. So we
could be backward compatible at XML level (read: 4.1 XML works without
changes to 4.2)
>
>
> <ovirt-instance:graphics>
>
>
> What is ovirt-instance?
Gone, merged with ovirt-vm
ovirt-vm will hold both per-vm metadata and per-device metadata.
>
>
> <specParams>
> <fileTransferEnable>true</fileTransferEnable>
> <copyPasteEnable>true</copyPasteEnable>
> <displayIp>192.168.1.51</displayIp>
> <keyMap>en-us</keyMap>
>
<spiceSslCipherSuite>DEFAULT</spiceSslCipherSuite>
>
>
<spiceSecureChannels>smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir</spiceSecureChannels>
> <displayNetwork>ovirtmgmt</displayNetwork>
> </specParams>
> </ovirt-instance:graphics>
> <ovirt-instance:disk>
> <specParams>
>
>
> Why do we need this nesting?
*Here* we had this nesting because the example took vm.conf and brutally
translated to XML. Is a worst case scenario.
Why is it relevant? There it was initial discussion about how to deal
with complex device; to minimize changes, we could
marshal the existing vm.conf into the device metadata, then unmarshal on
Vdsm side and just use it to rebuild the devices
with the very same code we have today (yes, this means
sneaking/embedding vm.conf into the XML)
Should we go that way, it could look like the above.
Let's talk in general now. There are three main use cases requiring nesting:
1. per-vm metadta. Attach key/value pairs. We need one level of nesting
to avoid to mess up with other data. So it could look like
<metadata>
<ovirt-vm:vm>
<foo>1</foo>
<bar>2</bar>
</ovirt-vm:vm>
</metadata>
this is simple and nice and I think is not bothering anyone (hopefully :))
2. per-device metadata: it has to fit into vm section, and we could
possibly have more than one device with metadata, so the simplest format
is something like
<metadata>
<ovirt-vm:vm>
<foo>1</foo>
<bar>2</bar>
<devices>
<device>
<fast>true</fast>
<cheap>false</cheap>
</device>
</devices>
</ovirt-vm:vm>
</metadata>
This is the minimal nesting level we need. We could gather the
per-device metadata in a dict and feed device with it, like with a new
"meta" argument to device constructor, much like "custom" and
"specParams"
Would that look good?
OK to me. Won't be easy to design an API that doesn't look bad, isn't
too generic and is semi-hard to use, but the design seems fine.
3. QoS. we need to support the current layout for obvious backward
compatibility questions. We could postpone this and use existing code
for some more time, but ultimately this should handled by metadata
module, just because it is supposed to be
the process-wide metadata gateway.
I'd say let's wait with QoS and consider options to change it without
breaking backwards compatibility first. If that fails, let's see how
new code could handle that.
>
>
> <path />
> </specParams>
> </ovirt-instance:disk>
> <ovirt-instance:disk>
> <specParams />
>
> <domainID>c578566d-bc61-420c-8f1e-8dfa0a18efd5</domainID>
>
>
> We are using sd_id, img_id and vol_id now for these in new storage code.
Noted
>
>
>
> <volumeID>5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</volumeID>
>
<poolID>5890a292-0390-01d2-01ed-00000000029a</poolID>
>
> <imageID>66441539-f7ac-4946-8a25-75e422f939d4</imageID>
>
>
> We need more info about disks, like diskType, shared, discard, etc.
>
> Can you show a complete vm xml with metadata?
Will post as soon as possible.
--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
3:27 p.m.
On Wed, Mar 15, 2017 at 2:28 PM Francesco Romani <fromani(a)redhat.com> wrote:
Hi everyone,
This is both a report of the current state of my Vdsm patches for Engine
XML support, and a proposal to how move forward and solve
the current open issues.
TL;DR:
1. we can and IMO should reuse the current JSON schema to describe the
structure (layout) and the types of the metadata section.
2. we don't need a priori validation of stuff in the metadata section.
We will just raise in the creation flow if data is missing, or wrong,
according to our schema.
2. we will add *few* items to the metadata section, only thing we can't
express clearly-or at all in the libvirt XML. Redundancy and verbosiness
will be thus kept at bay
3. I believe [3] is the best tool to do (de)serialize data to the
metadata section. Existing tools fits poorly in our very specific use case
Examples below
+++
Long(er) discussion:
I have working code[1][2] to encode any custom, picklable, python
object in the metadata section.
We should decide which module will do the actual python<=>XML
transformation.
Please note that this actually also influences how the data in the
medata section look like, so the two things are a bit coupled.
I'm eager to reinvent another wheel, but after
initial evaluation I honestly think that my pyxmlpickle[3] is the best
tool for the job over the current alternatives: plistlib[4] and
xmltodict[5].
I added the initial rationale here:
https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
I have completed the initial draft of patches to make it possible to
initialize devices from their XML representation [6]. This is the bare
minimum we need to support the Engine XML, and we *need* this anyway to
unlock the cleanup we planned and I outlined in my google doc.
So we are progressing, but I'd like to speed up things. Those [6]
patches are not yet complete, many flows are not covered or tested; but
they are good enough to demonstrate that there *are* pieces of
information wen need to properly initialize the devices, but we can't
easily extract from the XML.
First examples that come to my mind are the storage.Drive UUIDs; there
could also be some ambiguity I'm investigating right now for
displayIp/displayNetwork in Graphics devices. In [6] there are various
TODO to mark more of those cases. Most likely, few more cases will pop
out as I cover all the flows we support.
Long story short: it is hard to correctly rebuild the device conf from
the XML. This is why in [6] I added the 'meta' argument to from_xml_tree
classmethod in [7].
'meta' is supposed to be the device metadata: extra data related to a
device which doesn't (yet) fit in the libvirt XML representation.
For example, we can store 'displayIp' and 'displayNetwork' here and be
done with that: using both per-device metadata and the XML
representation of one graphic device, we will have everything we need to
properly build one graphics.Graphics device.
This example may (hopefully) be bogus, but I'm keeping it because it is
one case easy to follow.
The device metadata is going to be stored in the vm metadata for the
short/mid term future. Even if the per-device metadata idea/RFE is
accepted (no answer yet, but we are working on it), we will not have in
7.4, and unlikely in 7.5.
As it stands today, I believe there are two open questions:
1. do we need a schema for the metadata section?
2. how do we bind the metadata to the devices? How do we know which
metadata belongs to which metadata, if we don't have aliases nor
addresses to match? (e.g. very first time the VM is created!)
My current stance is the following
1. In general, one schema gives us two benefits: 1.a. we document how
the layout of the data should be, including types; 1.b. we can validate
the data we receive.
So yes, we need a schema, but we don't need a *new* schema. I think we
are in good enough shape with the current Vdsm schema: we can just
translate the python object layout to a XML layout.
One example is probably more explicative. Some actual data may look
like, using my pyxmlpickle module:
<domain type='kvm' id='5'>
<name>a0</name>
<uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid>
<metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0"
The url is broken - do we have this?
xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0">
<ovirt-tune:qos/>
What do we keep here? why does it need its own namespace?
<ovirt-vm:vm/>
What do we keep here? why does it need its own namespace?
Can we merge all namespaces into one generic namespace?
<ovirt-instance:instance>
What is instance?
<ovirt-instance:value type="dict">
<ovirt-instance:item key="devices" type="list">
<ovirt-instance:item index="0" type="dict">
Isn't index redundant?
<ovirt-instance:item key="device"
type="str">vnc</ovirt-instance:item>
<ovirt-instance:item key="specParams" type="dict">
<ovirt-instance:item key="displayNetwork"
type="str">ovirtmgmt</ovirt-instance:item>
<ovirt-instance:item key="displayIp"
type="str">192.168.1.53</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">graphics</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item index="1" type="dict">
<ovirt-instance:item key="device"
type="str">spice</ovirt-instance:item>
<ovirt-instance:item key="specParams" type="dict">
<ovirt-instance:item key="displayNetwork"
type="str">ovirtmgmt</ovirt-instance:item>
<ovirt-instance:item key="displayIp"
type="str">192.168.1.53</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">graphics</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item index="2" type="dict">
<ovirt-instance:item key="poolID"
type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item>
<ovirt-instance:item key="imageID"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
<ovirt-instance:item key="domainID"
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
<ovirt-instance:item key="device"
type="str">disk</ovirt-instance:item>
<ovirt-instance:item key="path"
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="volumeID"
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:value>
</ovirt-instance:instance>
</metadata>
<!-- omitted for brevity -->
</domain>
How about a generic json namespace?
json:
{
"devices": [
{
"foo": "bar"
}
]
}
xml:
<json:object>
<json:key>devices</json:key>
<json:list>
<json:object>
<json:key>foo</json:key>
<json:string>bar</json:string>
</json:object>
</json:list>
</json:object>
We can query and modify this xml using xpath, like this:
>>
r.findall("./metadata/json:object/[json:key='devices']/json:list/json:object/",
namespaces={"json": "http://ovirt.org/json/1.0"})
[<Element '{http://ovirt.org/json/1.0}key' at 0x7fd3386a69d0>, <Element
'{
http://ovirt.org/json/1.0}string' at 0x7fd3386a6a50>]
Not sure this format will be easy to modify, needs to tinker more
with this.
It will be probably easier to parse the entire metadata, change it,
and serialize it back.
Maybe using key and type attribute as you suggest makes it simpler to use,
and we since we convert from xml to python or python to xml, we can
have a "py" namespace.
<py:item>
<py:item key="devices" type="list">
<py:item type="dict">
<py:item key="foo" type="str">bar</py:item>
</py:item>
</py:item>
</py:item>
With this we can have:
>>
r.findall("./metadata/py:item/py:item[@key='devices']/py:item[1]/py:item[@key='foo']",
namespaces={"py": "http://ovirt.org/py/1.0"})[0].text
'bar'
Something like this can be useful for others as well.
Please note that yes, this is still verbose, but we don't want to add
much data here, for most of information the most reliable source will
be the domain XML. We will add here only the extra info we can't really
fetch from that.
2. I don't think we need explicit validation: we could just raise along
the way in the creation flow if we don't find some extra metadata we
need. This will also solve the issue that if we reuse the current schema
and we omit most of data, we will lack quite a lot of elements
marked mandatory.
Once we reached agreement, I will update my
https://docs.google.com/document/d/1eD8KSLwwyo2Sk64MytbmE0wBxxMlpIyEI1GRc...
accordingly.
Final note: while device take the lion's share, we will likely need help
from the metadata section also to store VM extra info, but all the above
discussion also applies here.
+++
[1]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
- uses xmltodict
[2]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
ported the 'virt-metadata3' topic to pyxmlpickle
[3] https://github.com/fromanirh/pyxmlpickle
[4] https://docs.python.org/2/library/plistlib.html
[5] https://github.com/martinblech/xmltodict
[6]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
[7] https://gerrit.ovirt.org/#/c/72880/15/lib/vdsm/virt/vmdevices/core.py
--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
5 p.m.
This is a multi-part message in MIME format.
--------------15F6CAD0BC7802BD4F39C8F8
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Please note that the approach described here is outdated, because
xmlpickle was too easy to use
and too generic: that lead to bloated XML and too high risk of misusing
metadata;
the new approach (https://gerrit.ovirt.org/#/c/74206/) tries to balance
things making usage
convenient but disallowing arbitrarily nesting
On 03/18/2017 03:27 PM, Nir Soffer wrote:
On Wed, Mar 15, 2017 at 2:28 PM Francesco Romani
<fromani(a)redhat.com
<mailto:fromani@redhat.com>> wrote:
Hi everyone,
This is both a report of the current state of my Vdsm patches for
Engine
XML support, and a proposal to how move forward and solve
the current open issues.
TL;DR:
1. we can and IMO should reuse the current JSON schema to describe the
structure (layout) and the types of the metadata section.
2. we don't need a priori validation of stuff in the metadata section.
We will just raise in the creation flow if data is missing, or wrong,
according to our schema.
2. we will add *few* items to the metadata section, only thing we
can't
express clearly-or at all in the libvirt XML. Redundancy and
verbosiness
will be thus kept at bay
3. I believe [3] is the best tool to do (de)serialize data to the
metadata section. Existing tools fits poorly in our very specific
use case
Examples below
+++
Long(er) discussion:
I have working code[1][2] to encode any custom, picklable, python
object in the metadata section.
We should decide which module will do the actual python<=>XML
transformation.
Please note that this actually also influences how the data in the
medata section look like, so the two things are a bit coupled.
I'm eager to reinvent another wheel, but after
initial evaluation I honestly think that my pyxmlpickle[3] is the best
tool for the job over the current alternatives: plistlib[4] and
xmltodict[5].
I added the initial rationale here:
https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
I have completed the initial draft of patches to make it possible to
initialize devices from their XML representation [6]. This is the bare
minimum we need to support the Engine XML, and we *need* this
anyway to
unlock the cleanup we planned and I outlined in my google doc.
So we are progressing, but I'd like to speed up things. Those [6]
patches are not yet complete, many flows are not covered or
tested; but
they are good enough to demonstrate that there *are* pieces of
information wen need to properly initialize the devices, but we can't
easily extract from the XML.
First examples that come to my mind are the storage.Drive UUIDs; there
could also be some ambiguity I'm investigating right now for
displayIp/displayNetwork in Graphics devices. In [6] there are various
TODO to mark more of those cases. Most likely, few more cases will pop
out as I cover all the flows we support.
Long story short: it is hard to correctly rebuild the device conf from
the XML. This is why in [6] I added the 'meta' argument to
from_xml_tree
classmethod in [7].
'meta' is supposed to be the device metadata: extra data related to a
device which doesn't (yet) fit in the libvirt XML representation.
For example, we can store 'displayIp' and 'displayNetwork' here and
be
done with that: using both per-device metadata and the XML
representation of one graphic device, we will have everything we
need to
properly build one graphics.Graphics device.
This example may (hopefully) be bogus, but I'm keeping it because
it is
one case easy to follow.
The device metadata is going to be stored in the vm metadata for the
short/mid term future. Even if the per-device metadata idea/RFE is
accepted (no answer yet, but we are working on it), we will not
have in
7.4, and unlikely in 7.5.
As it stands today, I believe there are two open questions:
1. do we need a schema for the metadata section?
2. how do we bind the metadata to the devices? How do we know which
metadata belongs to which metadata, if we don't have aliases nor
addresses to match? (e.g. very first time the VM is created!)
My current stance is the following
1. In general, one schema gives us two benefits: 1.a. we document how
the layout of the data should be, including types; 1.b. we can
validate
the data we receive.
So yes, we need a schema, but we don't need a *new* schema. I think we
are in good enough shape with the current Vdsm schema: we can just
translate the python object layout to a XML layout.
One example is probably more explicative. Some actual data may look
like, using my pyxmlpickle module:
<domain type='kvm' id='5'>
<name>a0</name>
<uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid>
<metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0"
The url is broken - do we have this?
I guess no. This is old code, dating back to 3.6,/which I'm taking
without changes./
xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0">
<ovirt-tune:qos/>
What do we keep here? why does it need its own namespace?
yes, because of how we use the metadata:
http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainGetMetadata
http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainSetMetadata
we use type =
http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_METADATA_E...
-- I don't think we should change that
<ovirt-vm:vm/>
What do we keep here? why does it need its own namespace?
same as above
Can we merge all namespaces into one generic namespace?
Probably, but we have the nice property that code can trivially access
one subset of metadata, e.g.
only the qos subtree, only the container subtree.
Furthermore if we don't change that we are backward compatible at XML
level (inbound migrations
care about that).
<ovirt-instance:instance>
What is instance?
Gone, it duplicated vm without a good reason
<ovirt-instance:value type="dict">
<ovirt-instance:item key="devices" type="list">
<ovirt-instance:item index="0" type="dict">
Isn't index redundant?
It is, but now is gone
<ovirt-instance:item key="device"
type="str">vnc</ovirt-instance:item>
<ovirt-instance:item key="specParams"
type="dict">
<ovirt-instance:item key="displayNetwork"
type="str">ovirtmgmt</ovirt-instance:item>
<ovirt-instance:item key="displayIp"
type="str">192.168.1.53</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">graphics</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item index="1" type="dict">
<ovirt-instance:item key="device"
type="str">spice</ovirt-instance:item>
<ovirt-instance:item key="specParams"
type="dict">
<ovirt-instance:item key="displayNetwork"
type="str">ovirtmgmt</ovirt-instance:item>
<ovirt-instance:item key="displayIp"
type="str">192.168.1.53</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">graphics</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item index="2" type="dict">
<ovirt-instance:item key="poolID"
type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item>
<ovirt-instance:item key="imageID"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
<ovirt-instance:item key="domainID"
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
<ovirt-instance:item key="device"
type="str">disk</ovirt-instance:item>
<ovirt-instance:item key="path"
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="volumeID"
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:value>
</ovirt-instance:instance>
</metadata>
<!-- omitted for brevity -->
</domain>
How about a generic json namespace?
json:
{
"devices": [
{
"foo": "bar"
}
]
}
xml:
<json:object>
<json:key>devices</json:key>
<json:list>
<json:object>
<json:key>foo</json:key>
<json:string>bar</json:string>
</json:object>
</json:list>
</json:object>
We can query and modify this xml using xpath, like this:
>>>
r.findall("./metadata/json:object/[json:key='devices']/json:list/json:object/",
namespaces={"json": "http://ovirt.org/json/1.0"})
[<Element '{http://ovirt.org/json/1.0}key
<http://ovirt.org/json/1.0%7Dkey>' at 0x7fd3386a69d0>, <Element
'{http://ovirt.org/json/1.0}string
<http://ovirt.org/json/1.0%7Dstring>' at 0x7fd3386a6a50>]
Not sure this format will be easy to modify, needs to tinker more
with this.
It will be probably easier to parse the entire metadata, change it,
and serialize it back.
Maybe using key and type attribute as you suggest makes it simpler to use,
and we since we convert from xml to python or python to xml, we can
have a "py" namespace.
<py:item>
<py:item key="devices" type="list">
<py:item type="dict">
<py:item key="foo" type="str">bar</py:item>
</py:item>
</py:item>
</py:item>
With this we can have:
>>>
r.findall("./metadata/py:item/py:item[@key='devices']/py:item[1]/py:item[@key='foo']",
namespaces={"py": "http://ovirt.org/py/1.0"})[0].text
'bar'
Something like this can be useful for others as well.
Great suggestions, I like the py namespace, but concerns were raised
about the solution here being too generic, so I limited it, see
https://gerrit.ovirt.org/#/c/74206/11
but I will add my pyxmlpickle module
(https://github.com/fromanirh/pyxmlpickle) anyway in the future
according to your comments.
--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
--------------15F6CAD0BC7802BD4F39C8F8
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Please note that the approach described here is outdated, because
xmlpickle was too easy to use</p>
<p>and too generic: that lead to bloated XML and too high risk of
misusing metadata;</p>
<p>the new approach (<a class="moz-txt-link-freetext"
href="https://gerrit.ovirt.org/#/c/74206/">https://gerrit.ov...>)
tries to
balance things making usage</p>
<p>convenient but disallowing arbitrarily nesting<br>
</p>
<br>
<div class="moz-cite-prefix">On 03/18/2017 03:27 PM, Nir Soffer
wrote:<br>
</div>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div dir="ltr" class="gmail_msg">On Wed, Mar 15,
2017 at
2:28 PM Francesco Romani <<a moz-do-not-send="true"
href="mailto:fromani@redhat.com" class="gmail_msg"
target="_blank">fromani(a)redhat.com</a>&gt;
wrote:<br
class="gmail_msg">
</div>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi
everyone,<br class="gmail_msg">
<br class="gmail_msg">
This is both a report of the current state of my Vdsm
patches for Engine<br class="gmail_msg">
XML support, and a proposal to how move forward and solve<br
class="gmail_msg">
the current open issues.<br class="gmail_msg">
<br class="gmail_msg">
TL;DR:<br class="gmail_msg">
1. we can and IMO should reuse the current JSON schema to
describe the<br class="gmail_msg">
structure (layout) and the types of the metadata section.<br
class="gmail_msg">
2. we don't need a priori validation of stuff in the
metadata section.<br class="gmail_msg">
We will just raise in the creation flow if data is
missing, or wrong,<br class="gmail_msg">
according to our schema.<br class="gmail_msg">
2. we will add *few* items to the metadata section, only
thing we can't<br class="gmail_msg">
express clearly-or at all in the libvirt XML. Redundancy
and verbosiness<br class="gmail_msg">
will be thus kept at bay<br class="gmail_msg">
3. I believe [3] is the best tool to do (de)serialize data
to the<br class="gmail_msg">
metadata section. Existing tools fits poorly in our very
specific use case<br class="gmail_msg">
<br class="gmail_msg">
Examples below<br class="gmail_msg">
<br class="gmail_msg">
+++<br class="gmail_msg">
<br class="gmail_msg">
Long(er) discussion:<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
I have working code[1][2] to encode any custom, picklable,
python<br class="gmail_msg">
object in the metadata section.<br class="gmail_msg">
<br class="gmail_msg">
We should decide which module will do the actual
python<=>XML<br class="gmail_msg">
transformation.<br class="gmail_msg">
Please note that this actually also influences how the
data in the<br class="gmail_msg">
medata section look like, so the two things are a bit
coupled.<br class="gmail_msg">
<br class="gmail_msg">
I'm eager to reinvent another wheel, but after<br
class="gmail_msg">
initial evaluation I honestly think that my pyxmlpickle[3]
is the best<br class="gmail_msg">
tool for the job over the current alternatives:
plistlib[4] and<br class="gmail_msg">
xmltodict[5].<br class="gmail_msg">
<br class="gmail_msg">
I added the initial rationale here:<br class="gmail_msg">
<a moz-do-not-send="true"
href="https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG"
rel="noreferrer" class="gmail_msg"
target="_blank">https://gerrit.ovirt.org/#/c/73790/4//COMMIT...
class="gmail_msg">
<br class="gmail_msg">
I have completed the initial draft of patches to make it
possible to<br class="gmail_msg">
initialize devices from their XML representation [6]. This
is the bare<br class="gmail_msg">
minimum we need to support the Engine XML, and we *need*
this anyway to<br class="gmail_msg">
unlock the cleanup we planned and I outlined in my google
doc.<br class="gmail_msg">
<br class="gmail_msg">
So we are progressing, but I'd like to speed up things.
Those [6]<br class="gmail_msg">
patches are not yet complete, many flows are not covered
or tested; but<br class="gmail_msg">
they are good enough to demonstrate that there *are*
pieces of<br class="gmail_msg">
information wen need to properly initialize the devices,
but we can't<br class="gmail_msg">
easily extract from the XML.<br class="gmail_msg">
<br class="gmail_msg">
First examples that come to my mind are the storage.Drive
UUIDs; there<br class="gmail_msg">
could also be some ambiguity I'm investigating right now
for<br class="gmail_msg">
displayIp/displayNetwork in Graphics devices. In [6] there
are various<br class="gmail_msg">
TODO to mark more of those cases. Most likely, few more
cases will pop<br class="gmail_msg">
out as I cover all the flows we support.<br
class="gmail_msg">
<br class="gmail_msg">
Long story short: it is hard to correctly rebuild the
device conf from<br class="gmail_msg">
the XML. This is why in [6] I added the 'meta' argument to
from_xml_tree<br class="gmail_msg">
classmethod in [7].<br class="gmail_msg">
<br class="gmail_msg">
'meta' is supposed to be the device metadata: extra data
related to a<br class="gmail_msg">
device which doesn't (yet) fit in the libvirt XML
representation.<br class="gmail_msg">
For example, we can store 'displayIp' and 'displayNetwork'
here and be<br class="gmail_msg">
done with that: using both per-device metadata and the XML<br
class="gmail_msg">
representation of one graphic device, we will have
everything we need to<br class="gmail_msg">
properly build one graphics.Graphics device.<br
class="gmail_msg">
This example may (hopefully) be bogus, but I'm keeping it
because it is<br class="gmail_msg">
one case easy to follow.<br class="gmail_msg">
<br class="gmail_msg">
The device metadata is going to be stored in the vm
metadata for the<br class="gmail_msg">
short/mid term future. Even if the per-device metadata
idea/RFE is<br class="gmail_msg">
accepted (no answer yet, but we are working on it), we
will not have in<br class="gmail_msg">
7.4, and unlikely in 7.5.<br class="gmail_msg">
<br class="gmail_msg">
As it stands today, I believe there are two open
questions:<br class="gmail_msg">
<br class="gmail_msg">
1. do we need a schema for the metadata section?<br
class="gmail_msg">
2. how do we bind the metadata to the devices? How do we
know which<br class="gmail_msg">
metadata belongs to which metadata, if we don't have
aliases nor<br class="gmail_msg">
addresses to match? (e.g. very first time the VM is
created!)<br class="gmail_msg">
<br class="gmail_msg">
My current stance is the following<br class="gmail_msg">
1. In general, one schema gives us two benefits: 1.a. we
document how<br class="gmail_msg">
the layout of the data should be, including types; 1.b. we
can validate<br class="gmail_msg">
the data we receive.<br class="gmail_msg">
So yes, we need a schema, but we don't need a *new*
schema. I think we<br class="gmail_msg">
are in good enough shape with the current Vdsm schema: we
can just<br class="gmail_msg">
translate the python object layout to a XML layout.<br
class="gmail_msg">
<br class="gmail_msg">
One example is probably more explicative. Some actual data
may look<br class="gmail_msg">
like, using my pyxmlpickle module:<br class="gmail_msg">
<br class="gmail_msg">
<domain type='kvm' id='5'><br
class="gmail_msg">
<name>a0</name><br
class="gmail_msg">
<uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid><br
class="gmail_msg">
<metadata xmlns:ovirt-tune="<a
moz-do-not-send="true"
href="http://ovirt.org/vm/tune/1.0" rel="noreferrer"
class="gmail_msg"
target="_blank">http://ovirt.org/vm/tune/1.0</a>"...
class="gmail_msg">
</blockquote>
<div><br>
</div>
<div>The url is broken - do we have this?</div>
</div>
</div>
</div>
</blockquote>
<br>
I guess no. This is old code, dating back to 3.6,<i> which I'm
taking without changes.</i><br>
<br>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div> </div>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
xmlns:ovirt-vm="<a moz-do-not-send="true"
href="http://ovirt.org/vm/1.0" rel="noreferrer"
class="gmail_msg"
target="_blank">http://ovirt.org/vm/1.0</a>"<br
class="gmail_msg">
xmlns:ovirt-instance="<a moz-do-not-send="true"
href="http://ovirt.org/vm/instance/1.0"
rel="noreferrer"
class="gmail_msg"
target="_blank">http://ovirt.org/vm/instance/1.0</a>&q...
class="gmail_msg">
<ovirt-tune:qos/> </blockquote>
<div><br>
</div>
<div>What do we keep here? why does it need its own
namespace?</div>
<div> </div>
</div>
</div>
</div>
</blockquote>
<br>
yes, because of how we use the metadata:<br>
<br>
<a class="moz-txt-link-freetext"
href="http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainG...
<a class="moz-txt-link-freetext"
href="http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainS...
<br>
we use type =
<a class="moz-txt-link-freetext"
href="http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN...
<br>
-- I don't think we should change that<br>
<br>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-vm:vm/><br class="gmail_msg">
</blockquote>
<div><br>
</div>
<div>
<div>What do we keep here? why does it need its own
namespace?</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
same as above<br>
<br>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div><br class="inbox-inbox-Apple-interchange-newline">
</div>
<div>Can we merge all namespaces into one generic
namespace?</div>
</div>
</div>
</div>
</blockquote>
<br>
Probably, but we have the nice property that code can trivially
access one subset of metadata, e.g.<br>
only the qos subtree, only the container subtree.<br>
Furthermore if we don't change that we are backward compatible at
XML level (inbound migrations<br>
care about that).<br>
<br>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div> </div>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-instance:instance><br
class="gmail_msg">
</blockquote>
<div class="gmail_msg"><br class="gmail_msg">
</div>
</div>
</div>
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div class="gmail_msg">What is instance?</div>
</div>
</div>
</div>
</blockquote>
<br>
Gone, it duplicated vm without a good reason<br>
<br>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div class="gmail_msg"> </div>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-instance:value type="dict">
</blockquote>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-instance:item key="devices"
type="list"> </blockquote>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-instance:item index="0"
type="dict"></blockquote>
<div><br>
</div>
<div>Isn't index redundant?</div>
</div>
</div>
</div>
</blockquote>
<br>
It is, but now is gone<br>
<br>
<blockquote
cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div> </div>
<blockquote class="gmail_quote gmail_msg" style="margin:0 0
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<ovirt-instance:item key="device"<br
class="gmail_msg">
type="str">vnc</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="specParams"
type="dict"><br class="gmail_msg">
<ovirt-instance:item
key="displayNetwork"<br
class="gmail_msg">
type="str">ovirtmgmt</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="displayIp"<br
class="gmail_msg">
type="str">192.168.1.53</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="type"<br
class="gmail_msg">
type="str">graphics</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item index="1"
type="dict"><br class="gmail_msg">
<ovirt-instance:item key="device"<br
class="gmail_msg">
type="str">spice</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="specParams"
type="dict"><br class="gmail_msg">
<ovirt-instance:item
key="displayNetwork"<br
class="gmail_msg">
type="str">ovirtmgmt</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="displayIp"<br
class="gmail_msg">
type="str">192.168.1.53</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="type"<br
class="gmail_msg">
type="str">graphics</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item index="2"
type="dict"><br class="gmail_msg">
<ovirt-instance:item key="poolID"<br
class="gmail_msg">
type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="imageID"<br
class="gmail_msg">
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="domainID"<br
class="gmail_msg">
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="device"<br
class="gmail_msg">
type="str">disk</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="path"<br
class="gmail_msg">
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item><br
class="gmail_msg">
<ovirt-instance:item key="volumeID"<br
class="gmail_msg">
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:item><br
class="gmail_msg">
</ovirt-instance:value><br
class="gmail_msg">
</ovirt-instance:instance><br
class="gmail_msg">
</metadata><br class="gmail_msg">
<!-- omitted for brevity --><br
class="gmail_msg">
</domain><br class="gmail_msg">
</blockquote>
<div class="gmail_msg"><br class="gmail_msg">
</div>
</div>
</div>
<div dir="ltr" class="gmail_msg">
<div class="gmail_quote gmail_msg">
<div class="gmail_msg">How about a generic json
namespace?<br
class="gmail_msg">
</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">json:</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">{</div>
<div class="gmail_msg"> "devices": [</div>
<div class="gmail_msg"> {</div>
<div class="gmail_msg"> "foo":
"bar"</div>
<div class="gmail_msg"> }</div>
<div class="gmail_msg"> ]</div>
<div class="gmail_msg">}</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">xml:</div>
<div class="gmail_msg"><br>
</div>
<div
class="gmail_msg"><json:object></div>
<div class="gmail_msg">
<json:key>devices</json:key></div>
<div class="gmail_msg">
<json:list></div>
<div class="gmail_msg">
<json:object></div>
<div class="gmail_msg">
<json:key>foo</json:key></div>
<div class="gmail_msg">
<div class="gmail_msg">
<json:string>bar</json:string></div>
</div>
<div class="gmail_msg">
</json:object></div>
<div class="gmail_msg"> </json:list><br
class="gmail_msg">
</div>
<div
class="gmail_msg"></json:object></div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg">We can query and modify this xml
using xpath, like this:</div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">>>>
r.findall("./metadata/json:object/[json:key='devices']/json:list/json:object/",
namespaces={"json": "<a
moz-do-not-send="true"
href="http://ovirt.org/json/1.0">http://ovirt.org/json/1.0&l...
<div class="gmail_msg">[<Element '{<a
moz-do-not-send="true"
href="http://ovirt.org/json/1.0%7Dkey">http://ovirt.org/json...
at 0x7fd3386a69d0>, <Element '{<a
moz-do-not-send="true"
href="http://ovirt.org/json/1.0%7Dstring">http://ovirt.org/j...
at 0x7fd3386a6a50>]</div>
<div><br>
</div>
</div>
</div>
<div>Not sure this format will be easy to modify, needs
to tinker more</div>
<div>with this.</div>
<div><br>
</div>
</div>
</div>
</div>
<div class="gmail_quote gmail_msg">
<div class="gmail_msg">It will be probably easier to parse
the entire metadata, change it,</div>
<div class="gmail_msg">and serialize it back.</div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg">Maybe using key and type attribute as
you suggest makes it simpler to use,</div>
<div class="gmail_msg">and we since we convert from xml to
python or python to xml, we can</div>
<div class="gmail_msg">have a "py"
namespace.</div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg">
<div class="gmail_msg"><py:item></div>
<div class="gmail_msg"> <py:item
key="devices"
type="list"></div>
<div class="gmail_msg"> <py:item
type="dict"></div>
<div class="gmail_msg"> <py:item
key="foo"
type="str">bar</py:item></div>
<div class="gmail_msg">
</py:item><br>
</py:item></div>
<div class="gmail_msg"></py:item></div>
</div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg">With this we can have:</div>
<div class="gmail_msg"><br>
</div>
<div class="gmail_msg">
<div class="gmail_msg">>>>
r.findall("./metadata/py:item/py:item[@key='devices']/py:item[1]/py:item[@key='foo']",
namespaces={"py": "<a moz-do-not-send="true"
href="http://ovirt.org/py/1.0">http://ovirt.org/py/1.0</a...
<div class="gmail_msg">'bar'</div>
<div><br>
</div>
</div>
<div class="gmail_msg">Something like this can be useful for
others as well.</div>
</div>
</div>
</div>
</blockquote>
<br>
Great suggestions, I like the py namespace, but concerns were raised
about the solution here being too generic, so I limited it, see<br>
<a class="moz-txt-link-freetext"
href="https://gerrit.ovirt.org/#/c/74206/11">https://gerrit....
<br>
but I will add my pyxmlpickle module (<a moz-do-not-send="true"
href="https://github.com/fromanirh/pyxmlpickle"
rel="noreferrer"
class="gmail_msg"
target="_blank">https://github.com/fromanirh/pyxmlpickle</a>)
anyway in the future according to your comments.<br>
<br>
<br>
<pre class="moz-signature" cols="72">--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani</pre>
</body>
</html>
--------------15F6CAD0BC7802BD4F39C8F8--
8:26 p.m.
On Wed, Mar 15, 2017 at 2:28 PM Francesco Romani <fromani(a)redhat.com> wrote:
Hi everyone,
This is both a report of the current state of my Vdsm patches for Engine
XML support, and a proposal to how move forward and solve
the current open issues.
TL;DR:
1. we can and IMO should reuse the current JSON schema to describe the
structure (layout) and the types of the metadata section.
2. we don't need a priori validation of stuff in the metadata section.
We will just raise in the creation flow if data is missing, or wrong,
according to our schema.
2. we will add *few* items to the metadata section, only thing we can't
express clearly-or at all in the libvirt XML. Redundancy and verbosiness
will be thus kept at bay
3. I believe [3] is the best tool to do (de)serialize data to the
metadata section. Existing tools fits poorly in our very specific use case
Examples below
+++
Long(er) discussion:
I have working code[1][2] to encode any custom, picklable, python
object in the metadata section.
We should decide which module will do the actual python<=>XML
transformation.
Please note that this actually also influences how the data in the
medata section look like, so the two things are a bit coupled.
I'm eager to reinvent another wheel, but after
initial evaluation I honestly think that my pyxmlpickle[3] is the best
tool for the job over the current alternatives: plistlib[4] and
xmltodict[5].
I added the initial rationale here:
https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
I have completed the initial draft of patches to make it possible to
initialize devices from their XML representation [6]. This is the bare
minimum we need to support the Engine XML, and we *need* this anyway to
unlock the cleanup we planned and I outlined in my google doc.
So we are progressing, but I'd like to speed up things. Those [6]
patches are not yet complete, many flows are not covered or tested; but
they are good enough to demonstrate that there *are* pieces of
information wen need to properly initialize the devices, but we can't
easily extract from the XML.
First examples that come to my mind are the storage.Drive UUIDs; there
could also be some ambiguity I'm investigating right now for
displayIp/displayNetwork in Graphics devices. In [6] there are various
TODO to mark more of those cases. Most likely, few more cases will pop
out as I cover all the flows we support.
Long story short: it is hard to correctly rebuild the device conf from
the XML. This is why in [6] I added the 'meta' argument to from_xml_tree
classmethod in [7].
'meta' is supposed to be the device metadata: extra data related to a
device which doesn't (yet) fit in the libvirt XML representation.
For example, we can store 'displayIp' and 'displayNetwork' here and be
done with that: using both per-device metadata and the XML
representation of one graphic device, we will have everything we need to
properly build one graphics.Graphics device.
This example may (hopefully) be bogus, but I'm keeping it because it is
one case easy to follow.
The device metadata is going to be stored in the vm metadata for the
short/mid term future. Even if the per-device metadata idea/RFE is
accepted (no answer yet, but we are working on it), we will not have in
7.4, and unlikely in 7.5.
As it stands today, I believe there are two open questions:
1. do we need a schema for the metadata section?
2. how do we bind the metadata to the devices? How do we know which
metadata belongs to which metadata, if we don't have aliases nor
addresses to match? (e.g. very first time the VM is created!)
My current stance is the following
1. In general, one schema gives us two benefits: 1.a. we document how
the layout of the data should be, including types; 1.b. we can validate
the data we receive.
So yes, we need a schema, but we don't need a *new* schema. I think we
are in good enough shape with the current Vdsm schema: we can just
translate the python object layout to a XML layout.
One example is probably more explicative. Some actual data may look
like, using my pyxmlpickle module:
<domain type='kvm' id='5'>
<name>a0</name>
<uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid>
<metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0"
xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0">
<ovirt-tune:qos/>
<ovirt-vm:vm/>
<ovirt-instance:instance>
<ovirt-instance:value type="dict">
<ovirt-instance:item key="devices" type="list">
<ovirt-instance:item index="0" type="dict">
<ovirt-instance:item key="device"
type="str">vnc</ovirt-instance:item>
<ovirt-instance:item key="specParams" type="dict">
<ovirt-instance:item key="displayNetwork"
type="str">ovirtmgmt</ovirt-instance:item>
<ovirt-instance:item key="displayIp"
type="str">192.168.1.53</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">graphics</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item index="1" type="dict">
<ovirt-instance:item key="device"
type="str">spice</ovirt-instance:item>
<ovirt-instance:item key="specParams" type="dict">
<ovirt-instance:item key="displayNetwork"
type="str">ovirtmgmt</ovirt-instance:item>
<ovirt-instance:item key="displayIp"
type="str">192.168.1.53</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item key="type"
type="str">graphics</ovirt-instance:item>
</ovirt-instance:item>
<ovirt-instance:item index="2" type="dict">
<ovirt-instance:item key="poolID"
type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item>
<ovirt-instance:item key="imageID"
type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item>
<ovirt-instance:item key="domainID"
type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item>
<ovirt-instance:item key="device"
type="str">disk</ovirt-instance:item>
<ovirt-instance:item key="path"
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
<ovirt-instance:item key="volumeID"
type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:item>
</ovirt-instance:value>
</ovirt-instance:instance>
</metadata>
<!-- omitted for brevity -->
</domain>
Please note that yes, this is still verbose, but we don't want to add
much data here, for most of information the most reliable source will
be the domain XML. We will add here only the extra info we can't really
fetch from that.
2. I don't think we need explicit validation: we could just raise along
the way in the creation flow if we don't find some extra metadata we
need. This will also solve the issue that if we reuse the current schema
and we omit most of data, we will lack quite a lot of elements
marked mandatory.
Once we reached agreement, I will update my
https://docs.google.com/document/d/1eD8KSLwwyo2Sk64MytbmE0wBxxMlpIyEI1GRc...
accordingly.
Final note: while device take the lion's share, we will likely need help
from the metadata section also to store VM extra info, but all the above
discussion also applies here.
+++
[1]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
- uses xmltodict
[2]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
ported the 'virt-metadata3' topic to pyxmlpickle
[3] https://github.com/fromanirh/pyxmlpickle
Looks good, I like the simple loads() and dumps().
Issues:
- index attribute seems unneeded
- not sure why we need the value element, seems that everything can be an
item
- not sure special root element is needed, why not parse the contents of
any element, and raise ValueError if there is not type info?
- strange name - here some alternative names:
- xmlon (xml object notation)
- pxon (python xml object notation)
If there is no other library that we can use, this seems to be
the best direction.
Bug I think plist xml format is much nicer - it should be easy to
support this format instead of the type attributes:
pyxmlpickle:
<pyxmlpickle>
<value type="dict">
<item key="foo" type="str">bar</item>
<item key="list" type="list">
<item index="0" type="int">42</item>
<item index="1" type="float">3.14</item>
</item>
</value>
</pyxmlpicle>
plist:
<plist>
<dict>
<key>foo</key>
<string>bar</string>
<key>list</key>
<array>
<integer>42</integer>
<real>3.14</real>
</array>
</dict>
</plist>
Less noise, more readable, easier to parse?
Since we cannot use plistlib as is, we can make
the format nicer and more pythonic:
<py>
<dict>
<key>foo</key>
<str>bar</str>
<key>list</key>
<list>
<int>42</int>
<float>3.14</float>
</list>
</dict>
</py>
Adding namespace will ruin this but if this required by libvirt
we have no choice.
We cannot use it as is, since it does not support reading and writing
elements, only complete document. We need integration with etree
- convert dict to etree element and etree element to dict. Finally, it does
not support None.
We can borrow code from this module, it is well tested and the author
is the same author of simplejson and other nice stuff.
xmltodict converts everything to string, and does not parse the
type from the xml, this is very far from the json module.
I found also dicttoxml, which does keep the types, but does
not support parsing xml.
I don't see any value in including this dependency and hacking
around it to make it do what we want.
There is also:
https://pypi.python.org/pypi/xmljson
Using the parker module seems nice:
>> from xml.etree.ElementTree import tostring, fromstring
>> d = {'devices': [{'int': 42, 'none': None,
'float': 3.14, 'string':
'value'}]}
>> from xmljson import parker
>> root = parker.etree(d)[0]
>> tostring(root)
'<devices><int>42</int><none>None</none><float>3.14</float><string>value</string></devices>'
>> parker.data(root)
OrderedDict([('int', 42),
('none', 'None'), ('float', 3.14), ('string',
'value')])
But the list was converted to dict :-)
Seems that this project focus on converting any xml to json
while we care about converting json to xml and back, we
don't care about any xml.
The integration with etree is nice, we should have this.
[6]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic...
[7] https://gerrit.ovirt.org/#/c/72880/15/lib/vdsm/virt/vmdevices/core.py
--
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
2583
days inactive
2591
days old
11 comments
4 participants
participants (4)
-
Francesco Romani -
Martin Polednik -
Michal Skrivanek -
Nir Soffer