[vdsm] Engine XML: metadata and devices from XML
Hi everyone, This is both a report of the current state of my Vdsm patches for Engine XML support, and a proposal to how move forward and solve the current open issues. TL;DR: 1. we can and IMO should reuse the current JSON schema to describe the structure (layout) and the types of the metadata section. 2. we don't need a priori validation of stuff in the metadata section. We will just raise in the creation flow if data is missing, or wrong, according to our schema. 2. we will add *few* items to the metadata section, only thing we can't express clearly-or at all in the libvirt XML. Redundancy and verbosiness will be thus kept at bay 3. I believe [3] is the best tool to do (de)serialize data to the metadata section. Existing tools fits poorly in our very specific use case Examples below +++ Long(er) discussion: I have working code[1][2] to encode any custom, picklable, python object in the metadata section. We should decide which module will do the actual python<=>XML transformation. Please note that this actually also influences how the data in the medata section look like, so the two things are a bit coupled. I'm eager to reinvent another wheel, but after initial evaluation I honestly think that my pyxmlpickle[3] is the best tool for the job over the current alternatives: plistlib[4] and xmltodict[5]. I added the initial rationale here: https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG I have completed the initial draft of patches to make it possible to initialize devices from their XML representation [6]. This is the bare minimum we need to support the Engine XML, and we *need* this anyway to unlock the cleanup we planned and I outlined in my google doc. So we are progressing, but I'd like to speed up things. Those [6] patches are not yet complete, many flows are not covered or tested; but they are good enough to demonstrate that there *are* pieces of information wen need to properly initialize the devices, but we can't easily extract from the XML. First examples that come to my mind are the storage.Drive UUIDs; there could also be some ambiguity I'm investigating right now for displayIp/displayNetwork in Graphics devices. In [6] there are various TODO to mark more of those cases. Most likely, few more cases will pop out as I cover all the flows we support. Long story short: it is hard to correctly rebuild the device conf from the XML. This is why in [6] I added the 'meta' argument to from_xml_tree classmethod in [7]. 'meta' is supposed to be the device metadata: extra data related to a device which doesn't (yet) fit in the libvirt XML representation. For example, we can store 'displayIp' and 'displayNetwork' here and be done with that: using both per-device metadata and the XML representation of one graphic device, we will have everything we need to properly build one graphics.Graphics device. This example may (hopefully) be bogus, but I'm keeping it because it is one case easy to follow. The device metadata is going to be stored in the vm metadata for the short/mid term future. Even if the per-device metadata idea/RFE is accepted (no answer yet, but we are working on it), we will not have in 7.4, and unlikely in 7.5. As it stands today, I believe there are two open questions: 1. do we need a schema for the metadata section? 2. how do we bind the metadata to the devices? How do we know which metadata belongs to which metadata, if we don't have aliases nor addresses to match? (e.g. very first time the VM is created!) My current stance is the following 1. In general, one schema gives us two benefits: 1.a. we document how the layout of the data should be, including types; 1.b. we can validate the data we receive. So yes, we need a schema, but we don't need a *new* schema. I think we are in good enough shape with the current Vdsm schema: we can just translate the python object layout to a XML layout. One example is probably more explicative. Some actual data may look like, using my pyxmlpickle module: <domain type='kvm' id='5'> <name>a0</name> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0" xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0"> <ovirt-tune:qos/> <ovirt-vm:vm/> <ovirt-instance:instance> <ovirt-instance:value type="dict"> <ovirt-instance:item key="devices" type="list"> <ovirt-instance:item index="0" type="dict"> <ovirt-instance:item key="device" type="str">vnc</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="1" type="dict"> <ovirt-instance:item key="device" type="str">spice</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="2" type="dict"> <ovirt-instance:item key="poolID" type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="device" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:value> </ovirt-instance:instance> </metadata> <!-- omitted for brevity --> </domain> Please note that yes, this is still verbose, but we don't want to add much data here, for most of information the most reliable source will be the domain XML. We will add here only the extra info we can't really fetch from that. 2. I don't think we need explicit validation: we could just raise along the way in the creation flow if we don't find some extra metadata we need. This will also solve the issue that if we reuse the current schema and we omit most of data, we will lack quite a lot of elements marked mandatory. Once we reached agreement, I will update my https://docs.google.com/document/d/1eD8KSLwwyo2Sk64MytbmE0wBxxMlpIyEI1GRcHDk... accordingly. Final note: while device take the lion's share, we will likely need help from the metadata section also to store VM extra info, but all the above discussion also applies here. +++ [1] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... - uses xmltodict [2] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... ported the 'virt-metadata3' topic to pyxmlpickle [3] https://github.com/fromanirh/pyxmlpickle [4] https://docs.python.org/2/library/plistlib.html [5] https://github.com/martinblech/xmltodict [6] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vm... [7] https://gerrit.ovirt.org/#/c/72880/15/lib/vdsm/virt/vmdevices/core.py -- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani
We talked about sending storage device purely on metadata, letting Vdsm rebuild them and getting the XML like today. In the other direction, Vdsm will pass through the XML (perhaps only parts of it, e.g. the devices subtree) like before. This way we can minimize the changes we are uncertain of, and more importantly, we can minimize the risky changes. The following is a realistic example of how the XML could look like if we send all but the storage devices. It is built using my pyxmlpickle module (see [3] below). <domain type='kvm' id='10'> <name>a0</name> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0" xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0"> <ovirt-tune:qos/> <ovirt-vm:vm/> <ovirt-instance:instance> <ovirt-instance:value type="dict"> <ovirt-instance:item key="devices" type="list"> <ovirt-instance:item index="0" type="dict"> <ovirt-instance:item key="index" type="str">2</ovirt-instance:item> <ovirt-instance:item key="iface" type="str">ide</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="path" type="str"/> </ovirt-instance:item> <ovirt-instance:item key="readonly" type="str">true</ovirt-instance:item> <ovirt-instance:item key="deviceId" type="str">e59c985c-46c2-4489-b355-a6f374125eb9</ovirt-instance:item> <ovirt-instance:item key="address" type="dict"> <ovirt-instance:item key="bus" type="str">1</ovirt-instance:item> <ovirt-instance:item key="controller" type="str">0</ovirt-instance:item> <ovirt-instance:item key="type" type="str">drive</ovirt-instance:item> <ovirt-instance:item key="target" type="str">0</ovirt-instance:item> <ovirt-instance:item key="unit" type="str">0</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="device" type="str">cdrom</ovirt-instance:item> <ovirt-instance:item key="shared" type="str">false</ovirt-instance:item> <ovirt-instance:item key="path" type="str"/> <ovirt-instance:item key="type" type="str">disk</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="1" type="dict"> <ovirt-instance:item key="poolID" type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item> <ovirt-instance:item key="volumeInfo" type="dict"> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="volType" type="str">path</ovirt-instance:item> <ovirt-instance:item key="leaseOffset" type="int">0</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="leasePath" type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="index" type="int">0</ovirt-instance:item> <ovirt-instance:item key="iface" type="str">virtio</ovirt-instance:item> <ovirt-instance:item key="apparentsize" type="str">8589934592</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="readonly" type="str">false</ovirt-instance:item> <ovirt-instance:item key="shared" type="str">false</ovirt-instance:item> <ovirt-instance:item key="truesize" type="str">0</ovirt-instance:item> <ovirt-instance:item key="type" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="reqsize" type="str">0</ovirt-instance:item> <ovirt-instance:item key="format" type="str">raw</ovirt-instance:item> <ovirt-instance:item key="deviceId" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="address" type="dict"> <ovirt-instance:item key="function" type="str">0x0</ovirt-instance:item> <ovirt-instance:item key="bus" type="str">0x00</ovirt-instance:item> <ovirt-instance:item key="domain" type="str">0x0000</ovirt-instance:item> <ovirt-instance:item key="type" type="str">pci</ovirt-instance:item> <ovirt-instance:item key="slot" type="str">0x05</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="device" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="propagateErrors" type="str">off</ovirt-instance:item> <ovirt-instance:item key="optional" type="str">false</ovirt-instance:item> <ovirt-instance:item key="bootOrder" type="str">1</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"/> <ovirt-instance:item key="volumeChain" type="list"> <ovirt-instance:item index="0" type="dict"> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="volType" type="str">path</ovirt-instance:item> <ovirt-instance:item key="leaseOffset" type="int">0</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="leasePath" type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:value> </ovirt-instance:instance> </metadata> <maxMemory slots='16' unit='KiB'>4294967296</maxMemory> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static' current='2'>16</vcpu> <cputune> <shares>1020</shares> </cputune> <resource> <partition>/machine</partition> </resource> <sysinfo type='smbios'> <system> <entry name='manufacturer'>oVirt</entry> <entry name='product'>oVirt Node</entry> <entry name='version'>7-3.1611.el7.centos</entry> <entry name='serial'>ccd79775-c888-4789-975a-fde1143dffc9</entry> <entry name='uuid'>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</entry> </system> </sysinfo> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type> <bios useserial='yes'/> <smbios mode='sysinfo'/> </os> <features> <acpi/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell-noTSX</model> <topology sockets='16' cores='1' threads='1'/> <numa> <cell id='0' cpus='0-1' memory='4194304' unit='KiB'/> </numa> </cpu> <clock offset='variable' adjustment='0' basis='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <controller type='virtio-serial' index='0' ports='16'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='usb' index='0'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='bridge'> <mac address='00:1a:4a:16:01:51'/> <source bridge='ovirtmgmt'/> <target dev='vnet0'/> <model type='virtio'/> <filterref filter='vdsm-no-mac-spoofing'/> <link state='up'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='unix'> <source mode='bind' path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='unix'> <source mode='bind' path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channels/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.com.redhat.rhevm.vdsm'/> <target type='virtio' name='com.redhat.rhevm.vdsm' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channels/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0' state='disconnected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <graphics type='vnc' port='5900' autoport='yes' listen='192.168.1.53' keymap='en-us' passwdValidTo='1970-01-01T00:00:01'> <listen type='network' address='192.168.1.53' network='vdsm-ovirtmgmt'/> </graphics> <graphics type='spice' tlsPort='5901' autoport='yes' listen='192.168.1.53' keymap='en-us' defaultMode='secure' passwdValidTo='1970-01-01T00:00:01'> <listen type='network' address='192.168.1.53' network='vdsm-ovirtmgmt'/> <channel name='main' mode='secure'/> <channel name='display' mode='secure'/> <channel name='inputs' mode='secure'/> <channel name='cursor' mode='secure'/> <channel name='playback' mode='secure'/> <channel name='record' mode='secure'/> <channel name='smartcard' mode='secure'/> <channel name='usbredir' mode='secure'/> </graphics> <sound model='ich6'> <alias name='sound0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='8192' vgamem='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c207,c629</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c207,c629</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+107:+107</label> <imagelabel>+107:+107</imagelabel> </seclabel> </domain> On 03/15/2017 01:28 PM, Francesco Romani wrote:
Hi everyone,
This is both a report of the current state of my Vdsm patches for Engine XML support, and a proposal to how move forward and solve the current open issues.
TL;DR: 1. we can and IMO should reuse the current JSON schema to describe the structure (layout) and the types of the metadata section. 2. we don't need a priori validation of stuff in the metadata section. We will just raise in the creation flow if data is missing, or wrong, according to our schema. 2. we will add *few* items to the metadata section, only thing we can't express clearly-or at all in the libvirt XML. Redundancy and verbosiness will be thus kept at bay 3. I believe [3] is the best tool to do (de)serialize data to the metadata section. Existing tools fits poorly in our very specific use case
Examples below
+++
Long(er) discussion:
I have working code[1][2] to encode any custom, picklable, python object in the metadata section.
We should decide which module will do the actual python<=>XML transformation. Please note that this actually also influences how the data in the medata section look like, so the two things are a bit coupled.
I'm eager to reinvent another wheel, but after initial evaluation I honestly think that my pyxmlpickle[3] is the best tool for the job over the current alternatives: plistlib[4] and xmltodict[5].
I added the initial rationale here: https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
I have completed the initial draft of patches to make it possible to initialize devices from their XML representation [6]. This is the bare minimum we need to support the Engine XML, and we *need* this anyway to unlock the cleanup we planned and I outlined in my google doc.
So we are progressing, but I'd like to speed up things. Those [6] patches are not yet complete, many flows are not covered or tested; but they are good enough to demonstrate that there *are* pieces of information wen need to properly initialize the devices, but we can't easily extract from the XML.
First examples that come to my mind are the storage.Drive UUIDs; there could also be some ambiguity I'm investigating right now for displayIp/displayNetwork in Graphics devices. In [6] there are various TODO to mark more of those cases. Most likely, few more cases will pop out as I cover all the flows we support.
Long story short: it is hard to correctly rebuild the device conf from the XML. This is why in [6] I added the 'meta' argument to from_xml_tree classmethod in [7].
'meta' is supposed to be the device metadata: extra data related to a device which doesn't (yet) fit in the libvirt XML representation. For example, we can store 'displayIp' and 'displayNetwork' here and be done with that: using both per-device metadata and the XML representation of one graphic device, we will have everything we need to properly build one graphics.Graphics device. This example may (hopefully) be bogus, but I'm keeping it because it is one case easy to follow.
The device metadata is going to be stored in the vm metadata for the short/mid term future. Even if the per-device metadata idea/RFE is accepted (no answer yet, but we are working on it), we will not have in 7.4, and unlikely in 7.5.
As it stands today, I believe there are two open questions:
1. do we need a schema for the metadata section? 2. how do we bind the metadata to the devices? How do we know which metadata belongs to which metadata, if we don't have aliases nor addresses to match? (e.g. very first time the VM is created!)
My current stance is the following 1. In general, one schema gives us two benefits: 1.a. we document how the layout of the data should be, including types; 1.b. we can validate the data we receive. So yes, we need a schema, but we don't need a *new* schema. I think we are in good enough shape with the current Vdsm schema: we can just translate the python object layout to a XML layout.
One example is probably more explicative. Some actual data may look like, using my pyxmlpickle module:
<domain type='kvm' id='5'> <name>a0</name> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0" xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0"> <ovirt-tune:qos/> <ovirt-vm:vm/> <ovirt-instance:instance> <ovirt-instance:value type="dict"> <ovirt-instance:item key="devices" type="list"> <ovirt-instance:item index="0" type="dict"> <ovirt-instance:item key="device" type="str">vnc</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="1" type="dict"> <ovirt-instance:item key="device" type="str">spice</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="2" type="dict"> <ovirt-instance:item key="poolID" type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="device" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:value> </ovirt-instance:instance> </metadata> <!-- omitted for brevity --> </domain>
Please note that yes, this is still verbose, but we don't want to add much data here, for most of information the most reliable source will be the domain XML. We will add here only the extra info we can't really fetch from that.
2. I don't think we need explicit validation: we could just raise along the way in the creation flow if we don't find some extra metadata we need. This will also solve the issue that if we reuse the current schema and we omit most of data, we will lack quite a lot of elements marked mandatory.
Once we reached agreement, I will update my https://docs.google.com/document/d/1eD8KSLwwyo2Sk64MytbmE0wBxxMlpIyEI1GRcHDk... accordingly.
Final note: while device take the lion's share, we will likely need help from the metadata section also to store VM extra info, but all the above discussion also applies here.
+++
[1] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... - uses xmltodict [2] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... ported the 'virt-metadata3' topic to pyxmlpickle [3] https://github.com/fromanirh/pyxmlpickle [4] https://docs.python.org/2/library/plistlib.html [5] https://github.com/martinblech/xmltodict [6] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vm... [7] https://gerrit.ovirt.org/#/c/72880/15/lib/vdsm/virt/vmdevices/core.py
-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani
On 16 Mar 2017, at 09:45, Francesco Romani <fromani@redhat.com> wrote:
We talked about sending storage device purely on metadata, letting Vdsm rebuild them and getting the XML like today.
In the other direction, Vdsm will pass through the XML (perhaps only parts of it, e.g. the devices subtree) like before.
This way we can minimize the changes we are uncertain of, and more importantly, we can minimize the risky changes.
The following is a realistic example of how the XML could look like if we send all but the storage devices. It is built using my pyxmlpickle module (see [3] below).
That’s quite verbose. How much work would it need to actually minimize it and turn it into something more simple. Most such stuff should go away and I believe it would be beneficial to make it difficult to use to discourage using metadata as a generic junkyard
<domain type='kvm' id='10'> <name>a0</name> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0" xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0"> <ovirt-tune:qos/> <ovirt-vm:vm/> <ovirt-instance:instance> <ovirt-instance:value type="dict"> <ovirt-instance:item key="devices" type="list"> <ovirt-instance:item index="0" type="dict"> <ovirt-instance:item key="index" type="str">2</ovirt-instance:item> <ovirt-instance:item key="iface" type="str">ide</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="path" type="str"/> </ovirt-instance:item> <ovirt-instance:item key="readonly" type="str">true</ovirt-instance:item> <ovirt-instance:item key="deviceId" type="str">e59c985c-46c2-4489-b355-a6f374125eb9</ovirt-instance:item> <ovirt-instance:item key="address" type="dict"> <ovirt-instance:item key="bus" type="str">1</ovirt-instance:item> <ovirt-instance:item key="controller" type="str">0</ovirt-instance:item> <ovirt-instance:item key="type" type="str">drive</ovirt-instance:item> <ovirt-instance:item key="target" type="str">0</ovirt-instance:item> <ovirt-instance:item key="unit" type="str">0</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="device" type="str">cdrom</ovirt-instance:item> <ovirt-instance:item key="shared" type="str">false</ovirt-instance:item> <ovirt-instance:item key="path" type="str"/> <ovirt-instance:item key="type" type="str">disk</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="1" type="dict"> <ovirt-instance:item key="poolID" type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item> <ovirt-instance:item key="volumeInfo" type="dict"> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="volType" type="str">path</ovirt-instance:item> <ovirt-instance:item key="leaseOffset" type="int">0</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="leasePath" type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="index" type="int">0</ovirt-instance:item> <ovirt-instance:item key="iface" type="str">virtio</ovirt-instance:item> <ovirt-instance:item key="apparentsize" type="str">8589934592</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="readonly" type="str">false</ovirt-instance:item> <ovirt-instance:item key="shared" type="str">false</ovirt-instance:item> <ovirt-instance:item key="truesize" type="str">0</ovirt-instance:item> <ovirt-instance:item key="type" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="reqsize" type="str">0</ovirt-instance:item> <ovirt-instance:item key="format" type="str">raw</ovirt-instance:item> <ovirt-instance:item key="deviceId" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="address" type="dict"> <ovirt-instance:item key="function" type="str">0x0</ovirt-instance:item> <ovirt-instance:item key="bus" type="str">0x00</ovirt-instance:item> <ovirt-instance:item key="domain" type="str">0x0000</ovirt-instance:item> <ovirt-instance:item key="type" type="str">pci</ovirt-instance:item> <ovirt-instance:item key="slot" type="str">0x05</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="device" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="propagateErrors" type="str">off</ovirt-instance:item> <ovirt-instance:item key="optional" type="str">false</ovirt-instance:item> <ovirt-instance:item key="bootOrder" type="str">1</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"/> <ovirt-instance:item key="volumeChain" type="list"> <ovirt-instance:item index="0" type="dict"> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="volType" type="str">path</ovirt-instance:item> <ovirt-instance:item key="leaseOffset" type="int">0</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="leasePath" type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:value> </ovirt-instance:instance> </metadata> <maxMemory slots='16' unit='KiB'>4294967296</maxMemory> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static' current='2'>16</vcpu> <cputune> <shares>1020</shares> </cputune> <resource> <partition>/machine</partition> </resource> <sysinfo type='smbios'> <system> <entry name='manufacturer'>oVirt</entry> <entry name='product'>oVirt Node</entry> <entry name='version'>7-3.1611.el7.centos</entry> <entry name='serial'>ccd79775-c888-4789-975a-fde1143dffc9</entry> <entry name='uuid'>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</entry> </system> </sysinfo> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type> <bios useserial='yes'/> <smbios mode='sysinfo'/> </os> <features> <acpi/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell-noTSX</model> <topology sockets='16' cores='1' threads='1'/> <numa> <cell id='0' cpus='0-1' memory='4194304' unit='KiB'/> </numa> </cpu> <clock offset='variable' adjustment='0' basis='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <controller type='virtio-serial' index='0' ports='16'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='usb' index='0'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='bridge'> <mac address='00:1a:4a:16:01:51'/> <source bridge='ovirtmgmt'/> <target dev='vnet0'/> <model type='virtio'/> <filterref filter='vdsm-no-mac-spoofing'/> <link state='up'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='unix'> <source mode='bind' path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='unix'> <source mode='bind' path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channels/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.com.redhat.rhevm.vdsm'/> <target type='virtio' name='com.redhat.rhevm.vdsm' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channels/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0' state='disconnected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <graphics type='vnc' port='5900' autoport='yes' listen='192.168.1.53' keymap='en-us' passwdValidTo='1970-01-01T00:00:01'> <listen type='network' address='192.168.1.53' network='vdsm-ovirtmgmt'/> </graphics> <graphics type='spice' tlsPort='5901' autoport='yes' listen='192.168.1.53' keymap='en-us' defaultMode='secure' passwdValidTo='1970-01-01T00:00:01'> <listen type='network' address='192.168.1.53' network='vdsm-ovirtmgmt'/> <channel name='main' mode='secure'/> <channel name='display' mode='secure'/> <channel name='inputs' mode='secure'/> <channel name='cursor' mode='secure'/> <channel name='playback' mode='secure'/> <channel name='record' mode='secure'/> <channel name='smartcard' mode='secure'/> <channel name='usbredir' mode='secure'/> </graphics> <sound model='ich6'> <alias name='sound0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='8192' vgamem='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c207,c629</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c207,c629</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+107:+107</label> <imagelabel>+107:+107</imagelabel> </seclabel> </domain>
On 03/15/2017 01:28 PM, Francesco Romani wrote:
Hi everyone,
This is both a report of the current state of my Vdsm patches for Engine XML support, and a proposal to how move forward and solve the current open issues.
TL;DR: 1. we can and IMO should reuse the current JSON schema to describe the structure (layout) and the types of the metadata section. 2. we don't need a priori validation of stuff in the metadata section. We will just raise in the creation flow if data is missing, or wrong, according to our schema. 2. we will add *few* items to the metadata section, only thing we can't express clearly-or at all in the libvirt XML. Redundancy and verbosiness will be thus kept at bay 3. I believe [3] is the best tool to do (de)serialize data to the metadata section. Existing tools fits poorly in our very specific use case
Examples below
+++
Long(er) discussion:
I have working code[1][2] to encode any custom, picklable, python object in the metadata section.
We should decide which module will do the actual python<=>XML transformation. Please note that this actually also influences how the data in the medata section look like, so the two things are a bit coupled.
I'm eager to reinvent another wheel, but after initial evaluation I honestly think that my pyxmlpickle[3] is the best tool for the job over the current alternatives: plistlib[4] and xmltodict[5].
I added the initial rationale here: https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
I have completed the initial draft of patches to make it possible to initialize devices from their XML representation [6]. This is the bare minimum we need to support the Engine XML, and we *need* this anyway to unlock the cleanup we planned and I outlined in my google doc.
So we are progressing, but I'd like to speed up things. Those [6] patches are not yet complete, many flows are not covered or tested; but they are good enough to demonstrate that there *are* pieces of information wen need to properly initialize the devices, but we can't easily extract from the XML.
First examples that come to my mind are the storage.Drive UUIDs; there could also be some ambiguity I'm investigating right now for displayIp/displayNetwork in Graphics devices. In [6] there are various TODO to mark more of those cases. Most likely, few more cases will pop out as I cover all the flows we support.
Long story short: it is hard to correctly rebuild the device conf from the XML. This is why in [6] I added the 'meta' argument to from_xml_tree classmethod in [7].
'meta' is supposed to be the device metadata: extra data related to a device which doesn't (yet) fit in the libvirt XML representation. For example, we can store 'displayIp' and 'displayNetwork' here and be done with that: using both per-device metadata and the XML representation of one graphic device, we will have everything we need to properly build one graphics.Graphics device. This example may (hopefully) be bogus, but I'm keeping it because it is one case easy to follow.
The device metadata is going to be stored in the vm metadata for the short/mid term future. Even if the per-device metadata idea/RFE is accepted (no answer yet, but we are working on it), we will not have in 7.4, and unlikely in 7.5.
As it stands today, I believe there are two open questions:
1. do we need a schema for the metadata section? 2. how do we bind the metadata to the devices? How do we know which metadata belongs to which metadata, if we don't have aliases nor addresses to match? (e.g. very first time the VM is created!)
My current stance is the following 1. In general, one schema gives us two benefits: 1.a. we document how the layout of the data should be, including types; 1.b. we can validate the data we receive. So yes, we need a schema, but we don't need a *new* schema. I think we are in good enough shape with the current Vdsm schema: we can just translate the python object layout to a XML layout.
One example is probably more explicative. Some actual data may look like, using my pyxmlpickle module:
<domain type='kvm' id='5'> <name>a0</name> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0" xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0"> <ovirt-tune:qos/> <ovirt-vm:vm/> <ovirt-instance:instance> <ovirt-instance:value type="dict"> <ovirt-instance:item key="devices" type="list"> <ovirt-instance:item index="0" type="dict"> <ovirt-instance:item key="device" type="str">vnc</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="1" type="dict"> <ovirt-instance:item key="device" type="str">spice</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="2" type="dict"> <ovirt-instance:item key="poolID" type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="device" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:value> </ovirt-instance:instance> </metadata> <!-- omitted for brevity --> </domain>
Please note that yes, this is still verbose, but we don't want to add much data here, for most of information the most reliable source will be the domain XML. We will add here only the extra info we can't really fetch from that.
2. I don't think we need explicit validation: we could just raise along the way in the creation flow if we don't find some extra metadata we need. This will also solve the issue that if we reuse the current schema and we omit most of data, we will lack quite a lot of elements marked mandatory.
Once we reached agreement, I will update my https://docs.google.com/document/d/1eD8KSLwwyo2Sk64MytbmE0wBxxMlpIyEI1GRcHDk... accordingly.
Final note: while device take the lion's share, we will likely need help from the metadata section also to store VM extra info, but all the above discussion also applies here.
+++
[1] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... - uses xmltodict [2] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... ported the 'virt-metadata3' topic to pyxmlpickle [3] https://github.com/fromanirh/pyxmlpickle [4] https://docs.python.org/2/library/plistlib.html [5] https://github.com/martinblech/xmltodict [6] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vm... [7] https://gerrit.ovirt.org/#/c/72880/15/lib/vdsm/virt/vmdevices/core.py
-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
On 16 Mar 2017, at 09:45, Francesco Romani <fromani@redhat.com> wrote:
We talked about sending storage device purely on metadata, letting Vdsm rebuild them and getting the XML like today.
In the other direction, Vdsm will pass through the XML (perhaps only parts of it, e.g. the devices subtree) like before.
This way we can minimize the changes we are uncertain of, and more importantly, we can minimize the risky changes.
The following is a realistic example of how the XML could look like if we send all but the storage devices. It is built using my pyxmlpickle module (see [3] below). That’s quite verbose. How much work would it need to actually minimize it and turn it into something more simple. Most such stuff should go away and I believe it would be beneficial to make it difficult to use to discourage using metadata as a generic junkyard
It is verbose because it is generic - indeed perhaps too generic. I can try something else based on a concept from Martin Polednik. Will follow up soon. Bests,
<domain type='kvm' id='10'> <name>a0</name> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0" xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0"> <ovirt-tune:qos/> <ovirt-vm:vm/> <ovirt-instance:instance> <ovirt-instance:value type="dict"> <ovirt-instance:item key="devices" type="list"> <ovirt-instance:item index="0" type="dict"> <ovirt-instance:item key="index" type="str">2</ovirt-instance:item> <ovirt-instance:item key="iface" type="str">ide</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="path" type="str"/> </ovirt-instance:item> <ovirt-instance:item key="readonly" type="str">true</ovirt-instance:item> <ovirt-instance:item key="deviceId" type="str">e59c985c-46c2-4489-b355-a6f374125eb9</ovirt-instance:item> <ovirt-instance:item key="address" type="dict"> <ovirt-instance:item key="bus" type="str">1</ovirt-instance:item> <ovirt-instance:item key="controller" type="str">0</ovirt-instance:item> <ovirt-instance:item key="type" type="str">drive</ovirt-instance:item> <ovirt-instance:item key="target" type="str">0</ovirt-instance:item> <ovirt-instance:item key="unit" type="str">0</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="device" type="str">cdrom</ovirt-instance:item> <ovirt-instance:item key="shared" type="str">false</ovirt-instance:item> <ovirt-instance:item key="path" type="str"/> <ovirt-instance:item key="type" type="str">disk</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="1" type="dict"> <ovirt-instance:item key="poolID" type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item> <ovirt-instance:item key="volumeInfo" type="dict"> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="volType" type="str">path</ovirt-instance:item> <ovirt-instance:item key="leaseOffset" type="int">0</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="leasePath" type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="index" type="int">0</ovirt-instance:item> <ovirt-instance:item key="iface" type="str">virtio</ovirt-instance:item> <ovirt-instance:item key="apparentsize" type="str">8589934592</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="readonly" type="str">false</ovirt-instance:item> <ovirt-instance:item key="shared" type="str">false</ovirt-instance:item> <ovirt-instance:item key="truesize" type="str">0</ovirt-instance:item> <ovirt-instance:item key="type" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="reqsize" type="str">0</ovirt-instance:item> <ovirt-instance:item key="format" type="str">raw</ovirt-instance:item> <ovirt-instance:item key="deviceId" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="address" type="dict"> <ovirt-instance:item key="function" type="str">0x0</ovirt-instance:item> <ovirt-instance:item key="bus" type="str">0x00</ovirt-instance:item> <ovirt-instance:item key="domain" type="str">0x0000</ovirt-instance:item> <ovirt-instance:item key="type" type="str">pci</ovirt-instance:item> <ovirt-instance:item key="slot" type="str">0x05</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="device" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="propagateErrors" type="str">off</ovirt-instance:item> <ovirt-instance:item key="optional" type="str">false</ovirt-instance:item> <ovirt-instance:item key="bootOrder" type="str">1</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"/> <ovirt-instance:item key="volumeChain" type="list"> <ovirt-instance:item index="0" type="dict"> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="volType" type="str">path</ovirt-instance:item> <ovirt-instance:item key="leaseOffset" type="int">0</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="leasePath" type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:value> </ovirt-instance:instance> </metadata> <maxMemory slots='16' unit='KiB'>4294967296</maxMemory> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static' current='2'>16</vcpu> <cputune> <shares>1020</shares> </cputune> <resource> <partition>/machine</partition> </resource> <sysinfo type='smbios'> <system> <entry name='manufacturer'>oVirt</entry> <entry name='product'>oVirt Node</entry> <entry name='version'>7-3.1611.el7.centos</entry> <entry name='serial'>ccd79775-c888-4789-975a-fde1143dffc9</entry> <entry name='uuid'>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</entry> </system> </sysinfo> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type> <bios useserial='yes'/> <smbios mode='sysinfo'/> </os> <features> <acpi/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell-noTSX</model> <topology sockets='16' cores='1' threads='1'/> <numa> <cell id='0' cpus='0-1' memory='4194304' unit='KiB'/> </numa> </cpu> <clock offset='variable' adjustment='0' basis='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <controller type='virtio-serial' index='0' ports='16'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='usb' index='0'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='bridge'> <mac address='00:1a:4a:16:01:51'/> <source bridge='ovirtmgmt'/> <target dev='vnet0'/> <model type='virtio'/> <filterref filter='vdsm-no-mac-spoofing'/> <link state='up'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='unix'> <source mode='bind' path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='unix'> <source mode='bind' path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channels/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.com.redhat.rhevm.vdsm'/> <target type='virtio' name='com.redhat.rhevm.vdsm' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channels/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0' state='disconnected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <graphics type='vnc' port='5900' autoport='yes' listen='192.168.1.53' keymap='en-us' passwdValidTo='1970-01-01T00:00:01'> <listen type='network' address='192.168.1.53' network='vdsm-ovirtmgmt'/> </graphics> <graphics type='spice' tlsPort='5901' autoport='yes' listen='192.168.1.53' keymap='en-us' defaultMode='secure' passwdValidTo='1970-01-01T00:00:01'> <listen type='network' address='192.168.1.53' network='vdsm-ovirtmgmt'/> <channel name='main' mode='secure'/> <channel name='display' mode='secure'/> <channel name='inputs' mode='secure'/> <channel name='cursor' mode='secure'/> <channel name='playback' mode='secure'/> <channel name='record' mode='secure'/> <channel name='smartcard' mode='secure'/> <channel name='usbredir' mode='secure'/> </graphics> <sound model='ich6'> <alias name='sound0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='8192' vgamem='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c207,c629</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c207,c629</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+107:+107</label> <imagelabel>+107:+107</imagelabel> </seclabel> </domain>
On 03/15/2017 01:28 PM, Francesco Romani wrote:
Hi everyone,
This is both a report of the current state of my Vdsm patches for Engine XML support, and a proposal to how move forward and solve the current open issues.
TL;DR: 1. we can and IMO should reuse the current JSON schema to describe the structure (layout) and the types of the metadata section. 2. we don't need a priori validation of stuff in the metadata section. We will just raise in the creation flow if data is missing, or wrong, according to our schema. 2. we will add *few* items to the metadata section, only thing we can't express clearly-or at all in the libvirt XML. Redundancy and verbosiness will be thus kept at bay 3. I believe [3] is the best tool to do (de)serialize data to the metadata section. Existing tools fits poorly in our very specific use case
Examples below
+++
Long(er) discussion:
I have working code[1][2] to encode any custom, picklable, python object in the metadata section.
We should decide which module will do the actual python<=>XML transformation. Please note that this actually also influences how the data in the medata section look like, so the two things are a bit coupled.
I'm eager to reinvent another wheel, but after initial evaluation I honestly think that my pyxmlpickle[3] is the best tool for the job over the current alternatives: plistlib[4] and xmltodict[5].
I added the initial rationale here: https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
I have completed the initial draft of patches to make it possible to initialize devices from their XML representation [6]. This is the bare minimum we need to support the Engine XML, and we *need* this anyway to unlock the cleanup we planned and I outlined in my google doc.
So we are progressing, but I'd like to speed up things. Those [6] patches are not yet complete, many flows are not covered or tested; but they are good enough to demonstrate that there *are* pieces of information wen need to properly initialize the devices, but we can't easily extract from the XML.
First examples that come to my mind are the storage.Drive UUIDs; there could also be some ambiguity I'm investigating right now for displayIp/displayNetwork in Graphics devices. In [6] there are various TODO to mark more of those cases. Most likely, few more cases will pop out as I cover all the flows we support.
Long story short: it is hard to correctly rebuild the device conf from the XML. This is why in [6] I added the 'meta' argument to from_xml_tree classmethod in [7].
'meta' is supposed to be the device metadata: extra data related to a device which doesn't (yet) fit in the libvirt XML representation. For example, we can store 'displayIp' and 'displayNetwork' here and be done with that: using both per-device metadata and the XML representation of one graphic device, we will have everything we need to properly build one graphics.Graphics device. This example may (hopefully) be bogus, but I'm keeping it because it is one case easy to follow.
The device metadata is going to be stored in the vm metadata for the short/mid term future. Even if the per-device metadata idea/RFE is accepted (no answer yet, but we are working on it), we will not have in 7.4, and unlikely in 7.5.
As it stands today, I believe there are two open questions:
1. do we need a schema for the metadata section? 2. how do we bind the metadata to the devices? How do we know which metadata belongs to which metadata, if we don't have aliases nor addresses to match? (e.g. very first time the VM is created!)
My current stance is the following 1. In general, one schema gives us two benefits: 1.a. we document how the layout of the data should be, including types; 1.b. we can validate the data we receive. So yes, we need a schema, but we don't need a *new* schema. I think we are in good enough shape with the current Vdsm schema: we can just translate the python object layout to a XML layout.
One example is probably more explicative. Some actual data may look like, using my pyxmlpickle module:
<domain type='kvm' id='5'> <name>a0</name> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0" xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0"> <ovirt-tune:qos/> <ovirt-vm:vm/> <ovirt-instance:instance> <ovirt-instance:value type="dict"> <ovirt-instance:item key="devices" type="list"> <ovirt-instance:item index="0" type="dict"> <ovirt-instance:item key="device" type="str">vnc</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="1" type="dict"> <ovirt-instance:item key="device" type="str">spice</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="2" type="dict"> <ovirt-instance:item key="poolID" type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="device" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:value> </ovirt-instance:instance> </metadata> <!-- omitted for brevity --> </domain>
Please note that yes, this is still verbose, but we don't want to add much data here, for most of information the most reliable source will be the domain XML. We will add here only the extra info we can't really fetch from that.
2. I don't think we need explicit validation: we could just raise along the way in the creation flow if we don't find some extra metadata we need. This will also solve the issue that if we reuse the current schema and we omit most of data, we will lack quite a lot of elements marked mandatory.
Once we reached agreement, I will update my https://docs.google.com/document/d/1eD8KSLwwyo2Sk64MytbmE0wBxxMlpIyEI1GRcHDk... accordingly.
Final note: while device take the lion's share, we will likely need help from the metadata section also to store VM extra info, but all the above discussion also applies here.
+++
[1] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... - uses xmltodict [2] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... ported the 'virt-metadata3' topic to pyxmlpickle [3] https://github.com/fromanirh/pyxmlpickle [4] https://docs.python.org/2/library/plistlib.html [5] https://github.com/martinblech/xmltodict [6] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vm... [7] https://gerrit.ovirt.org/#/c/72880/15/lib/vdsm/virt/vmdevices/core.py
-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani
On 03/16/2017 01:26 PM, Francesco Romani wrote:
On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
On 16 Mar 2017, at 09:45, Francesco Romani <fromani@redhat.com> wrote:
We talked about sending storage device purely on metadata, letting Vdsm rebuild them and getting the XML like today.
In the other direction, Vdsm will pass through the XML (perhaps only parts of it, e.g. the devices subtree) like before.
This way we can minimize the changes we are uncertain of, and more importantly, we can minimize the risky changes.
The following is a realistic example of how the XML could look like if we send all but the storage devices. It is built using my pyxmlpickle module (see [3] below). That’s quite verbose. How much work would it need to actually minimize it and turn it into something more simple. Most such stuff should go away and I believe it would be beneficial to make it difficult to use to discourage using metadata as a generic junkyard It is verbose because it is generic - indeed perhaps too generic. I can try something else based on a concept from Martin Polednik. Will follow up soon.
Early preview: https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... still plenty of TODOs, I expect to be reviewable material worst case monday morning. -- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani
On 03/16/2017 08:03 PM, Francesco Romani wrote:
On 03/16/2017 01:26 PM, Francesco Romani wrote:
On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
On 16 Mar 2017, at 09:45, Francesco Romani <fromani@redhat.com> wrote:
We talked about sending storage device purely on metadata, letting Vdsm rebuild them and getting the XML like today.
In the other direction, Vdsm will pass through the XML (perhaps only parts of it, e.g. the devices subtree) like before.
This way we can minimize the changes we are uncertain of, and more importantly, we can minimize the risky changes.
The following is a realistic example of how the XML could look like if we send all but the storage devices. It is built using my pyxmlpickle module (see [3] below). That’s quite verbose. How much work would it need to actually minimize it and turn it into something more simple. Most such stuff should go away and I believe it would be beneficial to make it difficult to use to discourage using metadata as a generic junkyard It is verbose because it is generic - indeed perhaps too generic. I can try something else based on a concept from Martin Polednik. Will follow up soon. Early preview: https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi...
still plenty of TODOs, I expect to be reviewable material worst case monday morning.
This is how typical XML could look like: <metadata> <ovirt-tune:qos /> <ovirt-vm:vm /> <devices> <ovirt-instance:graphics> <specParams> <fileTransferEnable>true</fileTransferEnable> <copyPasteEnable>true</copyPasteEnable> <displayIp>192.168.1.51</displayIp> <keyMap>en-us</keyMap> <spiceSslCipherSuite>DEFAULT</spiceSslCipherSuite> <spiceSecureChannels>smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir</spiceSecureChannels> <displayNetwork>ovirtmgmt</displayNetwork> </specParams> </ovirt-instance:graphics> <ovirt-instance:disk> <specParams> <path /> </specParams> </ovirt-instance:disk> <ovirt-instance:disk> <specParams /> <domainID>c578566d-bc61-420c-8f1e-8dfa0a18efd5</domainID> <volumeID>5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</volumeID> <poolID>5890a292-0390-01d2-01ed-00000000029a</poolID> <imageID>66441539-f7ac-4946-8a25-75e422f939d4</imageID> </ovirt-instance:disk> </devices> </metadata> still working on this -- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani
On Fri, Mar 17, 2017 at 4:58 PM Francesco Romani <fromani@redhat.com> wrote:
On 03/16/2017 08:03 PM, Francesco Romani wrote:
On 03/16/2017 01:26 PM, Francesco Romani wrote:
On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
On 16 Mar 2017, at 09:45, Francesco Romani <fromani@redhat.com> wrote:
We talked about sending storage device purely on metadata, letting Vdsm rebuild them and getting the XML like today.
In the other direction, Vdsm will pass through the XML (perhaps only parts of it, e.g. the devices subtree) like before.
This way we can minimize the changes we are uncertain of, and more importantly, we can minimize the risky changes.
The following is a realistic example of how the XML could look like if we send all but the storage devices. It is built using my pyxmlpickle module (see [3] below). That’s quite verbose. How much work would it need to actually minimize it and turn it into something more simple. Most such stuff should go away and I believe it would be beneficial to make it difficult to use to discourage using metadata as a generic junkyard It is verbose because it is generic - indeed perhaps too generic. I can try something else based on a concept from Martin Polednik. Will follow up soon. Early preview:
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi...
still plenty of TODOs, I expect to be reviewable material worst case monday morning.
This is how typical XML could look like:
<metadata> <ovirt-tune:qos /> <ovirt-vm:vm /> <devices>
Why do we need this nesting?
<ovirt-instance:graphics>
What is ovirt-instance? ovirt-tune and ovirt-vm make sense, I can guess what you mean, I don't have any idea what is ovirt-instance.
<specParams> <fileTransferEnable>true</fileTransferEnable> <copyPasteEnable>true</copyPasteEnable> <displayIp>192.168.1.51</displayIp> <keyMap>en-us</keyMap> <spiceSslCipherSuite>DEFAULT</spiceSslCipherSuite>
<spiceSecureChannels>smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir</spiceSecureChannels> <displayNetwork>ovirtmgmt</displayNetwork> </specParams> </ovirt-instance:graphics> <ovirt-instance:disk> <specParams>
Why do we need this nesting?
<path /> </specParams> </ovirt-instance:disk> <ovirt-instance:disk> <specParams /> <domainID>c578566d-bc61-420c-8f1e-8dfa0a18efd5</domainID>
We are using sd_id, img_id and vol_id now for these in new storage code.
<volumeID>5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</volumeID> <poolID>5890a292-0390-01d2-01ed-00000000029a</poolID> <imageID>66441539-f7ac-4946-8a25-75e422f939d4</imageID>
We need more info about disks, like diskType, shared, discard, etc. Can you show a complete vm xml with metadata?
</ovirt-instance:disk> </devices> </metadata>
still working on this
-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
This is a multi-part message in MIME format. --------------346863F9A850F6C853C7142A Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit On 03/18/2017 01:14 PM, Nir Soffer wrote:
On Fri, Mar 17, 2017 at 4:58 PM Francesco Romani <fromani@redhat.com <mailto:fromani@redhat.com>> wrote:
On 03/16/2017 08:03 PM, Francesco Romani wrote: > On 03/16/2017 01:26 PM, Francesco Romani wrote: >> On 03/16/2017 11:47 AM, Michal Skrivanek wrote: >>>> On 16 Mar 2017, at 09:45, Francesco Romani <fromani@redhat.com <mailto:fromani@redhat.com>> wrote: >>>> >>>> We talked about sending storage device purely on metadata, letting Vdsm >>>> rebuild them and getting the XML like today. >>>> >>>> In the other direction, Vdsm will pass through the XML (perhaps only >>>> parts of it, e.g. the devices subtree) like before. >>>> >>>> This way we can minimize the changes we are uncertain of, and more >>>> importantly, we can minimize the risky changes. >>>> >>>> >>>> The following is a realistic example of how the XML could look like if >>>> we send all but the storage devices. It is built using my pyxmlpickle >>>> module (see [3] below). >>> That’s quite verbose. How much work would it need to actually minimize it and turn it into something more simple. >>> Most such stuff should go away and I believe it would be beneficial to make it difficult to use to discourage using metadata as a generic junkyard >> It is verbose because it is generic - indeed perhaps too generic. >> I can try something else based on a concept from Martin Polednik. Will >> follow up soon. > Early preview: > https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... > > still plenty of TODOs, I expect to be reviewable material worst case > monday morning.
This is how typical XML could look like:
<metadata> <ovirt-tune:qos /> <ovirt-vm:vm /> <devices>
Why do we need this nesting?
We use libvirt metadata for elements; so each direct child of metadata is a separate metadata grop: ovirt-tune:qos is one ovirt-vm:vm is one and so forth. I don't want to mess up with the existing elements. So we could be backward compatible at XML level (read: 4.1 XML works without changes to 4.2)
<ovirt-instance:graphics>
What is ovirt-instance?
Gone, merged with ovirt-vm ovirt-vm will hold both per-vm metadata and per-device metadata.
<specParams> <fileTransferEnable>true</fileTransferEnable> <copyPasteEnable>true</copyPasteEnable> <displayIp>192.168.1.51</displayIp> <keyMap>en-us</keyMap> <spiceSslCipherSuite>DEFAULT</spiceSslCipherSuite>
<spiceSecureChannels>smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir</spiceSecureChannels> <displayNetwork>ovirtmgmt</displayNetwork> </specParams> </ovirt-instance:graphics> <ovirt-instance:disk> <specParams>
Why do we need this nesting?
*Here* we had this nesting because the example took vm.conf and brutally translated to XML. Is a worst case scenario. Why is it relevant? There it was initial discussion about how to deal with complex device; to minimize changes, we could marshal the existing vm.conf into the device metadata, then unmarshal on Vdsm side and just use it to rebuild the devices with the very same code we have today (yes, this means sneaking/embedding vm.conf into the XML) Should we go that way, it could look like the above. Let's talk in general now. There are three main use cases requiring nesting: 1. per-vm metadta. Attach key/value pairs. We need one level of nesting to avoid to mess up with other data. So it could look like <metadata> <ovirt-vm:vm> <foo>1</foo> <bar>2</bar> </ovirt-vm:vm> </metadata> this is simple and nice and I think is not bothering anyone (hopefully :)) 2. per-device metadata: it has to fit into vm section, and we could possibly have more than one device with metadata, so the simplest format is something like <metadata> <ovirt-vm:vm> <foo>1</foo> <bar>2</bar> <devices> <device> <fast>true</fast> <cheap>false</cheap> </device> </devices> </ovirt-vm:vm> </metadata> This is the minimal nesting level we need. We could gather the per-device metadata in a dict and feed device with it, like with a new "meta" argument to device constructor, much like "custom" and "specParams" Would that look good? 3. QoS. we need to support the current layout for obvious backward compatibility questions. We could postpone this and use existing code for some more time, but ultimately this should handled by metadata module, just because it is supposed to be the process-wide metadata gateway.
<path /> </specParams> </ovirt-instance:disk> <ovirt-instance:disk> <specParams />
<domainID>c578566d-bc61-420c-8f1e-8dfa0a18efd5</domainID>
We are using sd_id, img_id and vol_id now for these in new storage code.
Noted
<volumeID>5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</volumeID> <poolID>5890a292-0390-01d2-01ed-00000000029a</poolID>
<imageID>66441539-f7ac-4946-8a25-75e422f939d4</imageID>
We need more info about disks, like diskType, shared, discard, etc.
Can you show a complete vm xml with metadata?
Will post as soon as possible. -- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani --------------346863F9A850F6C853C7142A Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> On 03/18/2017 01:14 PM, Nir Soffer wrote:<br> <blockquote cite="mid:CAMRbyyu0i7vE6Qz7Xr2vz=3BmBVH6CtATavuGhL06==mO_K2bw@mail.gmail.com" type="cite"> <div dir="ltr"><br> <br> <div class="gmail_quote"> <div dir="ltr">On Fri, Mar 17, 2017 at 4:58 PM Francesco Romani <<a moz-do-not-send="true" href="mailto:fromani@redhat.com">fromani@redhat.com</a>> wrote:<br> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 03/16/2017 08:03 PM, Francesco Romani wrote:<br class="gmail_msg"> > On 03/16/2017 01:26 PM, Francesco Romani wrote:<br class="gmail_msg"> >> On 03/16/2017 11:47 AM, Michal Skrivanek wrote:<br class="gmail_msg"> >>>> On 16 Mar 2017, at 09:45, Francesco Romani <<a moz-do-not-send="true" href="mailto:fromani@redhat.com" class="gmail_msg" target="_blank">fromani@redhat.com</a>> wrote:<br class="gmail_msg"> >>>><br class="gmail_msg"> >>>> We talked about sending storage device purely on metadata, letting Vdsm<br class="gmail_msg"> >>>> rebuild them and getting the XML like today.<br class="gmail_msg"> >>>><br class="gmail_msg"> >>>> In the other direction, Vdsm will pass through the XML (perhaps only<br class="gmail_msg"> >>>> parts of it, e.g. the devices subtree) like before.<br class="gmail_msg"> >>>><br class="gmail_msg"> >>>> This way we can minimize the changes we are uncertain of, and more<br class="gmail_msg"> >>>> importantly, we can minimize the risky changes.<br class="gmail_msg"> >>>><br class="gmail_msg"> >>>><br class="gmail_msg"> >>>> The following is a realistic example of how the XML could look like if<br class="gmail_msg"> >>>> we send all but the storage devices. It is built using my pyxmlpickle<br class="gmail_msg"> >>>> module (see [3] below).<br class="gmail_msg"> >>> That’s quite verbose. How much work would it need to actually minimize it and turn it into something more simple.<br class="gmail_msg"> >>> Most such stuff should go away and I believe it would be beneficial to make it difficult to use to discourage using metadata as a generic junkyard<br class="gmail_msg"> >> It is verbose because it is generic - indeed perhaps too generic.<br class="gmail_msg"> >> I can try something else based on a concept from Martin Polednik. Will<br class="gmail_msg"> >> follow up soon.<br class="gmail_msg"> > Early preview:<br class="gmail_msg"> > <a moz-do-not-send="true" href="https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi..." rel="noreferrer" class="gmail_msg" target="_blank">https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:virt-metadata-compact</a><br class="gmail_msg"> ><br class="gmail_msg"> > still plenty of TODOs, I expect to be reviewable material worst case<br class="gmail_msg"> > monday morning.<br class="gmail_msg"> <br class="gmail_msg"> This is how typical XML could look like:<br class="gmail_msg"> <br class="gmail_msg"> <metadata><br class="gmail_msg"> <ovirt-tune:qos /><br class="gmail_msg"> <ovirt-vm:vm /><br class="gmail_msg"> <devices><br class="gmail_msg"> </blockquote> <div><br> </div> <div>Why do we need this nesting?</div> </div> </div> </blockquote> <br> We use libvirt metadata for elements; so each direct child of metadata is a separate metadata grop:<br> <br> ovirt-tune:qos is one<br> ovirt-vm:vm is one<br> <br> and so forth. I don't want to mess up with the existing elements. So we could be backward compatible at XML level (read: 4.1 XML works without changes to 4.2)<br> <br> <br> <blockquote cite="mid:CAMRbyyu0i7vE6Qz7Xr2vz=3BmBVH6CtATavuGhL06==mO_K2bw@mail.gmail.com" type="cite"> <div dir="ltr"> <div class="gmail_quote"> <div> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <ovirt-instance:graphics><br class="gmail_msg"> </blockquote> <div><br> </div> <div>What is ovirt-instance?</div> </div> </div> </blockquote> <br> Gone, merged with ovirt-vm<br> <br> ovirt-vm will hold both per-vm metadata and per-device metadata.<br> <br> <blockquote cite="mid:CAMRbyyu0i7vE6Qz7Xr2vz=3BmBVH6CtATavuGhL06==mO_K2bw@mail.gmail.com" type="cite"> <div dir="ltr"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <specParams><br class="gmail_msg"> <fileTransferEnable>true</fileTransferEnable><br class="gmail_msg"> <copyPasteEnable>true</copyPasteEnable><br class="gmail_msg"> <displayIp>192.168.1.51</displayIp><br class="gmail_msg"> <keyMap>en-us</keyMap><br class="gmail_msg"> <spiceSslCipherSuite>DEFAULT</spiceSslCipherSuite><br class="gmail_msg"> <br class="gmail_msg"> <spiceSecureChannels>smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir</spiceSecureChannels><br class="gmail_msg"> <displayNetwork>ovirtmgmt</displayNetwork><br class="gmail_msg"> </specParams><br class="gmail_msg"> </ovirt-instance:graphics><br class="gmail_msg"> <ovirt-instance:disk><br class="gmail_msg"> <specParams><br class="gmail_msg"> </blockquote> <div><br> </div> <div>Why do we need this nesting?</div> </div> </div> </blockquote> <br> *Here* we had this nesting because the example took vm.conf and brutally translated to XML. Is a worst case scenario.<br> <br> Why is it relevant? There it was initial discussion about how to deal with complex device; to minimize changes, we could<br> marshal the existing vm.conf into the device metadata, then unmarshal on Vdsm side and just use it to rebuild the devices<br> with the very same code we have today (yes, this means sneaking/embedding vm.conf into the XML)<br> <br> Should we go that way, it could look like the above.<br> <br> <br> Let's talk in general now. There are three main use cases requiring nesting:<br> <br> 1. per-vm metadta. Attach key/value pairs. We need one level of nesting to avoid to mess up with other data. So it could look like<br> <br> <metadata><br> <ovirt-vm:vm><br> <foo>1</foo><br> <bar>2</bar><br> </ovirt-vm:vm><br> </metadata><br> <br> this is simple and nice and I think is not bothering anyone (hopefully :))<br> <br> 2. per-device metadata: it has to fit into vm section, and we could possibly have more than one device with metadata, so the simplest format is something like<br> <br> <metadata><br> <ovirt-vm:vm><br> <foo>1</foo><br> <bar>2</bar><br> <devices><br> <device><br> <fast>true</fast><br> <cheap>false</cheap><br> </device><br> </devices><br> </ovirt-vm:vm><br> </metadata><br> <br> This is the minimal nesting level we need. We could gather the per-device metadata in a dict and feed device with it, like with a new "meta" argument to device constructor, much like "custom" and "specParams"<br> <br> Would that look good?<br> <br> <br> 3. QoS. we need to support the current layout for obvious backward compatibility questions. We could postpone this and use existing code for some more time, but ultimately this should handled by metadata module, just because it is supposed to be<br> the process-wide metadata gateway.<br> <br> <blockquote cite="mid:CAMRbyyu0i7vE6Qz7Xr2vz=3BmBVH6CtATavuGhL06==mO_K2bw@mail.gmail.com" type="cite"> <div dir="ltr"> <div class="gmail_quote"> <div> </div> </div> </div> </blockquote> <blockquote cite="mid:CAMRbyyu0i7vE6Qz7Xr2vz=3BmBVH6CtATavuGhL06==mO_K2bw@mail.gmail.com" type="cite"> <div dir="ltr"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <path /><br class="gmail_msg"> </specParams><br class="gmail_msg"> </ovirt-instance:disk><br class="gmail_msg"> <ovirt-instance:disk><br class="gmail_msg"> <specParams /><br class="gmail_msg"> <domainID>c578566d-bc61-420c-8f1e-8dfa0a18efd5</domainID><br class="gmail_msg"> </blockquote> <div><br> </div> <div>We are using sd_id, img_id and vol_id now for these in new storage code.</div> </div> </div> </blockquote> <br> Noted<br> <br> <blockquote cite="mid:CAMRbyyu0i7vE6Qz7Xr2vz=3BmBVH6CtATavuGhL06==mO_K2bw@mail.gmail.com" type="cite"> <div dir="ltr"> <div class="gmail_quote"> <div> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <volumeID>5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</volumeID><br class="gmail_msg"> <poolID>5890a292-0390-01d2-01ed-00000000029a</poolID><br class="gmail_msg"> <imageID>66441539-f7ac-4946-8a25-75e422f939d4</imageID><br class="gmail_msg"> </blockquote> <div><br> </div> <div>We need more info about disks, like diskType, shared, discard, etc.</div> <div><br> </div> <div>Can you show a complete vm xml with metadata?</div> </div> </div> </blockquote> <br> Will post as soon as possible.<br> <br> <pre class="moz-signature" cols="72">-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani</pre> </body> </html> --------------346863F9A850F6C853C7142A--
On 22/03/17 16:52 +0100, Francesco Romani wrote:
On 03/18/2017 01:14 PM, Nir Soffer wrote:
On Fri, Mar 17, 2017 at 4:58 PM Francesco Romani <fromani@redhat.com <mailto:fromani@redhat.com>> wrote:
On 03/16/2017 08:03 PM, Francesco Romani wrote: > On 03/16/2017 01:26 PM, Francesco Romani wrote: >> On 03/16/2017 11:47 AM, Michal Skrivanek wrote: >>>> On 16 Mar 2017, at 09:45, Francesco Romani <fromani@redhat.com <mailto:fromani@redhat.com>> wrote: >>>> >>>> We talked about sending storage device purely on metadata, letting Vdsm >>>> rebuild them and getting the XML like today. >>>> >>>> In the other direction, Vdsm will pass through the XML (perhaps only >>>> parts of it, e.g. the devices subtree) like before. >>>> >>>> This way we can minimize the changes we are uncertain of, and more >>>> importantly, we can minimize the risky changes. >>>> >>>> >>>> The following is a realistic example of how the XML could look like if >>>> we send all but the storage devices. It is built using my pyxmlpickle >>>> module (see [3] below). >>> That’s quite verbose. How much work would it need to actually minimize it and turn it into something more simple. >>> Most such stuff should go away and I believe it would be beneficial to make it difficult to use to discourage using metadata as a generic junkyard >> It is verbose because it is generic - indeed perhaps too generic. >> I can try something else based on a concept from Martin Polednik. Will >> follow up soon. > Early preview: > https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... > > still plenty of TODOs, I expect to be reviewable material worst case > monday morning.
This is how typical XML could look like:
<metadata> <ovirt-tune:qos /> <ovirt-vm:vm /> <devices>
Why do we need this nesting?
We use libvirt metadata for elements; so each direct child of metadata is a separate metadata grop:
ovirt-tune:qos is one ovirt-vm:vm is one
and so forth. I don't want to mess up with the existing elements. So we could be backward compatible at XML level (read: 4.1 XML works without changes to 4.2)
<ovirt-instance:graphics>
What is ovirt-instance?
Gone, merged with ovirt-vm
ovirt-vm will hold both per-vm metadata and per-device metadata.
<specParams> <fileTransferEnable>true</fileTransferEnable> <copyPasteEnable>true</copyPasteEnable> <displayIp>192.168.1.51</displayIp> <keyMap>en-us</keyMap> <spiceSslCipherSuite>DEFAULT</spiceSslCipherSuite>
<spiceSecureChannels>smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir</spiceSecureChannels> <displayNetwork>ovirtmgmt</displayNetwork> </specParams> </ovirt-instance:graphics> <ovirt-instance:disk> <specParams>
Why do we need this nesting?
*Here* we had this nesting because the example took vm.conf and brutally translated to XML. Is a worst case scenario.
Why is it relevant? There it was initial discussion about how to deal with complex device; to minimize changes, we could marshal the existing vm.conf into the device metadata, then unmarshal on Vdsm side and just use it to rebuild the devices with the very same code we have today (yes, this means sneaking/embedding vm.conf into the XML)
Should we go that way, it could look like the above.
Let's talk in general now. There are three main use cases requiring nesting:
1. per-vm metadta. Attach key/value pairs. We need one level of nesting to avoid to mess up with other data. So it could look like
<metadata> <ovirt-vm:vm> <foo>1</foo> <bar>2</bar> </ovirt-vm:vm> </metadata>
this is simple and nice and I think is not bothering anyone (hopefully :))
2. per-device metadata: it has to fit into vm section, and we could possibly have more than one device with metadata, so the simplest format is something like
<metadata> <ovirt-vm:vm> <foo>1</foo> <bar>2</bar> <devices> <device> <fast>true</fast> <cheap>false</cheap> </device> </devices> </ovirt-vm:vm> </metadata>
This is the minimal nesting level we need. We could gather the per-device metadata in a dict and feed device with it, like with a new "meta" argument to device constructor, much like "custom" and "specParams"
Would that look good?
OK to me. Won't be easy to design an API that doesn't look bad, isn't too generic and is semi-hard to use, but the design seems fine.
3. QoS. we need to support the current layout for obvious backward compatibility questions. We could postpone this and use existing code for some more time, but ultimately this should handled by metadata module, just because it is supposed to be the process-wide metadata gateway.
I'd say let's wait with QoS and consider options to change it without breaking backwards compatibility first. If that fails, let's see how new code could handle that.
<path /> </specParams> </ovirt-instance:disk> <ovirt-instance:disk> <specParams />
<domainID>c578566d-bc61-420c-8f1e-8dfa0a18efd5</domainID>
We are using sd_id, img_id and vol_id now for these in new storage code.
Noted
<volumeID>5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</volumeID> <poolID>5890a292-0390-01d2-01ed-00000000029a</poolID>
<imageID>66441539-f7ac-4946-8a25-75e422f939d4</imageID>
We need more info about disks, like diskType, shared, discard, etc.
Can you show a complete vm xml with metadata?
Will post as soon as possible.
-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Wed, Mar 15, 2017 at 2:28 PM Francesco Romani <fromani@redhat.com> wrote: Hi everyone, This is both a report of the current state of my Vdsm patches for Engine XML support, and a proposal to how move forward and solve the current open issues. TL;DR: 1. we can and IMO should reuse the current JSON schema to describe the structure (layout) and the types of the metadata section. 2. we don't need a priori validation of stuff in the metadata section. We will just raise in the creation flow if data is missing, or wrong, according to our schema. 2. we will add *few* items to the metadata section, only thing we can't express clearly-or at all in the libvirt XML. Redundancy and verbosiness will be thus kept at bay 3. I believe [3] is the best tool to do (de)serialize data to the metadata section. Existing tools fits poorly in our very specific use case Examples below +++ Long(er) discussion: I have working code[1][2] to encode any custom, picklable, python object in the metadata section. We should decide which module will do the actual python<=>XML transformation. Please note that this actually also influences how the data in the medata section look like, so the two things are a bit coupled. I'm eager to reinvent another wheel, but after initial evaluation I honestly think that my pyxmlpickle[3] is the best tool for the job over the current alternatives: plistlib[4] and xmltodict[5]. I added the initial rationale here: https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG I have completed the initial draft of patches to make it possible to initialize devices from their XML representation [6]. This is the bare minimum we need to support the Engine XML, and we *need* this anyway to unlock the cleanup we planned and I outlined in my google doc. So we are progressing, but I'd like to speed up things. Those [6] patches are not yet complete, many flows are not covered or tested; but they are good enough to demonstrate that there *are* pieces of information wen need to properly initialize the devices, but we can't easily extract from the XML. First examples that come to my mind are the storage.Drive UUIDs; there could also be some ambiguity I'm investigating right now for displayIp/displayNetwork in Graphics devices. In [6] there are various TODO to mark more of those cases. Most likely, few more cases will pop out as I cover all the flows we support. Long story short: it is hard to correctly rebuild the device conf from the XML. This is why in [6] I added the 'meta' argument to from_xml_tree classmethod in [7]. 'meta' is supposed to be the device metadata: extra data related to a device which doesn't (yet) fit in the libvirt XML representation. For example, we can store 'displayIp' and 'displayNetwork' here and be done with that: using both per-device metadata and the XML representation of one graphic device, we will have everything we need to properly build one graphics.Graphics device. This example may (hopefully) be bogus, but I'm keeping it because it is one case easy to follow. The device metadata is going to be stored in the vm metadata for the short/mid term future. Even if the per-device metadata idea/RFE is accepted (no answer yet, but we are working on it), we will not have in 7.4, and unlikely in 7.5. As it stands today, I believe there are two open questions: 1. do we need a schema for the metadata section? 2. how do we bind the metadata to the devices? How do we know which metadata belongs to which metadata, if we don't have aliases nor addresses to match? (e.g. very first time the VM is created!) My current stance is the following 1. In general, one schema gives us two benefits: 1.a. we document how the layout of the data should be, including types; 1.b. we can validate the data we receive. So yes, we need a schema, but we don't need a *new* schema. I think we are in good enough shape with the current Vdsm schema: we can just translate the python object layout to a XML layout. One example is probably more explicative. Some actual data may look like, using my pyxmlpickle module: <domain type='kvm' id='5'> <name>a0</name> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" The url is broken - do we have this? xmlns:ovirt-vm="http://ovirt.org/vm/1.0" xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0"> <ovirt-tune:qos/> What do we keep here? why does it need its own namespace? <ovirt-vm:vm/> What do we keep here? why does it need its own namespace? Can we merge all namespaces into one generic namespace? <ovirt-instance:instance> What is instance? <ovirt-instance:value type="dict"> <ovirt-instance:item key="devices" type="list"> <ovirt-instance:item index="0" type="dict"> Isn't index redundant? <ovirt-instance:item key="device" type="str">vnc</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="1" type="dict"> <ovirt-instance:item key="device" type="str">spice</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="2" type="dict"> <ovirt-instance:item key="poolID" type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="device" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:value> </ovirt-instance:instance> </metadata> <!-- omitted for brevity --> </domain> How about a generic json namespace? json: { "devices": [ { "foo": "bar" } ] } xml: <json:object> <json:key>devices</json:key> <json:list> <json:object> <json:key>foo</json:key> <json:string>bar</json:string> </json:object> </json:list> </json:object> We can query and modify this xml using xpath, like this:
r.findall("./metadata/json:object/[json:key='devices']/json:list/json:object/", namespaces={"json": "http://ovirt.org/json/1.0"}) [<Element '{http://ovirt.org/json/1.0}key' at 0x7fd3386a69d0>, <Element '{ http://ovirt.org/json/1.0}string' at 0x7fd3386a6a50>]
Not sure this format will be easy to modify, needs to tinker more with this. It will be probably easier to parse the entire metadata, change it, and serialize it back. Maybe using key and type attribute as you suggest makes it simpler to use, and we since we convert from xml to python or python to xml, we can have a "py" namespace. <py:item> <py:item key="devices" type="list"> <py:item type="dict"> <py:item key="foo" type="str">bar</py:item> </py:item> </py:item> </py:item> With this we can have:
r.findall("./metadata/py:item/py:item[@key='devices']/py:item[1]/py:item[@key='foo']", namespaces={"py": "http://ovirt.org/py/1.0"})[0].text 'bar'
Something like this can be useful for others as well. Please note that yes, this is still verbose, but we don't want to add much data here, for most of information the most reliable source will be the domain XML. We will add here only the extra info we can't really fetch from that. 2. I don't think we need explicit validation: we could just raise along the way in the creation flow if we don't find some extra metadata we need. This will also solve the issue that if we reuse the current schema and we omit most of data, we will lack quite a lot of elements marked mandatory. Once we reached agreement, I will update my https://docs.google.com/document/d/1eD8KSLwwyo2Sk64MytbmE0wBxxMlpIyEI1GRcHDk... accordingly. Final note: while device take the lion's share, we will likely need help from the metadata section also to store VM extra info, but all the above discussion also applies here. +++ [1] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... - uses xmltodict [2] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... ported the 'virt-metadata3' topic to pyxmlpickle [3] https://github.com/fromanirh/pyxmlpickle [4] https://docs.python.org/2/library/plistlib.html [5] https://github.com/martinblech/xmltodict [6] https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vm... [7] https://gerrit.ovirt.org/#/c/72880/15/lib/vdsm/virt/vmdevices/core.py -- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
This is a multi-part message in MIME format. --------------15F6CAD0BC7802BD4F39C8F8 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Please note that the approach described here is outdated, because xmlpickle was too easy to use and too generic: that lead to bloated XML and too high risk of misusing metadata; the new approach (https://gerrit.ovirt.org/#/c/74206/) tries to balance things making usage convenient but disallowing arbitrarily nesting On 03/18/2017 03:27 PM, Nir Soffer wrote:
On Wed, Mar 15, 2017 at 2:28 PM Francesco Romani <fromani@redhat.com <mailto:fromani@redhat.com>> wrote:
Hi everyone,
This is both a report of the current state of my Vdsm patches for Engine XML support, and a proposal to how move forward and solve the current open issues.
TL;DR: 1. we can and IMO should reuse the current JSON schema to describe the structure (layout) and the types of the metadata section. 2. we don't need a priori validation of stuff in the metadata section. We will just raise in the creation flow if data is missing, or wrong, according to our schema. 2. we will add *few* items to the metadata section, only thing we can't express clearly-or at all in the libvirt XML. Redundancy and verbosiness will be thus kept at bay 3. I believe [3] is the best tool to do (de)serialize data to the metadata section. Existing tools fits poorly in our very specific use case
Examples below
+++
Long(er) discussion:
I have working code[1][2] to encode any custom, picklable, python object in the metadata section.
We should decide which module will do the actual python<=>XML transformation. Please note that this actually also influences how the data in the medata section look like, so the two things are a bit coupled.
I'm eager to reinvent another wheel, but after initial evaluation I honestly think that my pyxmlpickle[3] is the best tool for the job over the current alternatives: plistlib[4] and xmltodict[5].
I added the initial rationale here: https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
I have completed the initial draft of patches to make it possible to initialize devices from their XML representation [6]. This is the bare minimum we need to support the Engine XML, and we *need* this anyway to unlock the cleanup we planned and I outlined in my google doc.
So we are progressing, but I'd like to speed up things. Those [6] patches are not yet complete, many flows are not covered or tested; but they are good enough to demonstrate that there *are* pieces of information wen need to properly initialize the devices, but we can't easily extract from the XML.
First examples that come to my mind are the storage.Drive UUIDs; there could also be some ambiguity I'm investigating right now for displayIp/displayNetwork in Graphics devices. In [6] there are various TODO to mark more of those cases. Most likely, few more cases will pop out as I cover all the flows we support.
Long story short: it is hard to correctly rebuild the device conf from the XML. This is why in [6] I added the 'meta' argument to from_xml_tree classmethod in [7].
'meta' is supposed to be the device metadata: extra data related to a device which doesn't (yet) fit in the libvirt XML representation. For example, we can store 'displayIp' and 'displayNetwork' here and be done with that: using both per-device metadata and the XML representation of one graphic device, we will have everything we need to properly build one graphics.Graphics device. This example may (hopefully) be bogus, but I'm keeping it because it is one case easy to follow.
The device metadata is going to be stored in the vm metadata for the short/mid term future. Even if the per-device metadata idea/RFE is accepted (no answer yet, but we are working on it), we will not have in 7.4, and unlikely in 7.5.
As it stands today, I believe there are two open questions:
1. do we need a schema for the metadata section? 2. how do we bind the metadata to the devices? How do we know which metadata belongs to which metadata, if we don't have aliases nor addresses to match? (e.g. very first time the VM is created!)
My current stance is the following 1. In general, one schema gives us two benefits: 1.a. we document how the layout of the data should be, including types; 1.b. we can validate the data we receive. So yes, we need a schema, but we don't need a *new* schema. I think we are in good enough shape with the current Vdsm schema: we can just translate the python object layout to a XML layout.
One example is probably more explicative. Some actual data may look like, using my pyxmlpickle module:
<domain type='kvm' id='5'> <name>a0</name> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0"
The url is broken - do we have this?
I guess no. This is old code, dating back to 3.6,/which I'm taking without changes./
xmlns:ovirt-vm="http://ovirt.org/vm/1.0" xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0"> <ovirt-tune:qos/>
What do we keep here? why does it need its own namespace?
yes, because of how we use the metadata: http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainGetMetadata http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainSetMetadata we use type = http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_METADATA_ELEM... -- I don't think we should change that
<ovirt-vm:vm/>
What do we keep here? why does it need its own namespace?
same as above
Can we merge all namespaces into one generic namespace?
Probably, but we have the nice property that code can trivially access one subset of metadata, e.g. only the qos subtree, only the container subtree. Furthermore if we don't change that we are backward compatible at XML level (inbound migrations care about that).
<ovirt-instance:instance>
What is instance?
Gone, it duplicated vm without a good reason
<ovirt-instance:value type="dict">
<ovirt-instance:item key="devices" type="list">
<ovirt-instance:item index="0" type="dict">
Isn't index redundant?
It is, but now is gone
<ovirt-instance:item key="device" type="str">vnc</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="1" type="dict"> <ovirt-instance:item key="device" type="str">spice</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="2" type="dict"> <ovirt-instance:item key="poolID" type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="device" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="path" type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:value> </ovirt-instance:instance> </metadata> <!-- omitted for brevity --> </domain>
How about a generic json namespace?
json:
{ "devices": [ { "foo": "bar" } ] }
xml:
<json:object> <json:key>devices</json:key> <json:list> <json:object> <json:key>foo</json:key> <json:string>bar</json:string> </json:object> </json:list> </json:object>
We can query and modify this xml using xpath, like this:
r.findall("./metadata/json:object/[json:key='devices']/json:list/json:object/", namespaces={"json": "http://ovirt.org/json/1.0"}) [<Element '{http://ovirt.org/json/1.0}key <http://ovirt.org/json/1.0%7Dkey>' at 0x7fd3386a69d0>, <Element '{http://ovirt.org/json/1.0}string <http://ovirt.org/json/1.0%7Dstring>' at 0x7fd3386a6a50>]
Not sure this format will be easy to modify, needs to tinker more with this.
It will be probably easier to parse the entire metadata, change it, and serialize it back.
Maybe using key and type attribute as you suggest makes it simpler to use, and we since we convert from xml to python or python to xml, we can have a "py" namespace.
<py:item> <py:item key="devices" type="list"> <py:item type="dict"> <py:item key="foo" type="str">bar</py:item> </py:item> </py:item> </py:item>
With this we can have:
r.findall("./metadata/py:item/py:item[@key='devices']/py:item[1]/py:item[@key='foo']", namespaces={"py": "http://ovirt.org/py/1.0"})[0].text 'bar'
Something like this can be useful for others as well.
Great suggestions, I like the py namespace, but concerns were raised about the solution here being too generic, so I limited it, see https://gerrit.ovirt.org/#/c/74206/11 but I will add my pyxmlpickle module (https://github.com/fromanirh/pyxmlpickle) anyway in the future according to your comments. -- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani --------------15F6CAD0BC7802BD4F39C8F8 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <p>Please note that the approach described here is outdated, because xmlpickle was too easy to use</p> <p>and too generic: that lead to bloated XML and too high risk of misusing metadata;</p> <p>the new approach (<a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/74206/">https://gerrit.ovirt.org/#/c/74206/</a>) tries to balance things making usage</p> <p>convenient but disallowing arbitrarily nesting<br> </p> <br> <div class="moz-cite-prefix">On 03/18/2017 03:27 PM, Nir Soffer wrote:<br> </div> <blockquote cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com" type="cite"> <div dir="ltr"> <div dir="ltr" class="gmail_msg"> <div class="gmail_quote gmail_msg"> <div dir="ltr" class="gmail_msg">On Wed, Mar 15, 2017 at 2:28 PM Francesco Romani <<a moz-do-not-send="true" href="mailto:fromani@redhat.com" class="gmail_msg" target="_blank">fromani@redhat.com</a>> wrote:<br class="gmail_msg"> </div> <blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi everyone,<br class="gmail_msg"> <br class="gmail_msg"> This is both a report of the current state of my Vdsm patches for Engine<br class="gmail_msg"> XML support, and a proposal to how move forward and solve<br class="gmail_msg"> the current open issues.<br class="gmail_msg"> <br class="gmail_msg"> TL;DR:<br class="gmail_msg"> 1. we can and IMO should reuse the current JSON schema to describe the<br class="gmail_msg"> structure (layout) and the types of the metadata section.<br class="gmail_msg"> 2. we don't need a priori validation of stuff in the metadata section.<br class="gmail_msg"> We will just raise in the creation flow if data is missing, or wrong,<br class="gmail_msg"> according to our schema.<br class="gmail_msg"> 2. we will add *few* items to the metadata section, only thing we can't<br class="gmail_msg"> express clearly-or at all in the libvirt XML. Redundancy and verbosiness<br class="gmail_msg"> will be thus kept at bay<br class="gmail_msg"> 3. I believe [3] is the best tool to do (de)serialize data to the<br class="gmail_msg"> metadata section. Existing tools fits poorly in our very specific use case<br class="gmail_msg"> <br class="gmail_msg"> Examples below<br class="gmail_msg"> <br class="gmail_msg"> +++<br class="gmail_msg"> <br class="gmail_msg"> Long(er) discussion:<br class="gmail_msg"> <br class="gmail_msg"> <br class="gmail_msg"> I have working code[1][2] to encode any custom, picklable, python<br class="gmail_msg"> object in the metadata section.<br class="gmail_msg"> <br class="gmail_msg"> We should decide which module will do the actual python<=>XML<br class="gmail_msg"> transformation.<br class="gmail_msg"> Please note that this actually also influences how the data in the<br class="gmail_msg"> medata section look like, so the two things are a bit coupled.<br class="gmail_msg"> <br class="gmail_msg"> I'm eager to reinvent another wheel, but after<br class="gmail_msg"> initial evaluation I honestly think that my pyxmlpickle[3] is the best<br class="gmail_msg"> tool for the job over the current alternatives: plistlib[4] and<br class="gmail_msg"> xmltodict[5].<br class="gmail_msg"> <br class="gmail_msg"> I added the initial rationale here:<br class="gmail_msg"> <a moz-do-not-send="true" href="https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG" rel="noreferrer" class="gmail_msg" target="_blank">https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG</a><br class="gmail_msg"> <br class="gmail_msg"> I have completed the initial draft of patches to make it possible to<br class="gmail_msg"> initialize devices from their XML representation [6]. This is the bare<br class="gmail_msg"> minimum we need to support the Engine XML, and we *need* this anyway to<br class="gmail_msg"> unlock the cleanup we planned and I outlined in my google doc.<br class="gmail_msg"> <br class="gmail_msg"> So we are progressing, but I'd like to speed up things. Those [6]<br class="gmail_msg"> patches are not yet complete, many flows are not covered or tested; but<br class="gmail_msg"> they are good enough to demonstrate that there *are* pieces of<br class="gmail_msg"> information wen need to properly initialize the devices, but we can't<br class="gmail_msg"> easily extract from the XML.<br class="gmail_msg"> <br class="gmail_msg"> First examples that come to my mind are the storage.Drive UUIDs; there<br class="gmail_msg"> could also be some ambiguity I'm investigating right now for<br class="gmail_msg"> displayIp/displayNetwork in Graphics devices. In [6] there are various<br class="gmail_msg"> TODO to mark more of those cases. Most likely, few more cases will pop<br class="gmail_msg"> out as I cover all the flows we support.<br class="gmail_msg"> <br class="gmail_msg"> Long story short: it is hard to correctly rebuild the device conf from<br class="gmail_msg"> the XML. This is why in [6] I added the 'meta' argument to from_xml_tree<br class="gmail_msg"> classmethod in [7].<br class="gmail_msg"> <br class="gmail_msg"> 'meta' is supposed to be the device metadata: extra data related to a<br class="gmail_msg"> device which doesn't (yet) fit in the libvirt XML representation.<br class="gmail_msg"> For example, we can store 'displayIp' and 'displayNetwork' here and be<br class="gmail_msg"> done with that: using both per-device metadata and the XML<br class="gmail_msg"> representation of one graphic device, we will have everything we need to<br class="gmail_msg"> properly build one graphics.Graphics device.<br class="gmail_msg"> This example may (hopefully) be bogus, but I'm keeping it because it is<br class="gmail_msg"> one case easy to follow.<br class="gmail_msg"> <br class="gmail_msg"> The device metadata is going to be stored in the vm metadata for the<br class="gmail_msg"> short/mid term future. Even if the per-device metadata idea/RFE is<br class="gmail_msg"> accepted (no answer yet, but we are working on it), we will not have in<br class="gmail_msg"> 7.4, and unlikely in 7.5.<br class="gmail_msg"> <br class="gmail_msg"> As it stands today, I believe there are two open questions:<br class="gmail_msg"> <br class="gmail_msg"> 1. do we need a schema for the metadata section?<br class="gmail_msg"> 2. how do we bind the metadata to the devices? How do we know which<br class="gmail_msg"> metadata belongs to which metadata, if we don't have aliases nor<br class="gmail_msg"> addresses to match? (e.g. very first time the VM is created!)<br class="gmail_msg"> <br class="gmail_msg"> My current stance is the following<br class="gmail_msg"> 1. In general, one schema gives us two benefits: 1.a. we document how<br class="gmail_msg"> the layout of the data should be, including types; 1.b. we can validate<br class="gmail_msg"> the data we receive.<br class="gmail_msg"> So yes, we need a schema, but we don't need a *new* schema. I think we<br class="gmail_msg"> are in good enough shape with the current Vdsm schema: we can just<br class="gmail_msg"> translate the python object layout to a XML layout.<br class="gmail_msg"> <br class="gmail_msg"> One example is probably more explicative. Some actual data may look<br class="gmail_msg"> like, using my pyxmlpickle module:<br class="gmail_msg"> <br class="gmail_msg"> <domain type='kvm' id='5'><br class="gmail_msg"> <name>a0</name><br class="gmail_msg"> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid><br class="gmail_msg"> <metadata xmlns:ovirt-tune="<a moz-do-not-send="true" href="http://ovirt.org/vm/tune/1.0" rel="noreferrer" class="gmail_msg" target="_blank">http://ovirt.org/vm/tune/1.0</a>"<br class="gmail_msg"> </blockquote> <div><br> </div> <div>The url is broken - do we have this?</div> </div> </div> </div> </blockquote> <br> I guess no. This is old code, dating back to 3.6,<i> which I'm taking without changes.</i><br> <br> <blockquote cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com" type="cite"> <div dir="ltr"> <div dir="ltr" class="gmail_msg"> <div class="gmail_quote gmail_msg"> <div> </div> <blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> xmlns:ovirt-vm="<a moz-do-not-send="true" href="http://ovirt.org/vm/1.0" rel="noreferrer" class="gmail_msg" target="_blank">http://ovirt.org/vm/1.0</a>"<br class="gmail_msg"> xmlns:ovirt-instance="<a moz-do-not-send="true" href="http://ovirt.org/vm/instance/1.0" rel="noreferrer" class="gmail_msg" target="_blank">http://ovirt.org/vm/instance/1.0</a>"><br class="gmail_msg"> <ovirt-tune:qos/> </blockquote> <div><br> </div> <div>What do we keep here? why does it need its own namespace?</div> <div> </div> </div> </div> </div> </blockquote> <br> yes, because of how we use the metadata:<br> <br> <a class="moz-txt-link-freetext" href="http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainGetMetadata">http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainGetMetadata</a><br> <a class="moz-txt-link-freetext" href="http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainSetMetadata">http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainSetMetadata</a><br> <br> we use type = <a class="moz-txt-link-freetext" href="http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_METADATA_ELEMENT">http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_METADATA_ELEMENT</a><br> <br> -- I don't think we should change that<br> <br> <blockquote cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com" type="cite"> <div dir="ltr"> <div dir="ltr" class="gmail_msg"> <div class="gmail_quote gmail_msg"> <blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <ovirt-vm:vm/><br class="gmail_msg"> </blockquote> <div><br> </div> <div> <div>What do we keep here? why does it need its own namespace?</div> </div> </div> </div> </div> </blockquote> <br> same as above<br> <br> <blockquote cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com" type="cite"> <div dir="ltr"> <div dir="ltr" class="gmail_msg"> <div class="gmail_quote gmail_msg"> <div><br class="inbox-inbox-Apple-interchange-newline"> </div> <div>Can we merge all namespaces into one generic namespace?</div> </div> </div> </div> </blockquote> <br> Probably, but we have the nice property that code can trivially access one subset of metadata, e.g.<br> only the qos subtree, only the container subtree.<br> Furthermore if we don't change that we are backward compatible at XML level (inbound migrations<br> care about that).<br> <br> <blockquote cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com" type="cite"> <div dir="ltr"> <div dir="ltr" class="gmail_msg"> <div class="gmail_quote gmail_msg"> <div> </div> <blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <ovirt-instance:instance><br class="gmail_msg"> </blockquote> <div class="gmail_msg"><br class="gmail_msg"> </div> </div> </div> <div dir="ltr" class="gmail_msg"> <div class="gmail_quote gmail_msg"> <div class="gmail_msg">What is instance?</div> </div> </div> </div> </blockquote> <br> Gone, it duplicated vm without a good reason<br> <br> <blockquote cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com" type="cite"> <div dir="ltr"> <div dir="ltr" class="gmail_msg"> <div class="gmail_quote gmail_msg"> <div class="gmail_msg"> </div> <blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <ovirt-instance:value type="dict"> </blockquote> <blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <ovirt-instance:item key="devices" type="list"> </blockquote> <blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <ovirt-instance:item index="0" type="dict"></blockquote> <div><br> </div> <div>Isn't index redundant?</div> </div> </div> </div> </blockquote> <br> It is, but now is gone<br> <br> <blockquote cite="mid:CAMRbyysVZLDRCtdDRHL9=OCHWXz3XyWyMzNKkZ1=qg2x7gJLnA@mail.gmail.com" type="cite"> <div dir="ltr"> <div dir="ltr" class="gmail_msg"> <div class="gmail_quote gmail_msg"> <div> </div> <blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <ovirt-instance:item key="device"<br class="gmail_msg"> type="str">vnc</ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item key="specParams" type="dict"><br class="gmail_msg"> <ovirt-instance:item key="displayNetwork"<br class="gmail_msg"> type="str">ovirtmgmt</ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item key="displayIp"<br class="gmail_msg"> type="str">192.168.1.53</ovirt-instance:item><br class="gmail_msg"> </ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item key="type"<br class="gmail_msg"> type="str">graphics</ovirt-instance:item><br class="gmail_msg"> </ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item index="1" type="dict"><br class="gmail_msg"> <ovirt-instance:item key="device"<br class="gmail_msg"> type="str">spice</ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item key="specParams" type="dict"><br class="gmail_msg"> <ovirt-instance:item key="displayNetwork"<br class="gmail_msg"> type="str">ovirtmgmt</ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item key="displayIp"<br class="gmail_msg"> type="str">192.168.1.53</ovirt-instance:item><br class="gmail_msg"> </ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item key="type"<br class="gmail_msg"> type="str">graphics</ovirt-instance:item><br class="gmail_msg"> </ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item index="2" type="dict"><br class="gmail_msg"> <ovirt-instance:item key="poolID"<br class="gmail_msg"> type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item key="imageID"<br class="gmail_msg"> type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item key="domainID"<br class="gmail_msg"> type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item key="device"<br class="gmail_msg"> type="str">disk</ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item key="path"<br class="gmail_msg"> type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item><br class="gmail_msg"> <ovirt-instance:item key="volumeID"<br class="gmail_msg"> type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item><br class="gmail_msg"> </ovirt-instance:item><br class="gmail_msg"> </ovirt-instance:item><br class="gmail_msg"> </ovirt-instance:value><br class="gmail_msg"> </ovirt-instance:instance><br class="gmail_msg"> </metadata><br class="gmail_msg"> <!-- omitted for brevity --><br class="gmail_msg"> </domain><br class="gmail_msg"> </blockquote> <div class="gmail_msg"><br class="gmail_msg"> </div> </div> </div> <div dir="ltr" class="gmail_msg"> <div class="gmail_quote gmail_msg"> <div class="gmail_msg">How about a generic json namespace?<br class="gmail_msg"> </div> <div class="gmail_msg"><br class="gmail_msg"> </div> <div class="gmail_msg">json:</div> <div class="gmail_msg"><br class="gmail_msg"> </div> <div class="gmail_msg">{</div> <div class="gmail_msg"> "devices": [</div> <div class="gmail_msg"> {</div> <div class="gmail_msg"> "foo": "bar"</div> <div class="gmail_msg"> }</div> <div class="gmail_msg"> ]</div> <div class="gmail_msg">}</div> <div class="gmail_msg"><br class="gmail_msg"> </div> <div class="gmail_msg">xml:</div> <div class="gmail_msg"><br> </div> <div class="gmail_msg"><json:object></div> <div class="gmail_msg"> <json:key>devices</json:key></div> <div class="gmail_msg"> <json:list></div> <div class="gmail_msg"> <json:object></div> <div class="gmail_msg"> <json:key>foo</json:key></div> <div class="gmail_msg"> <div class="gmail_msg"> <json:string>bar</json:string></div> </div> <div class="gmail_msg"> </json:object></div> <div class="gmail_msg"> </json:list><br class="gmail_msg"> </div> <div class="gmail_msg"></json:object></div> <div class="gmail_msg"><br> </div> <div class="gmail_msg">We can query and modify this xml using xpath, like this:</div> <div class="gmail_msg"><br> </div> <div class="gmail_msg"> <div class="gmail_msg"> <div class="gmail_msg"> <div class="gmail_msg"> <div class="gmail_msg">>>> r.findall("./metadata/json:object/[json:key='devices']/json:list/json:object/", namespaces={"json": "<a moz-do-not-send="true" href="http://ovirt.org/json/1.0">http://ovirt.org/json/1.0</a>"})</div> <div class="gmail_msg">[<Element '{<a moz-do-not-send="true" href="http://ovirt.org/json/1.0%7Dkey">http://ovirt.org/json/1.0}key</a>' at 0x7fd3386a69d0>, <Element '{<a moz-do-not-send="true" href="http://ovirt.org/json/1.0%7Dstring">http://ovirt.org/json/1.0}string</a>' at 0x7fd3386a6a50>]</div> <div><br> </div> </div> </div> <div>Not sure this format will be easy to modify, needs to tinker more</div> <div>with this.</div> <div><br> </div> </div> </div> </div> <div class="gmail_quote gmail_msg"> <div class="gmail_msg">It will be probably easier to parse the entire metadata, change it,</div> <div class="gmail_msg">and serialize it back.</div> <div class="gmail_msg"><br> </div> <div class="gmail_msg">Maybe using key and type attribute as you suggest makes it simpler to use,</div> <div class="gmail_msg">and we since we convert from xml to python or python to xml, we can</div> <div class="gmail_msg">have a "py" namespace.</div> <div class="gmail_msg"><br> </div> <div class="gmail_msg"> <div class="gmail_msg"><py:item></div> <div class="gmail_msg"> <py:item key="devices" type="list"></div> <div class="gmail_msg"> <py:item type="dict"></div> <div class="gmail_msg"> <py:item key="foo" type="str">bar</py:item></div> <div class="gmail_msg"> </py:item><br> </py:item></div> <div class="gmail_msg"></py:item></div> </div> <div class="gmail_msg"><br> </div> <div class="gmail_msg">With this we can have:</div> <div class="gmail_msg"><br> </div> <div class="gmail_msg"> <div class="gmail_msg">>>> r.findall("./metadata/py:item/py:item[@key='devices']/py:item[1]/py:item[@key='foo']", namespaces={"py": "<a moz-do-not-send="true" href="http://ovirt.org/py/1.0">http://ovirt.org/py/1.0</a>"})[0].text</div> <div class="gmail_msg">'bar'</div> <div><br> </div> </div> <div class="gmail_msg">Something like this can be useful for others as well.</div> </div> </div> </div> </blockquote> <br> Great suggestions, I like the py namespace, but concerns were raised about the solution here being too generic, so I limited it, see<br> <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/74206/11">https://gerrit.ovirt.org/#/c/74206/11</a><br> <br> but I will add my pyxmlpickle module (<a moz-do-not-send="true" href="https://github.com/fromanirh/pyxmlpickle" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/fromanirh/pyxmlpickle</a>) anyway in the future according to your comments.<br> <br> <br> <pre class="moz-signature" cols="72">-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani</pre> </body> </html> --------------15F6CAD0BC7802BD4F39C8F8--
On Wed, Mar 15, 2017 at 2:28 PM Francesco Romani <fromani@redhat.com> wrote:
Hi everyone,
This is both a report of the current state of my Vdsm patches for Engine XML support, and a proposal to how move forward and solve the current open issues.
TL;DR: 1. we can and IMO should reuse the current JSON schema to describe the structure (layout) and the types of the metadata section. 2. we don't need a priori validation of stuff in the metadata section. We will just raise in the creation flow if data is missing, or wrong, according to our schema. 2. we will add *few* items to the metadata section, only thing we can't express clearly-or at all in the libvirt XML. Redundancy and verbosiness will be thus kept at bay 3. I believe [3] is the best tool to do (de)serialize data to the metadata section. Existing tools fits poorly in our very specific use case
Examples below
+++
Long(er) discussion:
I have working code[1][2] to encode any custom, picklable, python object in the metadata section.
We should decide which module will do the actual python<=>XML transformation. Please note that this actually also influences how the data in the medata section look like, so the two things are a bit coupled.
I'm eager to reinvent another wheel, but after initial evaluation I honestly think that my pyxmlpickle[3] is the best tool for the job over the current alternatives: plistlib[4] and xmltodict[5].
I added the initial rationale here: https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
I have completed the initial draft of patches to make it possible to initialize devices from their XML representation [6]. This is the bare minimum we need to support the Engine XML, and we *need* this anyway to unlock the cleanup we planned and I outlined in my google doc.
So we are progressing, but I'd like to speed up things. Those [6] patches are not yet complete, many flows are not covered or tested; but they are good enough to demonstrate that there *are* pieces of information wen need to properly initialize the devices, but we can't easily extract from the XML.
First examples that come to my mind are the storage.Drive UUIDs; there could also be some ambiguity I'm investigating right now for displayIp/displayNetwork in Graphics devices. In [6] there are various TODO to mark more of those cases. Most likely, few more cases will pop out as I cover all the flows we support.
Long story short: it is hard to correctly rebuild the device conf from the XML. This is why in [6] I added the 'meta' argument to from_xml_tree classmethod in [7].
'meta' is supposed to be the device metadata: extra data related to a device which doesn't (yet) fit in the libvirt XML representation. For example, we can store 'displayIp' and 'displayNetwork' here and be done with that: using both per-device metadata and the XML representation of one graphic device, we will have everything we need to properly build one graphics.Graphics device. This example may (hopefully) be bogus, but I'm keeping it because it is one case easy to follow.
The device metadata is going to be stored in the vm metadata for the short/mid term future. Even if the per-device metadata idea/RFE is accepted (no answer yet, but we are working on it), we will not have in 7.4, and unlikely in 7.5.
As it stands today, I believe there are two open questions:
1. do we need a schema for the metadata section? 2. how do we bind the metadata to the devices? How do we know which metadata belongs to which metadata, if we don't have aliases nor addresses to match? (e.g. very first time the VM is created!)
My current stance is the following 1. In general, one schema gives us two benefits: 1.a. we document how the layout of the data should be, including types; 1.b. we can validate the data we receive. So yes, we need a schema, but we don't need a *new* schema. I think we are in good enough shape with the current Vdsm schema: we can just translate the python object layout to a XML layout.
One example is probably more explicative. Some actual data may look like, using my pyxmlpickle module:
<domain type='kvm' id='5'> <name>a0</name> <uuid>ccd945c8-8069-4f31-8471-bbb58e9dd6ea</uuid> <metadata xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0" xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0"> <ovirt-tune:qos/> <ovirt-vm:vm/> <ovirt-instance:instance> <ovirt-instance:value type="dict"> <ovirt-instance:item key="devices" type="list"> <ovirt-instance:item index="0" type="dict"> <ovirt-instance:item key="device" type="str">vnc</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="1" type="dict"> <ovirt-instance:item key="device" type="str">spice</ovirt-instance:item> <ovirt-instance:item key="specParams" type="dict"> <ovirt-instance:item key="displayNetwork" type="str">ovirtmgmt</ovirt-instance:item> <ovirt-instance:item key="displayIp" type="str">192.168.1.53</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item key="type" type="str">graphics</ovirt-instance:item> </ovirt-instance:item> <ovirt-instance:item index="2" type="dict"> <ovirt-instance:item key="poolID" type="str">5890a292-0390-01d2-01ed-00000000029a</ovirt-instance:item> <ovirt-instance:item key="imageID" type="str">66441539-f7ac-4946-8a25-75e422f939d4</ovirt-instance:item> <ovirt-instance:item key="domainID" type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5</ovirt-instance:item> <ovirt-instance:item key="device" type="str">disk</ovirt-instance:item> <ovirt-instance:item key="path"
type="str">/rhev/data-center/5890a292-0390-01d2-01ed-00000000029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> <ovirt-instance:item key="volumeID" type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc</ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:item> </ovirt-instance:value> </ovirt-instance:instance> </metadata> <!-- omitted for brevity --> </domain>
Please note that yes, this is still verbose, but we don't want to add much data here, for most of information the most reliable source will be the domain XML. We will add here only the extra info we can't really fetch from that.
2. I don't think we need explicit validation: we could just raise along the way in the creation flow if we don't find some extra metadata we need. This will also solve the issue that if we reuse the current schema and we omit most of data, we will lack quite a lot of elements marked mandatory.
Once we reached agreement, I will update my
https://docs.google.com/document/d/1eD8KSLwwyo2Sk64MytbmE0wBxxMlpIyEI1GRcHDk... accordingly.
Final note: while device take the lion's share, we will likely need help from the metadata section also to store VM extra info, but all the above discussion also applies here.
+++
[1]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... - uses xmltodict
[2]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vi... ported the 'virt-metadata3' topic to pyxmlpickle
[3] https://github.com/fromanirh/pyxmlpickle Looks good, I like the simple loads() and dumps(). Issues: - index attribute seems unneeded - not sure why we need the value element, seems that everything can be an item - not sure special root element is needed, why not parse the contents of any element, and raise ValueError if there is not type info? - strange name - here some alternative names: - xmlon (xml object notation) - pxon (python xml object notation) If there is no other library that we can use, this seems to be the best direction. Bug I think plist xml format is much nicer - it should be easy to support this format instead of the type attributes: pyxmlpickle: <pyxmlpickle> <value type="dict"> <item key="foo" type="str">bar</item> <item key="list" type="list"> <item index="0" type="int">42</item> <item index="1" type="float">3.14</item> </item> </value> </pyxmlpicle> plist: <plist> <dict> <key>foo</key> <string>bar</string> <key>list</key> <array> <integer>42</integer> <real>3.14</real> </array> </dict> </plist> Less noise, more readable, easier to parse? Since we cannot use plistlib as is, we can make the format nicer and more pythonic: <py> <dict> <key>foo</key> <str>bar</str> <key>list</key> <list> <int>42</int> <float>3.14</float> </list> </dict> </py> Adding namespace will ruin this but if this required by libvirt we have no choice.
We cannot use it as is, since it does not support reading and writing elements, only complete document. We need integration with etree - convert dict to etree element and etree element to dict. Finally, it does not support None. We can borrow code from this module, it is well tested and the author is the same author of simplejson and other nice stuff.
xmltodict converts everything to string, and does not parse the type from the xml, this is very far from the json module. I found also dicttoxml, which does keep the types, but does not support parsing xml. I don't see any value in including this dependency and hacking around it to make it do what we want. There is also: https://pypi.python.org/pypi/xmljson Using the parker module seems nice:
from xml.etree.ElementTree import tostring, fromstring d = {'devices': [{'int': 42, 'none': None, 'float': 3.14, 'string': 'value'}]} from xmljson import parker root = parker.etree(d)[0] tostring(root) '<devices><int>42</int><none>None</none><float>3.14</float><string>value</string></devices>' parker.data(root) OrderedDict([('int', 42), ('none', 'None'), ('float', 3.14), ('string', 'value')])
But the list was converted to dict :-) Seems that this project focus on converting any xml to json while we care about converting json to xml and back, we don't care about any xml. The integration with etree is nice, we should have this. [6]
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:vm... [7] https://gerrit.ovirt.org/#/c/72880/15/lib/vdsm/virt/vmdevices/core.py
-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
participants (4)
-
Francesco Romani -
Martin Polednik -
Michal Skrivanek -
Nir Soffer