Re: [ovirt-devel] Heads-up: moving Libvirt xml creation to the engine

----- Original Message ----- > On Tue, Nov 29, 2016 at 4:30 PM, Arik Hadas <ahadas(a)redhat.com&gt; wrote: > > > > > > ----- Original Message ----- > >> On Tue, Nov 29, 2016 at 2:21 PM, Arik Hadas <ahadas(a)redhat.com&gt; wrote: > >> > > >> > > >> > ----- Original Message ----- > >> >> On Wed, Nov 23, 2016 at 07:59:40AM -0500, Arik Hadas wrote: > >> >> > Hi All, > >> >> > > >> >> > We are working on something that is expected to have a big impact, > >> >> > hence > >> >> > this heads-up. > >> >> > First, we want you to be aware of this change and provide your > >> >> > feedback > >> >> > to > >> >> > make it as good as possible. > >> >> > Second, until the proposed mechanism is fully merged there will be a > >> >> > chase > >> >> > to cover all features unless new features are also implemented with > >> >> > the > >> >> > new mechanism. So please, if you are working on something that > >> >> > adds/changes something in the Libvirt's domain xml, do it with this > >> >> > new > >> >> > mechanism as well (first version would be merged soon). > >> >> > > >> >> > * Goal > >> >> > Creating Libvirt XML in the engine rather than in VDSM. > >> >> > ** Today's flow > >> >> > Engine: VM business entity -> VM properties map > >> >> > VDSM: VM properties map -> Libvirt XML > >> >> > ** Desired flow > >> >> > Engine: VM business entity -> Libvirt XML > >> >> > > >> >> > * Potential Benefits > >> >> > 1. Reduce the number of conversions from 2 to 1, reducing chances for > >> >> > mistakes in the process. > >> >> > 2. Reduce the amount of code in VDSM. > >> >> > 3. Make VM related changes easier - today many of these changes need > >> >> > to > >> >> > be > >> >> > reviewed in 2 projects, this will eliminate the one that tends to > >> >> > take > >> >> > longer. > >> >> > 4. Prevent shortcuts in the form of VDSM-only changes that should be > >> >> > better > >> >> > reflected in the engine. > >> >> > 5. Not to re-generate the XML on each rerun attempt of VM > >> >> > run/migration. > >> >> > 6. Future - not to re-generate the XML on each attempt to auto-start > >> >> > HA > >> >> > VM > >> >> > when using vm-leases (need to make sure we're using the up-to-date VM > >> >> > configuration though). > >> >> > 7. We already found improvements and cleanups that could be made > >> >> > while > >> >> > touching this area (e.g., remove the boot order from devices in the > >> >> > database). > >> >> > > >> >> > * Challenges > >> >> > 1. Not to move host-specific information to the engine. For example, > >> >> > path > >> >> > to storage domain or sockets of channels. > >> >> > The solution is to use place-holders that will be replaced by > >> >> > VDSM. > >> >> > 2. Backward compatibility. > >> >> > 3. The more challenging part is the other direction - that will be > >> >> > the > >> >> > next > >> >> > phase. > >> >> > > >> >> > * Status > >> >> > As a first step, we began with producing the Libvirt XML in the > >> >> > engine > >> >> > by > >> >> > converting the VM properties map to XML in the engine [1] > >> >> > And using the XML that is received as an input in VDSM [2] > >> >> > > >> >> > > >> >> > [1] https://gerrit.ovirt.org/#/c/64473/ > >> >> > [2] https://gerrit.ovirt.org/#/c/65182/ > >> >> > >> >> I should start by saying that I love libvirt's domxml standard. Unlike > >> >> Vdsm's API, it is a real *standard* for defining VMs. In this regards, > >> >> you are suggesting a positive step. > >> >> > >> >> However, Engine is much more complex than Vdsm. It is also our > >> >> single-point-of-failure, and where CPU is the most scarce. I am worried > >> >> that in the foreseeable future it would only make Engine bigger, > >> >> without > >> >> reducing the size and complexity of Vdsm. > >> >> > >> >> Before taking this move, we must map what Vdsm does, because that logic > >> >> would have to be copied into Engine. Few things pop up to mind: > >> >> > >> >> - pci addresses. would Vdsm report back the libvirt-assigned addresses > >> >> in XML format, or would it keep parsing them? > >> > > >> > Ideally, VDSM will report back the devices in XML format. > >> > The engine will then add the unmanaged devices and update the pci > >> > addresses. > >> > Need to put some more thoughts into this, though. > >> > > >> >> > >> >> - hot plug. Device xml should be generated by Engine, much like in the > >> >> vm cteate flow > >> > > >> > Good point, I didn't think of hot plugs - right, they could be changed > >> > as > >> > well later on. > >> > > >> >> > >> >> - network rewiring. Vdsm uses the "dummy bridge" to implement a vNIC > >> >> that is connected no-where. Engine would need to care about what was > >> >> up until now a vdsm-side implementation detail. > >> > > >> > Right, I almost finished to copy the creation of the network interfaces > >> > to > >> > the engine. > >> > This knowledge that you refer to will only be in the module that creates > >> > the XML, it doesn't seem to be much of an issue. > >> > > >> >> > >> >> - storage path. this was mentioned above, but actually, the paths are > >> >> the same on all hosts. We inteded to have an abstraction layer there, > >> >> but we never ever used it. All volumes sit under > >> >> /rhev/data-center/poolID/domainID/imageID/volumeID > >> >> Basically, Engine can hard-code this in the domxml, and no one would > >> >> notice. > >> > >> This is wrong, and engine cannot hard code this or anything else. > >> > >> Engine should can describe only what is knows about disks, only vdsm > >> can add the disk xml. > > > > Of course, the engine will describe only the information it knows, but that > > seems to be most of the disk's related data. > > Let's say that the engine is managed to generate something like: > > <disk type='file' device='disk' snapshot='no'> > > file may be "block" or "network", in glusterfs engine may send "file" and > vdsm > will replace it to "network". Right right, that's what I meant by "and similar structures for the other kind of disks" - basically we'll move the same logic that determines this to the engine in order to produce the right elements. > > > <driver name='qemu' type='qcow2' cache='none'/> > > <source file='$PATH:PDIV or other representations$'/> > > file may need to be "dev" Same, we'll check the diskType in the engine in order to determine the right attributes.

> > > <target dev='hda' bus='ide'/> > > <serial>54-a672-23e5b495a9ea</serial> > > </disk> > > Basically you must inspect the code in vdsm to understand the difference > and what engine can do and cannot. > > Also, how will vdsm fix the xml without the pool, domin, image and volume > ids? > > How do you want to send them if we send xml? Note that the placeholder contains this information ($PATH:PDIV or other representations$)

It will be the engine's responsibility to provide all the data that VDSM needs in order to produce the right replacements within the placeholders > > > > > (and similar structures for the other kind of disks). > > > > Then, VDSM can simply replace the placeholder $PATH...$ with the concrete > > path of the disk. > > For VDSM it would save a lot of code (most of the code in > > vdsm/virt/vmdevices/storage.py I suppose) > > > > What else doesn't the engine know? And couldn't this data be set by VDSM by > > replacing place-holders as in the example above? > > > >> > >> > > >> > But I see that LUN and cinder disks are represented differently (not as > >> > PDIV) - I'll check this. > >> > >> Of course, and even disks using DIV can modified in by vdsm, for > >> example glusterfs > >> using libgfapi. > >> > >> > > >> >> > >> >> - OvS. Recently, we have changed how VMs can be connected to their > >> >> network. It is possible (albeit not recommended yet!) to connect a VM > >> >> to an OvS instead of Linux bridges. This is done without Engine > >> >> really > >> >> caring, or knowing how the domxml is modified. > >> > > >> > Yeah, I saw that. The only complication I see at this point is that for > >> > OvS > >> > we create more elements than only the 'source' element. > >> > I believe that we could use a place-holder that contains the network > >> > name > >> > and replace it with the tags that are needed for SR-IOV, OvS and > >> > ordinary > >> > interfaces, no? > >> > This seems to be the only thing that is difficult to generate on the > >> > engine's side (related to the network interfaces). > >> > > >> >> > >> >> - minor tweaks. exposing a new feature into Engine's UI is hard. Over > >> >> the years, few tweaks have been pushed as custom properties. > >> >> there are not many (I see now only sndbuf, queues, viodiskcache, > >> >> vhost) but the implementation should make sure they are not > >> >> forgotten. > >> > > >> > Sure. > >> > > >> >> > >> >> Maybe, Vdsm should consider Engine's domxml only as a "recomendation" > >> >> and modify it based on its hooks and custom properties. This can > >> >> surprise Engine, a defies the pupose of having xml-building logic moved > >> >> away from Vdsm. > >> >> > >> > _______________________________________________ > >> > Devel mailing list > >> > Devel(a)ovirt.org > >> > http://lists.ovirt.org/mailman/listinfo/devel > >> >