<html><body>
<p><tt>arch-bounces@ovirt.org wrote on 12/06/2011 05:18:56 AM:<br>
<br>
> From: "Geoff O'Callaghan" <geoffocallaghan@gmail.com></tt><br>
<tt>> To: arch@ovirt.org, node-devel <node-devel@ovirt.org>, </tt><br>
<tt>> Date: 12/06/2011 05:19 AM</tt><br>
<tt>> Subject: Re: oVirt Node designs for stateless operation and 3rd party plugins</tt><br>
<tt>> Sent by: arch-bounces@ovirt.org</tt><br>
<tt>> <br>
> On Thu, 2011-12-01 at 09:32 -0500, Perry Myers wrote: <br>
> > the Node development team has been trying to write up rough requirements<br>
> > around the stateless and plugins concepts. And also some working high<br>
> > level design.<br>
> > <br>
> > They can be reviewed on these two wiki pages:<br>
> > <br>
> > <a href="http://ovirt.org/wiki/Node_plugins">http://ovirt.org/wiki/Node_plugins</a><br>
> > <a href="http://ovirt.org/wiki/Node_Stateless">http://ovirt.org/wiki/Node_Stateless</a><br>
> > <br>
> > Since the plugin model and the stateless model affect more than just the<br>
> > oVirt Node itself, we definitely would like to get input from other<br>
> > teams on the oVirt project.<br>
> > <br>
> > Please add comments here or directly to the wiki.<br>
> > <br>
> <br>
> Hi There<br>
> <br>
> I work for a *large* organisation, I have issues with the goal of a<br>
> stateless design.<br>
> <br>
> * Being able to install without a local disk<br>
> <br>
> I don't see this as a compelling reason for doing anything. In fact,<br>
> in many cases for other nameless hypervisors we use local disk as a<br>
> source for logging / dumps etc.<br>
> </tt><br>
<br>
<tt>The ability to operate without disk does not imply the inability to take advantage of local disk when available and appropriate. </tt><br>
<tt><br>
> I think the goal for stateless should be instead be configuration<br>
> neutral. ie. if the node is destroyed the configuration can be<br>
> re-deployed without issue.</tt><br>
<tt>> <br>
> The other issue is that the node should continue to be re-bootable even<br>
> if the configuration server is unavailable, which is a reason for having<br>
> the configuration on a local disk or a san attached LUN. This should<br>
> apply to the entire operational environment - if the engine is<br>
> unavailable during a restart I should continue working the way I was<br>
> configured to do so - that implies state is retained. It needs to be<br>
> easily refreshable :-)</tt><br>
<br>
<tt>One, I would think 'the' configuration server would be a bit more robust than implied here. I think certain examples to date have been fairly poor, but a respectable architecture would work out well.</tt><br>
<br>
<tt>I think there is some opportunity for risk if a 'central' authority can repair configuration automagically (including restarting VMs on another node) *and* the downed node can also independently operate as last expected. There is a risk for split-brain scenario. Maybe there is a large degree of confidence in fencing and storage locking mitigating this sufficiently. </tt><br>
<tt><br>
> <br>
> The configuration bundle should be refreshable from a configuration<br>
> server (part of the engine) and that could either be just configuration<br>
> or agents or even s/w images - all would be preferred and it's pretty<br>
> simple conceptually to have an active/backup image on local disk concept<br>
> to allow easy rollbacks etc. Yes all this , except for the logging /<br>
> swap could be in a usb key.<br>
> <br>
> The bundle should all be pushed via a SSL encrypted RESTful api using<br>
> known non-priv credentials, preferably with rotating passwords or some<br>
> cert based approach. The server should also know who previously<br>
> managed it to reduce hostile attempts to change ownership of the node.<br>
> <br>
> * DHCP and PXE booting<br>
> <br>
> Many corporate security policies prohibit the use of DHCP or PXE booting<br>
> servers for production environments. I don't see it as a big issue to<br>
> boot an install image and be a good woodpecker and hit enter a few times<br>
> and configure a management IP address. It should be possible to script<br>
> the complete configuration / addition of the node after that step. I<br>
> see the initial install as a trivial part of the complete node<br>
> lifecycle.</tt><br>
<br>
<tt>I find it a big issue when configuring a few thousand servers. Of course, taking a bigger picture here, we can make unattended stateless image booting a secure prospect even over PXE. Either way from the inside, tolerating an ISO boot generally covers the same concerns as a PXE boot anyway.</tt><br>
<tt><br>
> <br>
> * DNS SRV records<br>
> <br>
> Sorry, I hate the idea. Large corporates have so many different teams<br>
> doing little things that adding this in as a requirement simply adds<br>
> delays to the deployments and opportunities for misconfiguration.</tt><br>
<br>
<tt>A capability does not imply requirement. Active Directory makes use of this as can Kerberos, so it's not exactly without precedent. I'd expect /proc/cmdline to be the typical path however. You could add SLP, mDNS, or roll-your-own multicast discovery to the list though.</tt><br>
<tt><br>
> <br>
> Having the node image and config on local disk (or usb) avoids this<br>
> requirement as the node knows who manages it. A complete rebuild could<br>
> occur and the configuration reloaded once added back into the engine.</tt><br>
<br>
<tt>Once you sanely provide for the automated 'rebuild' case, you've solved the problem for arbitrary boots anyway.</tt><br>
<tt><br>
> <br>
> * Previously configured state<br>
> <br>
> Yes, the node should remember the previous operational state if it<br>
> can't talk to the engine. This is not a bad thing. </tt><br>
<br>
<tt>Depends on what the 'weak' point is. If the chances are your inability to talk to the configuration infrastructure favor a split-brain scenario, restoring last state could be a bad thing.</tt><br>
<tt><br>
> <br>
> * Configuration server<br>
> <br>
> This should be part of the engine. It should know the complete<br>
> configuration of a node, right down to hypervisor 'firmware' image. The<br>
> process should be 2-way. An admin should be able to 'pull' the<br>
> image/config from an operational and accessible node and new<br>
> configurations/images should be pushable to it.<br>
> <br>
> I really don't think this needs to be a separate server to the engine.<br>
> <br>
> * New bundle deployments / Upgrades<br>
> <br>
> The engine should keep track of what images are on a node. If a new<br>
> config / image is to be deployed then for example, the node would be<br>
> tagged with the new image. If the node was online, an alternate image<br>
> would be pushed, vm's migrated to an alternate node and the node<br>
> restarted implementing the new image when requested.<br>
> <br>
> If the node was offline at the time the new image was configured in the<br>
> engine or if the node was built say with an old image then when it<br>
> connects to the engine the image would be refreshed and the node<br>
> recycled.</tt><br>
<br>
<tt>The nice thing about stateless, it's a lot more straightforward to work through these workflows.</tt><br>
<tt><br>
> <br>
> * Swap<br>
> <br>
> Local disk swap is likely to be required. Overcommit is common and SSD<br>
> local disk is something that is quite useful :-)</tt><br>
<br>
<tt>Flexibility is good, support swap and non-swap cases. In fact, stateless is a good match for this, you can use the local disk for things like distributed filesystems, logging and swap, the OS image isn't bound to disk and neither is the configuration.</tt><br>
<tt><br>
> <br>
> So in summary, I prefer to think that the target should be<br>
> configuration neutrality or even just plain old distributed<br>
> configuration from a central source rather than completely stateless.<br>
> The goal should be toleration of complete destruction of a node image<br>
> and configuration and a simple process to re-add it and automatically<br>
> re-apply the configuration/sw image.</tt><br>
<br>
<tt>I think a bullet-proof configuration infrastructure that makes stateless just as good as stateful handles the failure cases a lot more smoothly. Emphasis on bulletproof.</tt><br>
<tt><br>
> <br>
> Just some thoughts for discussion / abuse ;-)<br>
> <br>
> Tks<br>
> Geoff<br>
> <br>
> > Cheers,<br>
> > <br>
> > Perry<br>
> > _______________________________________________<br>
> > Arch mailing list<br>
> > Arch@ovirt.org<br>
> > <a href="http://lists.ovirt.org/mailman/listinfo/arch">http://lists.ovirt.org/mailman/listinfo/arch</a><br>
> <br>
> <br>
> _______________________________________________<br>
> Arch mailing list<br>
> Arch@ovirt.org<br>
> <a href="http://lists.ovirt.org/mailman/listinfo/arch">http://lists.ovirt.org/mailman/listinfo/arch</a><br>
> <br>
</tt></body></html>