[Engine-devel] Support for stateless nodes
There has been a lot of interest in being able to run stateless Nodes with ovirt-engine. ovirt-node has designed a way [1] to achieve this on the node side, but we need input from the engine and vdsm teams to see if we're missing some requirement or if there needs to be changes on the engine/vdsm side to achieve this. As it currently stands, every time you reboot an ovirt-node that is stateless, it would require manually removing the host in engine, then re-registering/approving it again in engine. Any thoughts, concerns, input on how to solve this? Thanks Mike [1] http://ovirt.org/wiki/Node_Stateless
On 22/02/12 16:57, Mike Burns wrote:
There has been a lot of interest in being able to run stateless Nodes with ovirt-engine. ovirt-node has designed a way [1] to achieve this on the node side, but we need input from the engine and vdsm teams to see if we're missing some requirement or if there needs to be changes on the engine/vdsm side to achieve this.
As it currently stands, every time you reboot an ovirt-node that is stateless, it would require manually removing the host in engine, then re-registering/approving it again in engine.
Any thoughts, concerns, input on how to solve this?
Thanks
Mike
Some points need to be considered; - Installation issues * Just stating the obvious, which is users need to remove-add the host on every reboot. This will not make this feature a lovable one from user's point of view. * During initial boot, vdsm-reg configures the networking and creates a management network bridge. This is a very delicate process which may fail due to networking issues such as resolution, routing, etc. So re-doing this on every boot increases the chances of loosing a node due to network problems. * CA pollution; generating a certificate on each reboot for each node will create a huge number of certificates in the engine side, which eventually may damage the CA. (Unsure if there's a limitation to certificates number, but having hundreds of junk cert's can't be good). * Today there's a supported flow that for nodes with password, the user is allowed to use the "add host" scenario. For stateless, it means re-configuring a password on every boot... - Other issues * Local storage; so far we were able to define a local storage in ovirt node. Stateless will block this ability. * Node upgrade; currently it's possible to upgrade a node from the engine. In stateless it will error, since no where to d/l the iso file to. * Collecting information; core dumps and logging may not be available due to lack of space? Or will it cause kernel panic if all space is consumed? -- /d "Hi, my name is Any Key. Please don't hit me!"
On Wed, 2012-02-22 at 17:33 +0200, Doron Fediuck wrote:
On 22/02/12 16:57, Mike Burns wrote:
There has been a lot of interest in being able to run stateless Nodes with ovirt-engine. ovirt-node has designed a way [1] to achieve this on the node side, but we need input from the engine and vdsm teams to see if we're missing some requirement or if there needs to be changes on the engine/vdsm side to achieve this.
As it currently stands, every time you reboot an ovirt-node that is stateless, it would require manually removing the host in engine, then re-registering/approving it again in engine.
Any thoughts, concerns, input on how to solve this?
Thanks
Mike
Some points need to be considered;
- Installation issues
* Just stating the obvious, which is users need to remove-add the host on every reboot. This will not make this feature a lovable one from user's point of view.
Yes, this is something that will cause this to be a non-starter. We'd need to change something in the engine/vdsm to make it smoother. Perhaps, a flag in engine on the host saying that it's stateless. Then if a host comes up with the same information, but no certs, etc, it would validate some other embedded key (TPM, key embedded in the node itself), and auto-approve it to be the same state as the previous boot
* During initial boot, vdsm-reg configures the networking and creates a management network bridge. This is a very delicate process which may fail due to networking issues such as resolution, routing, etc. So re-doing this on every boot increases the chances of loosing a node due to network problems.
vdsm-reg runs on *every* boot anyway and renames the bridge. This is something that was debated previously, but it was decided to re-run it every boot.
* CA pollution; generating a certificate on each reboot for each node will create a huge number of certificates in the engine side, which eventually may damage the CA. (Unsure if there's a limitation to certificates number, but having hundreds of junk cert's can't be good).
We could have vdsm/engine store the certs on the engine side, and on boot, after validating the host (however that is done), it will load the certs onto the node machine.
* Today there's a supported flow that for nodes with password, the user is allowed to use the "add host" scenario. For stateless, it means re-configuring a password on every boot...
Stateless is really targeted for a PXE environment. There is a supported kernel param that can be set that will set this password. Also, if we follow the design mentioned ^^, then it's not an issue since the host will auto-approve itself when it connects
- Other issues
* Local storage; so far we were able to define a local storage in ovirt node. Stateless will block this ability.
Yes, this would be unavailable if you're running stateless. I think that's a fine tradeoff since people want the host to be diskless.
* Node upgrade; currently it's possible to upgrade a node from the engine. In stateless it will error, since no where to d/l the iso file to.
Upgrade is handled easily by rebooting the host after updating the pxe server
* Collecting information; core dumps and logging may not be available due to lack of space? Or will it cause kernel panic if all space is consumed?
A valid concern, but a stateless environment would likely have collectd/rsyslog/netconsole servers running elsewhere that will collect the logs. kdumps can be configured to dump remotely as well.
Another concern raised is swap and overcommit. First version would likely disable swap completely. This would disable overcommit as well. Future versions could enable a local disk to be used completely for swap, but that is another tradeoff that people would need to evaluate when choosing between stateless and stateful installs. Mike
On 22/02/12 17:58, Mike Burns wrote:
On Wed, 2012-02-22 at 17:33 +0200, Doron Fediuck wrote:
On 22/02/12 16:57, Mike Burns wrote:
There has been a lot of interest in being able to run stateless Nodes with ovirt-engine. ovirt-node has designed a way [1] to achieve this on the node side, but we need input from the engine and vdsm teams to see if we're missing some requirement or if there needs to be changes on the engine/vdsm side to achieve this.
As it currently stands, every time you reboot an ovirt-node that is stateless, it would require manually removing the host in engine, then re-registering/approving it again in engine.
Any thoughts, concerns, input on how to solve this?
Thanks
Mike
Some points need to be considered;
- Installation issues
* Just stating the obvious, which is users need to remove-add the host on every reboot. This will not make this feature a lovable one from user's point of view.
Yes, this is something that will cause this to be a non-starter. We'd need to change something in the engine/vdsm to make it smoother. Perhaps, a flag in engine on the host saying that it's stateless. Then if a host comes up with the same information, but no certs, etc, it would validate some other embedded key (TPM, key embedded in the node itself), and auto-approve it to be the same state as the previous boot
This will require some thinking.
* During initial boot, vdsm-reg configures the networking and creates a management network bridge. This is a very delicate process which may fail due to networking issues such as resolution, routing, etc. So re-doing this on every boot increases the chances of loosing a node due to network problems.
vdsm-reg runs on *every* boot anyway and renames the bridge. This is something that was debated previously, but it was decided to re-run it every boot.
Close, but not exactly; vdsm-reg will run on every boot, but if the relevant bridge is found, then networking is unchanged.
* CA pollution; generating a certificate on each reboot for each node will create a huge number of certificates in the engine side, which eventually may damage the CA. (Unsure if there's a limitation to certificates number, but having hundreds of junk cert's can't be good).
We could have vdsm/engine store the certs on the engine side, and on boot, after validating the host (however that is done), it will load the certs onto the node machine.
This is a security issue, since the key pair should be generated on the node. This will lead us back to your TPM suggestion, but (although I like it, ) will cause us to be tpm-dependent, not to mention a non-trivial implementation.
* Today there's a supported flow that for nodes with password, the user is allowed to use the "add host" scenario. For stateless, it means re-configuring a password on every boot...
Stateless is really targeted for a PXE environment. There is a supported kernel param that can be set that will set this password. Also, if we follow the design mentioned ^^, then it's not an issue since the host will auto-approve itself when it connects
- Other issues
* Local storage; so far we were able to define a local storage in ovirt node. Stateless will block this ability.
Yes, this would be unavailable if you're running stateless. I think that's a fine tradeoff since people want the host to be diskless.
* Node upgrade; currently it's possible to upgrade a node from the engine. In stateless it will error, since no where to d/l the iso file to.
Upgrade is handled easily by rebooting the host after updating the pxe server
* Collecting information; core dumps and logging may not be available due to lack of space? Or will it cause kernel panic if all space is consumed?
A valid concern, but a stateless environment would likely have collectd/rsyslog/netconsole servers running elsewhere that will collect the logs. kdumps can be configured to dump remotely as well. This will also need some work on the vdsm side.
Another concern raised is swap and overcommit. First version would likely disable swap completely. This would disable overcommit as well. Future versions could enable a local disk to be used completely for swap, but that is another tradeoff that people would need to evaluate when choosing between stateless and stateful installs. Indeed so- completely forgot about swap...
Mike
-- /d “Funny,” he intoned funereally, “how just when you think life can't possibly get any worse it suddenly does.” --Douglas Adams, The Hitchhiker's Guide to the Galaxy
* CA pollution; generating a certificate on each reboot for each node will create a huge number of certificates in the engine side, which eventually may damage the CA. (Unsure if there's a limitation to certificates number, but having hundreds of junk cert's can't be good).
We could have vdsm/engine store the certs on the engine side, and on boot, after validating the host (however that is done), it will load the certs onto the node machine.
This is a security issue, since the key pair should be generated on the node. This will lead us back to your TPM suggestion, but (although I like it, ) will cause us to be tpm-dependent, not to mention a non-trivial implementation.
Not necessarily 1. generate cert on oVirt Node 2. generate symmetric key and embed in TPM or use embedded symmetric key (for secured network model) 3. encrypt certs w/ symmetric key 4. push encryted cert to oVirt Engine On reboot 1. download encrypted cert from OE 2. use either embedded symmetric key or retrieve TPM based symmetric key and use to decrypt cert So no dependency on TPM, but the security is definitely much better if you have it. Use cases like this are one of the fundamental reasons why TPM exists :)
On 22/02/12 18:21, Perry Myers wrote:
* CA pollution; generating a certificate on each reboot for each node will create a huge number of certificates in the engine side, which eventually may damage the CA. (Unsure if there's a limitation to certificates number, but having hundreds of junk cert's can't be good).
We could have vdsm/engine store the certs on the engine side, and on boot, after validating the host (however that is done), it will load the certs onto the node machine.
This is a security issue, since the key pair should be generated on the node. This will lead us back to your TPM suggestion, but (although I like it, ) will cause us to be tpm-dependent, not to mention a non-trivial implementation.
Not necessarily
1. generate cert on oVirt Node 2. generate symmetric key and embed in TPM or use embedded symmetric key (for secured network model) IIUC in this step you're using TPM. What if there is no TPM (at all)?
3. encrypt certs w/ symmetric key 4. push encryted cert to oVirt Engine
On reboot
1. download encrypted cert from OE 2. use either embedded symmetric key or retrieve TPM based symmetric key and use to decrypt cert
So no dependency on TPM, but the security is definitely much better if you have it. Use cases like this are one of the fundamental reasons why TPM exists :) _______________________________________________ node-devel mailing list node-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/node-devel
-- /d "Ford," he said, "you're turning into a penguin. Stop it." --Douglas Adams, The Hitchhiker's Guide to the Galaxy
On 02/22/2012 11:40 AM, Doron Fediuck wrote:
On 22/02/12 18:21, Perry Myers wrote:
* CA pollution; generating a certificate on each reboot for each node will create a huge number of certificates in the engine side, which eventually may damage the CA. (Unsure if there's a limitation to certificates number, but having hundreds of junk cert's can't be good).
We could have vdsm/engine store the certs on the engine side, and on boot, after validating the host (however that is done), it will load the certs onto the node machine.
This is a security issue, since the key pair should be generated on the node. This will lead us back to your TPM suggestion, but (although I like it, ) will cause us to be tpm-dependent, not to mention a non-trivial implementation.
Not necessarily
1. generate cert on oVirt Node 2. generate symmetric key and embed in TPM or use embedded symmetric key (for secured network model) IIUC in this step you're using TPM. What if there is no TPM (at all)?
That statement had an 'or' in it. Either you use TPM with a self generated key 'or' you use a key that is preembedded in the image on either a node by node basis or per site.
3. encrypt certs w/ symmetric key 4. push encryted cert to oVirt Engine
On reboot
1. download encrypted cert from OE 2. use either embedded symmetric key or retrieve TPM based symmetric key and use to decrypt cert
So no dependency on TPM, but the security is definitely much better if you have it. Use cases like this are one of the fundamental reasons why TPM exists :) _______________________________________________ node-devel mailing list node-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/node-devel
On Wed, 2012-02-22 at 18:10 +0200, Doron Fediuck wrote:
On 22/02/12 17:58, Mike Burns wrote:
On Wed, 2012-02-22 at 17:33 +0200, Doron Fediuck wrote:
On 22/02/12 16:57, Mike Burns wrote:
<snip>
vdsm-reg runs on *every* boot anyway and renames the bridge. This is something that was debated previously, but it was decided to re-run it every boot.
Close, but not exactly; vdsm-reg will run on every boot, but if the relevant bridge is found, then networking is unchanged.
Yes, that's true, but vdsm-reg doesn't persist the changes it makes. So on the next boot, it will never find the management bridge it's looking for. So while the condition is there to skip it, it will actually never find the bridge and will run the rename every boot.
On 22/02/12 18:25, Mike Burns wrote:
On Wed, 2012-02-22 at 18:10 +0200, Doron Fediuck wrote:
On 22/02/12 17:58, Mike Burns wrote:
On Wed, 2012-02-22 at 17:33 +0200, Doron Fediuck wrote:
On 22/02/12 16:57, Mike Burns wrote:
<snip>
vdsm-reg runs on *every* boot anyway and renames the bridge. This is something that was debated previously, but it was decided to re-run it every boot.
Close, but not exactly; vdsm-reg will run on every boot, but if the relevant bridge is found, then networking is unchanged.
Yes, that's true, but vdsm-reg doesn't persist the changes it makes. So on the next boot, it will never find the management bridge it's looking for. So while the condition is there to skip it, it will actually never find the bridge and will run the rename every boot.
Sounds like a bug to me. Last time I handled it, network configuration was persisted. You can see in line 102 here: http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm_reg/vdsm-reg-setup.i... -- /d "Who's General Failure and why's he reading my disk?"
On 22/02/12 17:33, Doron Fediuck wrote:
On 22/02/12 16:57, Mike Burns wrote:
There has been a lot of interest in being able to run stateless Nodes with ovirt-engine. ovirt-node has designed a way [1] to achieve this on the node side, but we need input from the engine and vdsm teams to see if we're missing some requirement or if there needs to be changes on the engine/vdsm side to achieve this.
As it currently stands, every time you reboot an ovirt-node that is stateless, it would require manually removing the host in engine, then re-registering/approving it again in engine.
Any thoughts, concerns, input on how to solve this?
Thanks
Mike
Some points need to be considered;
- Installation issues
* Just stating the obvious, which is users need to remove-add the host on every reboot. This will not make this feature a lovable one from user's point of view.
* During initial boot, vdsm-reg configures the networking and creates a management network bridge. This is a very delicate process which may fail due to networking issues such as resolution, routing, etc. So re-doing this on every boot increases the chances of loosing a node due to network problems.
* CA pollution; generating a certificate on each reboot for each node will create a huge number of certificates in the engine side, which eventually may damage the CA. (Unsure if there's a limitation to certificates number, but having hundreds of junk cert's can't be good).
* Today there's a supported flow that for nodes with password, the user is allowed to use the "add host" scenario. For stateless, it means re-configuring a password on every boot...
- Other issues
* Local storage; so far we were able to define a local storage in ovirt node. Stateless will block this ability.
* Node upgrade; currently it's possible to upgrade a node from the engine. In stateless it will error, since no where to d/l the iso file to.
* Collecting information; core dumps and logging may not be available due to lack of space? Or will it cause kernel panic if all space is consumed?
One more question / thing to consider; Currently when you manually install a node, you need to configure the management-server's address. Will I need to re-do it on every boot of a stateless node? -- /d "Air conditioned environment - Do NOT open Windows!"
On Wed, 2012-02-22 at 17:59 +0200, Doron Fediuck wrote:
On 22/02/12 17:33, Doron Fediuck wrote:
On 22/02/12 16:57, Mike Burns wrote:
There has been a lot of interest in being able to run stateless Nodes with ovirt-engine. ovirt-node has designed a way [1] to achieve this on the node side, but we need input from the engine and vdsm teams to see if we're missing some requirement or if there needs to be changes on the engine/vdsm side to achieve this.
As it currently stands, every time you reboot an ovirt-node that is stateless, it would require manually removing the host in engine, then re-registering/approving it again in engine.
Any thoughts, concerns, input on how to solve this?
Thanks
Mike
Some points need to be considered;
- Installation issues
* Just stating the obvious, which is users need to remove-add the host on every reboot. This will not make this feature a lovable one from user's point of view.
* During initial boot, vdsm-reg configures the networking and creates a management network bridge. This is a very delicate process which may fail due to networking issues such as resolution, routing, etc. So re-doing this on every boot increases the chances of loosing a node due to network problems.
* CA pollution; generating a certificate on each reboot for each node will create a huge number of certificates in the engine side, which eventually may damage the CA. (Unsure if there's a limitation to certificates number, but having hundreds of junk cert's can't be good).
* Today there's a supported flow that for nodes with password, the user is allowed to use the "add host" scenario. For stateless, it means re-configuring a password on every boot...
- Other issues
* Local storage; so far we were able to define a local storage in ovirt node. Stateless will block this ability.
* Node upgrade; currently it's possible to upgrade a node from the engine. In stateless it will error, since no where to d/l the iso file to.
* Collecting information; core dumps and logging may not be available due to lack of space? Or will it cause kernel panic if all space is consumed?
One more question / thing to consider; Currently when you manually install a node, you need to configure the management-server's address. Will I need to re-do it on every boot of a stateless node?
As answered in the other response, there are kernel command line parameters to set the management_server. Since this will likely be in a pxe environment, setting the pxe profile to include management_server=<engine_url> should be fine. Another solution could be to setup a specific DNS SRV record that points to the ovirt-engine and have node automatically query that for the location. Mike
On 22/02/12 18:08, Mike Burns wrote:
On Wed, 2012-02-22 at 17:59 +0200, Doron Fediuck wrote:
On 22/02/12 17:33, Doron Fediuck wrote:
On 22/02/12 16:57, Mike Burns wrote:
There has been a lot of interest in being able to run stateless Nodes with ovirt-engine. ovirt-node has designed a way [1] to achieve this on the node side, but we need input from the engine and vdsm teams to see if we're missing some requirement or if there needs to be changes on the engine/vdsm side to achieve this.
As it currently stands, every time you reboot an ovirt-node that is stateless, it would require manually removing the host in engine, then re-registering/approving it again in engine.
Any thoughts, concerns, input on how to solve this?
Thanks
Mike
Some points need to be considered;
- Installation issues
* Just stating the obvious, which is users need to remove-add the host on every reboot. This will not make this feature a lovable one from user's point of view.
* During initial boot, vdsm-reg configures the networking and creates a management network bridge. This is a very delicate process which may fail due to networking issues such as resolution, routing, etc. So re-doing this on every boot increases the chances of loosing a node due to network problems.
* CA pollution; generating a certificate on each reboot for each node will create a huge number of certificates in the engine side, which eventually may damage the CA. (Unsure if there's a limitation to certificates number, but having hundreds of junk cert's can't be good).
* Today there's a supported flow that for nodes with password, the user is allowed to use the "add host" scenario. For stateless, it means re-configuring a password on every boot...
- Other issues
* Local storage; so far we were able to define a local storage in ovirt node. Stateless will block this ability.
* Node upgrade; currently it's possible to upgrade a node from the engine. In stateless it will error, since no where to d/l the iso file to.
* Collecting information; core dumps and logging may not be available due to lack of space? Or will it cause kernel panic if all space is consumed?
One more question / thing to consider; Currently when you manually install a node, you need to configure the management-server's address. Will I need to re-do it on every boot of a stateless node?
As answered in the other response, there are kernel command line parameters to set the management_server. Since this will likely be in a pxe environment, setting the pxe profile to include management_server=<engine_url> should be fine.
I agree it's a valid solution as long as you assume this is relevant for PXE only use case.
Another solution could be to setup a specific DNS SRV record that points to the ovirt-engine and have node automatically query that for the location. This was discussed in the past and for some reason not implemented.
Mike
_______________________________________________ node-devel mailing list node-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/node-devel
-- /d "Ford," he said, "you're turning into a penguin. Stop it." --Douglas Adams, The Hitchhiker's Guide to the Galaxy
As answered in the other response, there are kernel command line parameters to set the management_server. Since this will likely be in a pxe environment, setting the pxe profile to include management_server=<engine_url> should be fine.
I agree it's a valid solution as long as you assume this is relevant for PXE only use case.
Not necessarily... Take the ISO/USB Stick and you can embed the kargs into the ISO/USB itself so that it always boots with that mgmt server arg This actually also enables use of 'stateless' combined with static IP addressing as well. As you can create a USB Stick and embed the kargs for the NIC configuration, rsyslog config, etc, etc.
Another solution could be to setup a specific DNS SRV record that points to the ovirt-engine and have node automatically query that for the location. This was discussed in the past and for some reason not implemented.
Concerns about security, iirc. Assumption that someone could hijack the DNS SRV record and provide a man-in-the-middle oVirt Engine server. If you're paranoid about security, don't use DNS SRV of course, instead use hardcoded kargs as described above. But for some DNS SRV might be an ok option
Perry Myers píše v St 22. 02. 2012 v 11:54 -0500:
As answered in the other response, there are kernel command line parameters to set the management_server. Since this will likely be in a pxe environment, setting the pxe profile to include management_server=<engine_url> should be fine.
I agree it's a valid solution as long as you assume this is relevant for PXE only use case.
Not necessarily...
Take the ISO/USB Stick and you can embed the kargs into the ISO/USB itself so that it always boots with that mgmt server arg
This actually also enables use of 'stateless' combined with static IP addressing as well. As you can create a USB Stick and embed the kargs for the NIC configuration, rsyslog config, etc, etc.
Another solution could be to setup a specific DNS SRV record that points to the ovirt-engine and have node automatically query that for the location. This was discussed in the past and for some reason not implemented.
Concerns about security, iirc. Assumption that someone could hijack the DNS SRV record and provide a man-in-the-middle oVirt Engine server.
What about DNSSEC validation for DNS records in node? David
If you're paranoid about security, don't use DNS SRV of course, instead use hardcoded kargs as described above. But for some DNS SRV might be an ok option _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
-- David Jaša, RHCE SPICE QE based in Brno GPG Key: 22C33E24 Fingerprint: 513A 060B D1B4 2A72 7F0D 0278 B125 CD00 22C3 3E24
----- Original Message -----
Perry Myers píše v St 22. 02. 2012 v 11:54 -0500:
As answered in the other response, there are kernel command line parameters to set the management_server. Since this will likely be in a pxe environment, setting the pxe profile to include management_server=<engine_url> should be fine.
I agree it's a valid solution as long as you assume this is relevant for PXE only use case.
Not necessarily...
Take the ISO/USB Stick and you can embed the kargs into the ISO/USB itself so that it always boots with that mgmt server arg
This actually also enables use of 'stateless' combined with static IP addressing as well. As you can create a USB Stick and embed the kargs for the NIC configuration, rsyslog config, etc, etc.
Another solution could be to setup a specific DNS SRV record that points to the ovirt-engine and have node automatically query that for the location. This was discussed in the past and for some reason not implemented.
Concerns about security, iirc. Assumption that someone could hijack the DNS SRV record and provide a man-in-the-middle oVirt Engine server.
What about DNSSEC validation for DNS records in node?
This will require more than just changes to the registration process and it's quite difficult to track the required changes here on email. Let's setup a call to discuss this and try to capture the list of issues we already know about (I'm sure we'll discover more once we actually try to do this). To play devil's advocate though, I know there is interest, but I really don't understand the incentive. What is the *problem* you're trying to solve here (stateless is a solution)
David
If you're paranoid about security, don't use DNS SRV of course, instead use hardcoded kargs as described above. But for some DNS SRV might be an ok option _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
--
David Jaša, RHCE
SPICE QE based in Brno GPG Key: 22C33E24 Fingerprint: 513A 060B D1B4 2A72 7F0D 0278 B125 CD00 22C3 3E24
_______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On 02/27/2012 05:06 AM, Ayal Baron wrote:
----- Original Message -----
Perry Myers píše v St 22. 02. 2012 v 11:54 -0500:
As answered in the other response, there are kernel command line parameters to set the management_server. Since this will likely be in a pxe environment, setting the pxe profile to include management_server=<engine_url> should be fine.
I agree it's a valid solution as long as you assume this is relevant for PXE only use case.
Not necessarily...
Take the ISO/USB Stick and you can embed the kargs into the ISO/USB itself so that it always boots with that mgmt server arg
This actually also enables use of 'stateless' combined with static IP addressing as well. As you can create a USB Stick and embed the kargs for the NIC configuration, rsyslog config, etc, etc.
Another solution could be to setup a specific DNS SRV record that points to the ovirt-engine and have node automatically query that for the location. This was discussed in the past and for some reason not implemented.
Concerns about security, iirc. Assumption that someone could hijack the DNS SRV record and provide a man-in-the-middle oVirt Engine server.
What about DNSSEC validation for DNS records in node?
This will require more than just changes to the registration process and it's quite difficult to track the required changes here on email. Let's setup a call to discuss this and try to capture the list of issues we already know about (I'm sure we'll discover more once we actually try to do this).
I'm fine with setting up a call as long as we publish the dial in info on list so that folks in the community can join in if they are interested. Also, stateless design has been a topic on the oVirt Node weekly IRC meeting for the last few weeks as we flesh out design issues, etc. I'd be happy for folks from other teams to join in there
To play devil's advocate though, I know there is interest, but I really don't understand the incentive. What is the *problem* you're trying to solve here (stateless is a solution)
The primary motivation is diskless nodes. Being able to purchase mass quantities of servers and not needing to put any sort of storage in them. Since disks have much lower MTBF than other components (aside from fans probably), diskless servers require less maintenance. As mentioned earlier in this thread, diskless would mean no swap and no local storage. Both of which are suitable if you're using SAN (FC/iSCSI) based storage for VM images and if memory overcommit is not allowed. The argument for 'stateless without diskless' is a little more tenuous, however, if you're going to do stateless to get support for diskless, adding in support for local VM image storage and swap is fairly trivial. Andy, I think you've got some opinions here, care to weigh in? Otherwise if there is continued pushback on this feature, I'm happy to not waste effort on it. There are other things we can work on :) Perry
* Just stating the obvious, which is users need to remove-add the host on every reboot. This will not make this feature a lovable one from user's point of view.
I think the point mburns is trying to make in his initial email is that we're going to need to do some joint work between node and vdsm teams to change the registration process so that this is no longer necessary. It will require some redesigning of the registration process
* During initial boot, vdsm-reg configures the networking and creates a management network bridge. This is a very delicate process which may fail due to networking issues such as resolution, routing, etc. So re-doing this on every boot increases the chances of loosing a node due to network problems.
Well, if the network is busted which leads to the bridge rename failing, wouldn't the fact that the network is broken cause other problems anyhow? So I don't see this as a problem. If your network doesn't work properly, don't expect hosts in the network to subsequently work properly. As an aside, why is reverse DNS lookup a requirement? If we remove that it makes things a lot easier, no?
* CA pollution; generating a certificate on each reboot for each node will create a huge number of certificates in the engine side, which eventually may damage the CA. (Unsure if there's a limitation to certificates number, but having hundreds of junk cert's can't be good).
I don't think we should regenerate a new certificate on each boot. I think we need a way for 'an already registered host to retrieve it's certificate from the oVirt Engine server' Using an embedded encryption key (if you trust your mgmt network or are booting from embedded flash), or for the paranoid a key stored in TPM can be used to have vdsm safely retrieve this from the oVirt Engine server on each boot so that it's not required to regenerate/reregister on each boot
* Today there's a supported flow that for nodes with password, the user is allowed to use the "add host" scenario. For stateless, it means re-configuring a password on every boot...
This flow would still be applicable. We are going to allow setting of the admin password embedded in the core ISO via an offline process. Once vdsm is fixed to use a non-root account for installation flow, this is no longer a problem Also, if we (as described above) make registrations persistent across reboots by changing the registration flow a bit, then the install user password only need be set for the initial boot anyhow. Therefore I think as a requirement for stateless oVirt Node, we must have as a prerequsite removing root account usage for registration/installation
- Other issues
* Local storage; so far we were able to define a local storage in ovirt node. Stateless will block this ability.
It shouldn't. The Node should be able to automatically scan locally attached disks to look for a well defined VG or partition label and based on that automatically activate/mount Stateless doesn't imply diskless. It is a requirement even for stateless node usage to be able to leverage locally attached disks both for VM storage and also for Swap.
* Node upgrade; currently it's possible to upgrade a node from the engine. In stateless it will error, since no where to d/l the iso file to.
Upgrades are no longer needed with stateless. To upgrade a stateless node all you need to do is 'reboot from a newer image'. i.e. all upgrades would be done via PXE server image replacement. So the flow of 'upload ISO to running oVirt Node' is no longer even necessary
* Collecting information; core dumps and logging may not be available due to lack of space? Or will it cause kernel panic if all space is consumed?
We already provide ability to send kdumps to remote ssh/NFS location and already provide the ability to use both collectd and rsyslogs to pipe logs/stats to remote server(s). Local logs can be set to logrotate to a reasonable size so that local RAM FS always contains recent log information for quick triage, but long term historical logging would be maintained on the rsyslog server Perry
On 22/02/12 18:06, Perry Myers wrote:
* Just stating the obvious, which is users need to remove-add the host on every reboot. This will not make this feature a lovable one from user's point of view.
I think the point mburns is trying to make in his initial email is that we're going to need to do some joint work between node and vdsm teams to change the registration process so that this is no longer necessary.
It will require some redesigning of the registration process
I'm aware of it, and that's why I'm raising my concerns, so we'll have a (partial) task list ;)
* During initial boot, vdsm-reg configures the networking and creates a management network bridge. This is a very delicate process which may fail due to networking issues such as resolution, routing, etc. So re-doing this on every boot increases the chances of loosing a node due to network problems.
Well, if the network is busted which leads to the bridge rename failing, wouldn't the fact that the network is broken cause other problems anyhow?
Perry, my point is that we're increasing the chances to get into these holes. Network is not busted most of the time, but occasionally there's a glitch and we'd like to stay away from it. I'm sure you know what I'm talking about.
So I don't see this as a problem. If your network doesn't work properly, don't expect hosts in the network to subsequently work properly.
See previous answer.
As an aside, why is reverse DNS lookup a requirement? If we remove that it makes things a lot easier, no?
Not sure I'm the right guy to defend it, but in order to drop reverse-dns, you need to consider dropping SSL, LDAP and some other important shortcuts...
* CA pollution; generating a certificate on each reboot for each node will create a huge number of certificates in the engine side, which eventually may damage the CA. (Unsure if there's a limitation to certificates number, but having hundreds of junk cert's can't be good).
I don't think we should regenerate a new certificate on each boot. I think we need a way for 'an already registered host to retrieve it's certificate from the oVirt Engine server'
Using an embedded encryption key (if you trust your mgmt network or are booting from embedded flash), or for the paranoid a key stored in TPM can be used to have vdsm safely retrieve this from the oVirt Engine server on each boot so that it's not required to regenerate/reregister on each boot
Thoughtful redesign needed here...
* Today there's a supported flow that for nodes with password, the user is allowed to use the "add host" scenario. For stateless, it means re-configuring a password on every boot...
This flow would still be applicable. We are going to allow setting of the admin password embedded in the core ISO via an offline process. Once vdsm is fixed to use a non-root account for installation flow, this is no longer a problem This is not exactly vdsm. More like vdsm-bootstrap.
Also, if we (as described above) make registrations persistent across reboots by changing the registration flow a bit, then the install user password only need be set for the initial boot anyhow.
Therefore I think as a requirement for stateless oVirt Node, we must have as a prerequsite removing root account usage for registration/installation
This is both for vdsm and engine, and I'm not sure it's that trivial.
- Other issues
* Local storage; so far we were able to define a local storage in ovirt node. Stateless will block this ability.
It shouldn't. The Node should be able to automatically scan locally attached disks to look for a well defined VG or partition label and based on that automatically activate/mount
Stateless doesn't imply diskless. It is a requirement even for stateless node usage to be able to leverage locally attached disks both for VM storage and also for Swap.
Still, in a pure disk-less setup you will not have local storage. See also Mike's answer.
* Node upgrade; currently it's possible to upgrade a node from the engine. In stateless it will error, since no where to d/l the iso file to.
Upgrades are no longer needed with stateless. To upgrade a stateless node all you need to do is 'reboot from a newer image'. i.e. all upgrades would be done via PXE server image replacement. So the flow of 'upload ISO to running oVirt Node' is no longer even necessary
This is assuming PXE only use-case. I'm not sure it's the only one.
* Collecting information; core dumps and logging may not be available due to lack of space? Or will it cause kernel panic if all space is consumed?
We already provide ability to send kdumps to remote ssh/NFS location and already provide the ability to use both collectd and rsyslogs to pipe logs/stats to remote server(s). Local logs can be set to logrotate to a reasonable size so that local RAM FS always contains recent log information for quick triage, but long term historical logging would be maintained on the rsyslog server
This needs to be co-ordinated with log-collection, as well as the bootstrapping code.
Perry
-- /d "Willyoupleasehelpmefixmykeyboard?Thespacebarisbroken!"
Well, if the network is busted which leads to the bridge rename failing, wouldn't the fact that the network is broken cause other problems anyhow?
Perry, my point is that we're increasing the chances to get into these holes. Network is not busted most of the time, but occasionally there's a glitch and we'd like to stay away from it. I'm sure you know what I'm talking about.
What if oVirt Node creates ifcfg-ovirt (instead of ifcfg-br0) by default as part of bringing up the network on each boot (via either DHCP or kernel args)? Then vdsm would never need to do this. This particular step could be something that is turned on/off only if vdsm is installed so that it doesn't affect any non-oVirt usages of oVirt Node (Archipel, etc)
* Today there's a supported flow that for nodes with password, the user is allowed to use the "add host" scenario. For stateless, it means re-configuring a password on every boot...
This flow would still be applicable. We are going to allow setting of the admin password embedded in the core ISO via an offline process. Once vdsm is fixed to use a non-root account for installation flow, this is no longer a problem This is not exactly vdsm. More like vdsm-bootstrap.
ack
Also, if we (as described above) make registrations persistent across reboots by changing the registration flow a bit, then the install user password only need be set for the initial boot anyhow.
Therefore I think as a requirement for stateless oVirt Node, we must have as a prerequsite removing root account usage for registration/installation
This is both for vdsm and engine, and I'm not sure it's that trivial.
Understood, but it's a requirement for other things. There are security considerations for requiring remote root ssh access as part of your core infrastructure. So this needs to be dealt with regardless.
- Other issues
* Local storage; so far we were able to define a local storage in ovirt node. Stateless will block this ability.
It shouldn't. The Node should be able to automatically scan locally attached disks to look for a well defined VG or partition label and based on that automatically activate/mount
Stateless doesn't imply diskless. It is a requirement even for stateless node usage to be able to leverage locally attached disks both for VM storage and also for Swap.
Still, in a pure disk-less setup you will not have local storage. See also Mike's answer.
Sure. If you want diskless specifically and then complain about lack of swap or local storage for VMs... then you might not be getting the point :) That has no bearing on the stateless discussion, except that the first pass of stateless might not allow config of local disk/swap to start with. We might do it incrementally
* Node upgrade; currently it's possible to upgrade a node from the engine. In stateless it will error, since no where to d/l the iso file to.
Upgrades are no longer needed with stateless. To upgrade a stateless node all you need to do is 'reboot from a newer image'. i.e. all upgrades would be done via PXE server image replacement. So the flow of 'upload ISO to running oVirt Node' is no longer even necessary
This is assuming PXE only use-case. I'm not sure it's the only one.
Nope... copy oVirt Node 2.2.3 to a USB stick (via ovirt-iso-to-usb-disk) boot a host with it Later... copy oVirt Node 2.2.4 to same USB stick (via ovirt-iso-to-usb-disk) boot the host with it Yes, it requires you to touch the USB stick. If you specifically want stateless (implying no 'installation' of the Node) and you won't be using PXE to run, then it involves legwork. But again, we're not planning to eliminate the current 'install' methods. Stateless is in addition to installing to disk, and using the 'iso upload' upgrade method
* Collecting information; core dumps and logging may not be available due to lack of space? Or will it cause kernel panic if all space is consumed?
We already provide ability to send kdumps to remote ssh/NFS location and already provide the ability to use both collectd and rsyslogs to pipe logs/stats to remote server(s). Local logs can be set to logrotate to a reasonable size so that local RAM FS always contains recent log information for quick triage, but long term historical logging would be maintained on the rsyslog server
This needs to be co-ordinated with log-collection, as well as the bootstrapping code.
Yep. Lots of stuff for vdsm/oVirt Engine team to do in order to meet this requirement :) In contrast, making oVirt Node stateless is quite trivial. Most of the work here is actually for vdsm and other related utilities (like log collector) Perry
On 02/22/2012 07:23 PM, Perry Myers wrote:
Well, if the network is busted which leads to the bridge rename failing, wouldn't the fact that the network is broken cause other problems anyhow?
Perry, my point is that we're increasing the chances to get into these holes. Network is not busted most of the time, but occasionally there's a glitch and we'd like to stay away from it. I'm sure you know what I'm talking about.
What if oVirt Node creates ifcfg-ovirt (instead of ifcfg-br0) by default as part of bringing up the network on each boot (via either DHCP or kernel args)?
what if admin wants this bonded, or bridgeless, or jumbo frames? stateless doesn't mean you can't save configuration somewhere on the network, right (could be on engine, could be just on some nfs or http location). if you have a tpm, just encrypt all the data to make sure no one tampered with your config, or if you don't care, just download your config (well, you trust the network to download your image without encryption, so no need to be fanatic i guess). so in short: 1. pxe boot the image 2. download from known location (kernerl param) the rest of the config you care about, certificates, etc. 3. use tpm for private key (or get the password to keystore in config via kernel parameter if you don't want/have tpm. I guess my main question is why does stateless implies no saved config for all the various issues
Then vdsm would never need to do this. This particular step could be something that is turned on/off only if vdsm is installed so that it doesn't affect any non-oVirt usages of oVirt Node (Archipel, etc)
* Today there's a supported flow that for nodes with password, the user is allowed to use the "add host" scenario. For stateless, it means re-configuring a password on every boot...
This flow would still be applicable. We are going to allow setting of the admin password embedded in the core ISO via an offline process. Once vdsm is fixed to use a non-root account for installation flow, this is no longer a problem This is not exactly vdsm. More like vdsm-bootstrap.
ack
Also, if we (as described above) make registrations persistent across reboots by changing the registration flow a bit, then the install user password only need be set for the initial boot anyhow.
Therefore I think as a requirement for stateless oVirt Node, we must have as a prerequsite removing root account usage for registration/installation
This is both for vdsm and engine, and I'm not sure it's that trivial.
Understood, but it's a requirement for other things. There are security considerations for requiring remote root ssh access as part of your core infrastructure. So this needs to be dealt with regardless.
- Other issues
* Local storage; so far we were able to define a local storage in ovirt node. Stateless will block this ability.
It shouldn't. The Node should be able to automatically scan locally attached disks to look for a well defined VG or partition label and based on that automatically activate/mount
Stateless doesn't imply diskless. It is a requirement even for stateless node usage to be able to leverage locally attached disks both for VM storage and also for Swap.
Still, in a pure disk-less setup you will not have local storage. See also Mike's answer.
Sure. If you want diskless specifically and then complain about lack of swap or local storage for VMs... then you might not be getting the point :)
That has no bearing on the stateless discussion, except that the first pass of stateless might not allow config of local disk/swap to start with. We might do it incrementally
* Node upgrade; currently it's possible to upgrade a node from the engine. In stateless it will error, since no where to d/l the iso file to.
Upgrades are no longer needed with stateless. To upgrade a stateless node all you need to do is 'reboot from a newer image'. i.e. all upgrades would be done via PXE server image replacement. So the flow of 'upload ISO to running oVirt Node' is no longer even necessary
This is assuming PXE only use-case. I'm not sure it's the only one.
Nope...
copy oVirt Node 2.2.3 to a USB stick (via ovirt-iso-to-usb-disk) boot a host with it
Later...
copy oVirt Node 2.2.4 to same USB stick (via ovirt-iso-to-usb-disk) boot the host with it
Yes, it requires you to touch the USB stick. If you specifically want stateless (implying no 'installation' of the Node) and you won't be using PXE to run, then it involves legwork.
But again, we're not planning to eliminate the current 'install' methods. Stateless is in addition to installing to disk, and using the 'iso upload' upgrade method
* Collecting information; core dumps and logging may not be available due to lack of space? Or will it cause kernel panic if all space is consumed?
We already provide ability to send kdumps to remote ssh/NFS location and already provide the ability to use both collectd and rsyslogs to pipe logs/stats to remote server(s). Local logs can be set to logrotate to a reasonable size so that local RAM FS always contains recent log information for quick triage, but long term historical logging would be maintained on the rsyslog server
This needs to be co-ordinated with log-collection, as well as the bootstrapping code.
Yep. Lots of stuff for vdsm/oVirt Engine team to do in order to meet this requirement :)
In contrast, making oVirt Node stateless is quite trivial. Most of the work here is actually for vdsm and other related utilities (like log collector)
Perry _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
On 02/22/2012 03:38 PM, Itamar Heim wrote:
On 02/22/2012 07:23 PM, Perry Myers wrote:
Well, if the network is busted which leads to the bridge rename failing, wouldn't the fact that the network is broken cause other problems anyhow?
Perry, my point is that we're increasing the chances to get into these holes. Network is not busted most of the time, but occasionally there's a glitch and we'd like to stay away from it. I'm sure you know what I'm talking about.
What if oVirt Node creates ifcfg-ovirt (instead of ifcfg-br0) by default as part of bringing up the network on each boot (via either DHCP or kernel args)?
what if admin wants this bonded, or bridgeless, or jumbo frames?
Right now those aren't things you can configure via oVirt Node, they're things you configure via oVirt Engine/vdsm. If we want to add things like this to oVirt Node, we can (just file some RFEs for us) and then we'll expose this sort of stuff via the kernel cmd line args as we do with all of our other config params
stateless doesn't mean you can't save configuration somewhere on the network, right (could be on engine, could be just on some nfs or http location).
Correct. Our present thinking is that oVirt Node specific stuff us saved via kernel cmd line args persisted either on PXE server or directly on boot media. Anything else (oVirt Engine specific for example) would be stored in the mgmt server (OE) itself
if you have a tpm, just encrypt all the data to make sure no one tampered with your config, or if you don't care, just download your config (well, you trust the network to download your image without encryption, so no need to be fanatic i guess).
so in short: 1. pxe boot the image 2. download from known location (kernerl param) the rest of the config you care about, certificates, etc. 3. use tpm for private key (or get the password to keystore in config via kernel parameter if you don't want/have tpm.
Agreed, though w/ vdsm you can simplify by just encrypting the certs 1. boot up, network and basics are configured via kernel cmd args or DNS SRV 2. vdsm retrieves cert from OE and decrypts via locally stored key (TPM or embedded in ISO) 3. vdsm now can securely communicate with OE, so it retrieves config for network/storage from OE and applies that config
I guess my main question is why does stateless implies no saved config for all the various issues
It doesn't imply no saved config, it implies no saved config on the local node itself. Config absolutely will need to be retrieved from a remote server. Right now, oVirt Node config (outside of vdsm) consists of: * network config (dhcp vs. static) * logging/rsyslog config/collectd * mgmt server config (oVirt Engine IP/port) * kdump remote config * auth info (admin password, etc) All of that can be handled through some combination of: * DNS SRV * kernel command line args * embedding things in the ISO pre-deployment (passwords for example) The remaining config is what vdsm does after the node has the above configured, so things like bonding, add'l interfaces, storage, should all be configured by vdsm on each boot by retrieving the configuration details from oVirt Engine and applying them on vdsm startup Perry
On 02/22/2012 03:57 PM, Mike Burns wrote:
There has been a lot of interest in being able to run stateless Nodes with ovirt-engine. ovirt-node has designed a way [1] to achieve this on the node side, but we need input from the engine and vdsm teams to see if we're missing some requirement or if there needs to be changes on the engine/vdsm side to achieve this.
As it currently stands, every time you reboot an ovirt-node that is stateless, it would require manually removing the host in engine, then re-registering/approving it again in engine.
Any thoughts, concerns, input on how to solve this?
Perhaps the node can perform some very basic form of authentication based on IP-address and a key derived from hardware. I see that TPM is already mentioned on the wiki, but even on systems without it, one could simply take a hash of all the MAC-addresses of the system, the CPU serial and the BIOS info from /sys/class/dmi and use that as a form of password. It's better than nothing, or approving nodes all the time (how do you know if the node you are approving is really THE node?) -- Yours sincerely, Floris Bos
On 02/22/2012 03:57 PM, Mike Burns wrote:
There has been a lot of interest in being able to run stateless Nodes with ovirt-engine. ovirt-node has designed a way [1] to achieve this on the node side, but we need input from the engine and vdsm teams to see if we're missing some requirement or if there needs to be changes on the engine/vdsm side to achieve this.
As it currently stands, every time you reboot an ovirt-node that is stateless, it would require manually removing the host in engine, then re-registering/approving it again in engine.
Any thoughts, concerns, input on how to solve this?
Perhaps the node can perform some very basic form of authentication based on IP-address and a key derived from hardware. I see that TPM is already mentioned on the wiki, but even on systems without it, one could simply take a hash of all the MAC-addresses of the system, the CPU serial and the BIOS info from /sys/class/dmi and use that as a form of password. It's better than nothing, or approving nodes all the time (and how do you know if the node you are approving is really THE node?) -- Yours sincerely, Floris Bos
participants (8)
-
Ayal Baron -
David Jaša -
Doron Fediuck -
Floris Bos / Maxnet -
Floris Bos / Maxnet -
Itamar Heim -
Mike Burns -
Perry Myers