Relocating the vdsm upgrade logic into a new or existing service at boot time
Hey, the logic which is run on vdsm upgrades is located in the specfile, and thus run when the rpm get's an update. This is fine as long as rpm is involved (like when rpm or yum or dnf is used), but it becomes - and already is - an issue for flows where the upgrade happens without rpm, like on oVirt Node. On oVirt Node the image for booting the host is getting replaced, we currently mimic the upgrade process of the specfile. But this error prone - we had a lot of bugs around this area (remember the problems with the hooks?), and we need to watch out for changes in the upgrade logic. To simplify and especially unify the logic between Node and normal hosts, I'd like to discuss if the upgrade logic can be moved to a separate service or included in one of the existsing services. The simplest approach to me would be top create a new service which is started before libvirtd and vdsmd/supervdsmd to run some logic if an upgrade of vdsm is detected. This habit - to do the upgrade logic during boot - is actually not new and systemd provides some helpers (ConditionNeedsUpdate=) around this to ease the implementation. Thoughts? Greetings - fabian -- [1] http://0pointer.net/blog/projects/stateless.html [2] http://www.freedesktop.org/software/systemd/man/systemd.unit.html
On Thu, Apr 23, 2015 at 08:58:28AM -0400, Fabian Deutsch wrote:
Hey,
the logic which is run on vdsm upgrades is located in the specfile, and thus run when the rpm get's an update. This is fine as long as rpm is involved (like when rpm or yum or dnf is used), but it becomes - and already is - an issue for flows where the upgrade happens without rpm, like on oVirt Node.
On oVirt Node the image for booting the host is getting replaced, we currently mimic the upgrade process of the specfile. But this error prone - we had a lot of bugs around this area (remember the problems with the hooks?), and we need to watch out for changes in the upgrade logic.
To simplify and especially unify the logic between Node and normal hosts, I'd like to discuss if the upgrade logic can be moved to a separate service or included in one of the existsing services.
The simplest approach to me would be top create a new service which is started before libvirtd and vdsmd/supervdsmd to run some logic if an upgrade of vdsm is detected. This habit - to do the upgrade logic during boot - is actually not new and systemd provides some helpers (ConditionNeedsUpdate=) around this to ease the implementation.
Thoughts?
The problem is real: we have too much logic on %post and %postun scripts, and end up with upgrade bugs litterally on every release. However, for non-node installation we still need to allow configuration after upgrade, but before next boot. This means that everything in %post should be converted into vdsm-tool "configurators". https://gerrit.ovirt.org/#/c/39823/3/vdsm.spec.in is one small step in that direction. When Vdsm starts up, it performs several "upgrade" modules, which are used to upgrade vdsm-specific configurations. These too should operate properly also when run much after boot. Hence, I'm not sure about the benefit of adding an early systemd service, but I may be missing something.
----- Original Message -----
On Thu, Apr 23, 2015 at 08:58:28AM -0400, Fabian Deutsch wrote:
Hey,
the logic which is run on vdsm upgrades is located in the specfile, and thus run when the rpm get's an update. This is fine as long as rpm is involved (like when rpm or yum or dnf is used), but it becomes - and already is - an issue for flows where the upgrade happens without rpm, like on oVirt Node.
On oVirt Node the image for booting the host is getting replaced, we currently mimic the upgrade process of the specfile. But this error prone - we had a lot of bugs around this area (remember the problems with the hooks?), and we need to watch out for changes in the upgrade logic.
To simplify and especially unify the logic between Node and normal hosts, I'd like to discuss if the upgrade logic can be moved to a separate service or included in one of the existsing services.
The simplest approach to me would be top create a new service which is started before libvirtd and vdsmd/supervdsmd to run some logic if an upgrade of vdsm is detected. This habit - to do the upgrade logic during boot - is actually not new and systemd provides some helpers (ConditionNeedsUpdate=) around this to ease the implementation.
Thoughts?
The problem is real: we have too much logic on %post and %postun scripts, and end up with upgrade bugs litterally on every release.
However, for non-node installation we still need to allow configuration after upgrade, but before next boot.
Can you tell what kind of configuration needs to be done after upgrade, before the next reboot?
This means that everything in %post should be converted into vdsm-tool "configurators". https://gerrit.ovirt.org/#/c/39823/3/vdsm.spec.in is one small step in that direction.
I agree, that change is exactly what I'm looking for.
When Vdsm starts up, it performs several "upgrade" modules, which are used to upgrade vdsm-specific configurations. These too should operate properly also when run much after boot.
Hence, I'm not sure about the benefit of adding an early systemd service, but I may be missing something.
I was only thinking about the separate service, to make sure that any service who's configuration file get's possibly updated (i.e. libvirtd) has not yet been started. But if vdsmd and the configurators take care to restart the service who's configuration file got changed, then we do not need a separate one. - fabian
On Fri, Apr 24, 2015 at 04:53:43AM -0400, Fabian Deutsch wrote:
----- Original Message -----
On Thu, Apr 23, 2015 at 08:58:28AM -0400, Fabian Deutsch wrote:
Hey,
the logic which is run on vdsm upgrades is located in the specfile, and thus run when the rpm get's an update. This is fine as long as rpm is involved (like when rpm or yum or dnf is used), but it becomes - and already is - an issue for flows where the upgrade happens without rpm, like on oVirt Node.
On oVirt Node the image for booting the host is getting replaced, we currently mimic the upgrade process of the specfile. But this error prone - we had a lot of bugs around this area (remember the problems with the hooks?), and we need to watch out for changes in the upgrade logic.
To simplify and especially unify the logic between Node and normal hosts, I'd like to discuss if the upgrade logic can be moved to a separate service or included in one of the existsing services.
The simplest approach to me would be top create a new service which is started before libvirtd and vdsmd/supervdsmd to run some logic if an upgrade of vdsm is detected. This habit - to do the upgrade logic during boot - is actually not new and systemd provides some helpers (ConditionNeedsUpdate=) around this to ease the implementation.
Thoughts?
The problem is real: we have too much logic on %post and %postun scripts, and end up with upgrade bugs litterally on every release.
However, for non-node installation we still need to allow configuration after upgrade, but before next boot.
Can you tell what kind of configuration needs to be done after upgrade, before the next reboot?
All of them. When adding a new non-node host, we no longer require a reboot before the host is deemed operational. yum install vdsm ; vdsm-tool configure; service vdsmd start should do everything. I'm saying that we should take logic out of rpm and into `vdsm-tool configure`
This means that everything in %post should be converted into vdsm-tool "configurators". https://gerrit.ovirt.org/#/c/39823/3/vdsm.spec.in is one small step in that direction.
I agree, that change is exactly what I'm looking for.
When Vdsm starts up, it performs several "upgrade" modules, which are used to upgrade vdsm-specific configurations. These too should operate properly also when run much after boot.
Hence, I'm not sure about the benefit of adding an early systemd service, but I may be missing something.
I was only thinking about the separate service, to make sure that any service who's configuration file get's possibly updated (i.e. libvirtd) has not yet been started.
But if vdsmd and the configurators take care to restart the service who's configuration file got changed, then we do not need a separate one.
It's somewhat ugly, but that's what vdsm-tool --nodule libvirt configure already does.
participants (2)
-
Dan Kenigsberg -
Fabian Deutsch