packages that can be updated without maintening hosts
Hi The update notifier in the webadmin was originally designed to alert for new vdsm* packages. Now, I noticed that available update of virt packages and more are notified. I know that hot updating qemu-kvm package does break vms that are running on concerned hosts, but what about other one like libvirt-client? I know it is recommended to put in maintenance while updating, but can we update some minor packages without waiting for migration? -- Nathanaël Blanchet Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
On Mon, Jan 23, 2017 at 1:38 PM, Nathanaël Blanchet <blanchet@abes.fr> wrote:
Hi
The update notifier in the webadmin was originally designed to alert for new vdsm* packages. Now, I noticed that available update of virt packages and more are notified. I know that hot updating qemu-kvm package does break vms that are running on concerned hosts, but what about other one like libvirt-client? I know it is recommended to put in maintenance while updating, but can we update some minor packages without waiting for migration?
Hot-updating any package should not break any running VMs. If it does, it's a bug. I personally agree it's not the best habit to do it nevertheless, and I expect users to put hosts to maintenance before performing any upgrade. We cannot tell which update requires maintenance and which doesn't (or for that matter - requires a reboot or a service restart) - there's no metadata available to do attached to the packages that can tell us that. Y.
-- Nathanaël Blanchet
Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This is a multi-part message in MIME format. --------------E83A5F83A0804BCBACCC816A Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Le 23/01/2017 =C3=A0 13:08, Yaniv Kaul a =C3=A9crit :
On Mon, Jan 23, 2017 at 1:38 PM, Nathana=C3=ABl Blanchet <blanchet@abes=
.fr=20
<mailto:blanchet@abes.fr>> wrote:
Hi
The update notifier in the webadmin was originally designed to alert for new vdsm* packages. Now, I noticed that available update of virt packages and more are notified. I know that hot updating qemu-kvm package does break vms that are running on concerned hosts, but what about other one like libvirt-client? I know it is recommended to put in maintenance while updating, but can we update some minor packages without waiting for migration?
Hot-updating any package should not break any running VMs. If it does,=20 it's a bug. The last time I did a qemu-kvm upgrade, my host became down and the vms=20 on it with a question mark and no possibility to interact with them. My=20 only solution was to use the "confirm host has rebooted" to fence the=20 vms, and then the nightmare began : the high availaible vms rebooted on=20 an other host while they were still active on the first one, so they=20 were up on two hosts at the same time. Their disk began to be written by=20 two vms at the same time and I had to fscsk them to make them up on the=20 next boot. Some database vms were completely unusabled!
So I'm very surprised to hear that it is possible to do hot-updating on=20 these kind of upgrade, and I won't do it anymore!
I personally agree it's not the best habit to do it nevertheless, and=20 I expect users to put hosts to maintenance before performing any upgrad= e.
We cannot tell which update requires maintenance and which doesn't (or=20 for that matter - requires a reboot or a service restart) - there's no=20 metadata available to do attached to the packages that can tell us that= . Y.
--=20 Nathana=C3=ABl Blanchet
Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr <mailto:blanchet@abes.fr>
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
--=20 Nathana=C3=ABl Blanchet Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr --------------E83A5F83A0804BCBACCC816A Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html> <head> <meta content=3D"text/html; charset=3Dutf-8" http-equiv=3D"Content-Ty= pe"> </head> <body text=3D"#000000" bgcolor=3D"#FFFFFF"> <p><br> </p> <br> <div class=3D"moz-cite-prefix">Le 23/01/2017 =C3=A0 13:08, Yaniv Kaul= a =C3=A9crit=C2=A0:<br> </div> <blockquote cite=3D"mid:CAJgorsZscAF9uQ=3DWcjwFEg0EB1XNOEV2Abnrr0NxNiKZrpHwaA@mail.gm= ail.com" type=3D"cite"> <div dir=3D"ltr"><br> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">On Mon, Jan 23, 2017 at 1:38 PM, Nathana=C3=ABl Blanchet <span dir=3D"ltr"><<a moz-do-not-send=3D"true" href=3D"mailto:blanchet@abes.fr" target=3D"_blank"><a class=3D"moz-txt-link-abbreviated" h= ref=3D"mailto:blanchet@abes.fr">blanchet@abes.fr</a></a>></span> wrote= :<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi<br> <br> The update notifier in the webadmin was originally designed to alert for new vdsm* packages. Now, I noticed that available update of virt packages and more are notified. I know that hot updating qemu-kvm package does break vms that are running on concerned hosts, but what about other one like libvirt-client? I know it is recommended to put in maintenance while updating, but can we update some minor packages without waiting for migration?<br> </blockquote> <div><br> </div> <div>Hot-updating any package should not break any running VMs. If it does, it's a bug.</div> </div> </div> </div> </blockquote> The last time I did a qemu-kvm upgrade, my host became down and the vms on it with a question mark and no possibility to interact with them. My only solution was to use the "confirm host has rebooted" to fence the vms, and then the nightmare began : the high availaible vms rebooted on an other host while they were still active on the first one, so they were up on two hosts at the same time. Their disk began to be written by two vms at the same time and I had to fscsk them to make them up on the next boot. Some database vms were completely unusabled!<br> <br> So I'm very surprised to hear that it is possible to do=C2=A0 hot-updating on these kind of upgrade, and I won't do it anymore!<br> <blockquote cite=3D"mid:CAJgorsZscAF9uQ=3DWcjwFEg0EB1XNOEV2Abnrr0NxNiKZrpHwaA@mail.gm= ail.com" type=3D"cite"> <div dir=3D"ltr"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <div>I personally agree it's not the best habit to do it nevertheless, and I expect users to put hosts to maintenance before performing any upgrade.</div> <div><br> </div> <div>We cannot tell which update requires maintenance and which doesn't (or for that matter - requires a reboot or a service restart) - there's no metadata available to do attached to the packages that can tell us that.</div> <div>Y.</div> <div>=C2=A0</div> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br> <br> -- <br> Nathana=C3=ABl Blanchet<br> <br> Supervision r=C3=A9seau<br> P=C3=B4le Infrastrutures Informatiques<br> 227 avenue Professeur-Jean-Louis-Viala<br> 34193 MONTPELLIER CEDEX 5=C2=A0 =C2=A0 =C2=A0 =C2=A0<br> T=C3=A9l. 33 (0)4 67 54 84 55<br> Fax=C2=A0 33 (0)4 67 54 84 14<br> <a moz-do-not-send=3D"true" href=3D"mailto:blanchet@abes.fr= " target=3D"_blank">blanchet@abes.fr</a><br> <br> ______________________________<wbr>_________________<br> Users mailing list<br> <a moz-do-not-send=3D"true" href=3D"mailto:Users@ovirt.org" target=3D"_blank">Users@ovirt.org</a><br> <a moz-do-not-send=3D"true" href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer" target=3D"_blank">http://lists.ovirt.o= rg/mailman<wbr>/listinfo/users</a><br> </blockquote> </div> <br> </div> </div> </blockquote> <br> <pre class=3D"moz-signature" cols=3D"72">--=20 Nathana=C3=ABl Blanchet Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:blanchet@abes.fr">bl= anchet@abes.fr</a> </pre> </body> </html> --------------E83A5F83A0804BCBACCC816A--
On Jan 23, 2017 3:20 PM, "Nathanaël Blanchet" <blanchet@abes.fr> wrote: Le 23/01/2017 à 13:08, Yaniv Kaul a écrit : On Mon, Jan 23, 2017 at 1:38 PM, Nathanaël Blanchet < <blanchet@abes.fr> blanchet@abes.fr> wrote:
Hi
The update notifier in the webadmin was originally designed to alert for new vdsm* packages. Now, I noticed that available update of virt packages and more are notified. I know that hot updating qemu-kvm package does break vms that are running on concerned hosts, but what about other one like libvirt-client? I know it is recommended to put in maintenance while updating, but can we update some minor packages without waiting for migration?
Hot-updating any package should not break any running VMs. If it does, it's a bug. The last time I did a qemu-kvm upgrade, my host became down and the vms on it with a question mark and no possibility to interact with them. My only solution was to use the "confirm host has rebooted" to fence the vms, and then the nightmare began : the high Was the host indeed rebooted? availaible vms rebooted on an other host while they were still active on the first one, so they were up on two hosts at the same time. Their disk began to be written by two vms at the same time and I had to fscsk them to make them up on the next boot. Some database vms were completely unusabled! On 4.1 we are going to introduce a feature that will protect against this situation, by taking a lock on the storage. Y. So I'm very surprised to hear that it is possible to do hot-updating on these kind of upgrade, and I won't do it anymore! I personally agree it's not the best habit to do it nevertheless, and I expect users to put hosts to maintenance before performing any upgrade. We cannot tell which update requires maintenance and which doesn't (or for that matter - requires a reboot or a service restart) - there's no metadata available to do attached to the packages that can tell us that. Y.
-- Nathanaël Blanchet
Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Nathanaël Blanchet Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14blanchet@abes.fr
On Mon, Jan 23, 2017 at 3:27 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Jan 23, 2017 3:20 PM, "Nathanaël Blanchet" <blanchet@abes.fr> wrote:
Le 23/01/2017 à 13:08, Yaniv Kaul a écrit :
On Mon, Jan 23, 2017 at 1:38 PM, Nathanaël Blanchet <blanchet@abes.fr> wrote:
Hi
The update notifier in the webadmin was originally designed to alert for new vdsm* packages. Now, I noticed that available update of virt packages and more are notified. I know that hot updating qemu-kvm package does break vms that are running on concerned hosts, but what about other one like libvirt-client? I know it is recommended to put in maintenance while updating, but can we update some minor packages without waiting for migration?
Hot-updating any package should not break any running VMs. If it does, it's a bug.
Updating vdsm is not supported when the host is not in maintenance. The major issue is sanlock, if it is maintaining a lease on storage, updating sanlock will cause the host to reboot. Sanlock is not petting the host watchdog because you killed sanlock during the upgrade, the watchdog will reboot the host. Updating vdms while file based storage domain are mounted is also not supported, since the local mount path may change between versions, for example because of fixed bugs. If the local mount path changed, the domain will not be considered mounted, and some flows may fail. Nir
The last time I did a qemu-kvm upgrade, my host became down and the vms on it with a question mark and no possibility to interact with them. My only solution was to use the "confirm host has rebooted" to fence the vms, and then the nightmare began : the high
Was the host indeed rebooted?
availaible vms rebooted on an other host while they were still active on the first one, so they were up on two hosts at the same time. Their disk began to be written by two vms at the same time and I had to fscsk them to make them up on the next boot. Some database vms were completely unusabled!
On 4.1 we are going to introduce a feature that will protect against this situation, by taking a lock on the storage. Y.
So I'm very surprised to hear that it is possible to do hot-updating on these kind of upgrade, and I won't do it anymore!
I personally agree it's not the best habit to do it nevertheless, and I expect users to put hosts to maintenance before performing any upgrade.
We cannot tell which update requires maintenance and which doesn't (or for that matter - requires a reboot or a service restart) - there's no metadata available to do attached to the packages that can tell us that. Y.
-- Nathanaël Blanchet
Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Nathanaël Blanchet
Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mon, Jan 23, 2017 at 7:07 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Mon, Jan 23, 2017 at 3:27 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Jan 23, 2017 3:20 PM, "Nathanaël Blanchet" <blanchet@abes.fr> wrote:
Le 23/01/2017 à 13:08, Yaniv Kaul a écrit :
On Mon, Jan 23, 2017 at 1:38 PM, Nathanaël Blanchet <blanchet@abes.fr> wrote:
Hi
The update notifier in the webadmin was originally designed to alert for new vdsm* packages. Now, I noticed that available update of virt
packages
and more are notified. I know that hot updating qemu-kvm package does break vms that are running on concerned hosts, but what about other one like libvirt-client? I know it is recommended to put in maintenance while updating, but can we update some minor packages without waiting for migration?
Hot-updating any package should not break any running VMs. If it does, it's a bug.
Updating vdsm is not supported when the host is not in maintenance.
The major issue is sanlock, if it is maintaining a lease on storage, updating sanlock will cause the host to reboot. Sanlock is not petting the host watchdog because you killed sanlock during the upgrade, the watchdog will reboot the host.
Is the sanlock RPM preventing an upgrade (in the pre-upgrade script) if it has a lock? Y.
Updating vdms while file based storage domain are mounted is also not supported, since the local mount path may change between versions, for example because of fixed bugs. If the local mount path changed, the domain will not be considered mounted, and some flows may fail.
Nir
The last time I did a qemu-kvm upgrade, my host became down and the vms on it with a question mark and no possibility to interact with them. My only solution was to use the "confirm host has rebooted" to fence the vms, and then the nightmare began : the high
Was the host indeed rebooted?
availaible vms rebooted on an other host while they were still active on the first one, so they were up on two hosts at the same time. Their disk began to be written by two vms at the same time and I had to fscsk them to make them up on the next boot. Some database vms were completely unusabled!
On 4.1 we are going to introduce a feature that will protect against this situation, by taking a lock on the storage. Y.
So I'm very surprised to hear that it is possible to do hot-updating on these kind of upgrade, and I won't do it anymore!
I personally agree it's not the best habit to do it nevertheless, and I expect users to put hosts to maintenance before performing any upgrade.
We cannot tell which update requires maintenance and which doesn't (or for that matter - requires a reboot or a service restart) - there's no metadata available to do attached to the packages that can tell us that. Y.
-- Nathanaël Blanchet
Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Nathanaël Blanchet
Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mon, Jan 23, 2017 at 8:31 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Jan 23, 2017 at 7:07 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Mon, Jan 23, 2017 at 3:27 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Jan 23, 2017 3:20 PM, "Nathanaël Blanchet" <blanchet@abes.fr> wrote:
Le 23/01/2017 à 13:08, Yaniv Kaul a écrit :
On Mon, Jan 23, 2017 at 1:38 PM, Nathanaël Blanchet <blanchet@abes.fr> wrote:
Hi
The update notifier in the webadmin was originally designed to alert for new vdsm* packages. Now, I noticed that available update of virt packages and more are notified. I know that hot updating qemu-kvm package does break vms that are running on concerned hosts, but what about other one like libvirt-client? I know it is recommended to put in maintenance while updating, but can we update some minor packages without waiting for migration?
Hot-updating any package should not break any running VMs. If it does, it's a bug.
Updating vdsm is not supported when the host is not in maintenance.
The major issue is sanlock, if it is maintaining a lease on storage, updating sanlock will cause the host to reboot. Sanlock is not petting the host watchdog because you killed sanlock during the upgrade, the watchdog will reboot the host.
Is the sanlock RPM preventing an upgrade (in the pre-upgrade script) if it has a lock?
Adding David.
Y.
Updating vdms while file based storage domain are mounted is also not supported, since the local mount path may change between versions, for example because of fixed bugs. If the local mount path changed, the domain will not be considered mounted, and some flows may fail.
Nir
The last time I did a qemu-kvm upgrade, my host became down and the vms on it with a question mark and no possibility to interact with them. My only solution was to use the "confirm host has rebooted" to fence the vms, and then the nightmare began : the high
Was the host indeed rebooted?
availaible vms rebooted on an other host while they were still active on the first one, so they were up on two hosts at the same time. Their disk began to be written by two vms at the same time and I had to fscsk them to make them up on the next boot. Some database vms were completely unusabled!
On 4.1 we are going to introduce a feature that will protect against this situation, by taking a lock on the storage. Y.
So I'm very surprised to hear that it is possible to do hot-updating on these kind of upgrade, and I won't do it anymore!
I personally agree it's not the best habit to do it nevertheless, and I expect users to put hosts to maintenance before performing any upgrade.
We cannot tell which update requires maintenance and which doesn't (or for that matter - requires a reboot or a service restart) - there's no metadata available to do attached to the packages that can tell us that. Y.
-- Nathanaël Blanchet
Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Nathanaël Blanchet
Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mon, Jan 23, 2017 at 09:50:38PM +0200, Nir Soffer wrote:
The major issue is sanlock, if it is maintaining a lease on storage, updating sanlock will cause the host to reboot. Sanlock is not petting the host watchdog because you killed sanlock during the upgrade, the watchdog will reboot the host.
Is the sanlock RPM preventing an upgrade (in the pre-upgrade script) if it has a lock?
I think that rpm upgrade is not interacting with the running daemon, although there was a problem with that at one time.
participants (4)
-
David Teigland -
Nathanaël Blanchet -
Nir Soffer -
Yaniv Kaul