3:16 a.m.
Hello everyone,
I have an oVirt 4.3.2.5 hyperconverged 3 node production environment and we want to add
some RAM to it.
Can I upgrade the RAM without my users noticing any disruptions and keep the VMs running?
The way I thought I should do it was to migrate any running VMs to the other nodes, then
set one node in maintenance mode, shut it down, place the new memory, bring it back up,
remove it from maintenance mode and see how the installation reacts and repeat for the
other two nodes. Is this correct or should I follow another way?
Will there be a problem during the time when the nodes will not be identical in their
resources?
Thank you for your time,
Souvalioti Maria
6:14 a.m.
On May 19, 2020 11:16:35 AM GMT+03:00, souvaliotimaria(a)mail.com wrote:
Hello everyone,
I have an oVirt 4.3.2.5 hyperconverged 3 node production environment
and we want to add some RAM to it.
Can I upgrade the RAM without my users noticing any disruptions and
keep the VMs running?
The way I thought I should do it was to migrate any running VMs to the
other nodes, then set one node in maintenance mode, shut it down, place
the new memory, bring it back up, remove it from maintenance mode and
see how the installation reacts and repeat for the other two nodes. Is
this correct or should I follow another way?
Will there be a problem during the time when the nodes will not be
identical in their resources?
Thank you for your time,
Souvalioti Maria
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/F4E6DMLL23Q...
There is no requirement that your nodes should have the same ammount of RAM.
At least my setup doesn't have the same RAM on each node.
The problem with nonequal nodes is the scheduling policy, which reminds me to ask if there
are any guides for seting a scheduling policy based on memory usage ?
Best Regards,
Strahil Nikolov
6:48 p.m.
Just like Strahil, I would expect this to work just fine. RAM differences are actually the
smallest concern, unless you run out of it in the mean-time. And there you may want to be
careful and perhaps move VMs around manually with such a small HCI setup.
oVirt will properly optimize the VMs and the hosts to fit, but I don't know what it
will do when there simply isn't enough RAM to run the live VMs. Under the best of
circumstances it should refuse to shut down the node you want to upgrade. Under less
advantagious circumstances some VMs might get paused, or shut down (or killed?). I'd
be interested to hear your experiences, somewhat less inclinded to try myself ;-)
I'd play it safe and reduce the number of running VMs to relieve the RAM pressure. I
assume there is a reason you want more RAM but the only way to get there is to reduce it
first and that doesn't imply "unnoticeable".
10:47 p.m.
On May 20, 2020 2:48:24 AM GMT+03:00, thomas(a)hoberg.net wrote:
Just like Strahil, I would expect this to work just fine. RAM
differences are actually the smallest concern, unless you run out of it
in the mean-time. And there you may want to be careful and perhaps move
VMs around manually with such a small HCI setup.
oVirt will properly optimize the VMs and the hosts to fit, but I don't
know what it will do when there simply isn't enough RAM to run the live
VMs. Under the best of circumstances it should refuse to shut down the
node you want to upgrade. Under less advantagious circumstances some
VMs might get paused, or shut down (or killed?). I'd be interested to
hear your experiences, somewhat less inclinded to try myself ;-)
I'd play it safe and reduce the number of running VMs to relieve the
RAM pressure. I assume there is a reason you want more RAM but the only
way to get there is to reduce it first and that doesn't imply
"unnoticeable".
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BFCUPGDOSC3...
Actually,
If gou set a host into maintenance and there is no space to move - oVirt fails to migrate
the VM and the node cannot be put into maintenance.
In such case I usually power off the non-important VMs and then the node goes into the
maintenance mode (I guess I was fast enough, as there is some timeout setting for going
into that mode).
From there, you will be able to power off, upgrade and return the node in the cluster.
If a node reaches the 90% of memory usage, the ksm service kicks in and starts merging the
same memory pages in order to save memory. Of course this is a trade of cpu cycles for
extra memory - but I haven't seen a hypervisor that is running out of cpu (if you host
HPC VMs could change that).Don't expect magic from KSM, but in my case (32GB) it
gained 5-6 GB from my linux VMs, but this depends on the software in the VMs.
Best Rregards,
Strahil Nikolov
2:05 a.m.
On Tue, May 19, 2020 at 10:18 AM <souvaliotimaria(a)mail.com> wrote:
Hello everyone,
I have an oVirt 4.3.2.5 hyperconverged 3 node production environment and
we want to add some RAM to it.
Can I upgrade the RAM without my users noticing any disruptions and keep
the VMs running?
The way I thought I should do it was to migrate any running VMs to the
other nodes, then set one node in maintenance mode, shut it down, place the
new memory, bring it back up, remove it from maintenance mode and see how
the installation reacts and repeat for the other two nodes. Is this correct
or should I follow another way?
Will there be a problem during the time when the nodes will not be
identical in their resources?
Thank you for your time,
Souvalioti Maria
Hi,
as far as storage is concerned, are you using arbiter on one of the nodes
(keeping only metadata on it and so only 2 copies of data) or a full copy
on all of them?
See here for more info and details:
https://www.ovirt.org/develop/release-management/features/gluster/gluster...
If you are using arbiter on one node, when you are updating one of the
other two nodes, you will have only 1 copy of data active, so take care of
it.
Also, if your VMs make heavy I/O, keep in mind that when rejoining one
"data" node it could take time for it to align the data, depending on what
has been processed during its downtime, so wait for all healing operations
to have completed, before proceeding with the next data node.
You can see healing information on a node for its volumes with the command
gluster volume heal volume_name info
More info here:
https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volum...
Not verified on the field, but in theory oVirt should run those commands
under the curtains for you and not let you put a host into maintenance if
there are healing operations that need to be completed.
See here:
https://www.ovirt.org/develop/release-management/features/gluster/gluster...
HIH,
Gianluca
7:25 a.m.
Hello again,
Hope everyone's ok.
I'm really sorry for being so late, but I came back with the update.
We wanted to upgrade the memory because we plan on deploying several VMs on the platform,
mostly for services, and thought that now was a better time to do the upgrade than later
on.
I manually migrated some of the VMs on one another node, trying to keep the percentage of
the physical memory of each node somewhat equal. The upgrade of the memory of the nodes
happened with no problem (from 32GB to 72GB). The VMs that were active on the time (8 VMs
including HE) didn't show any kind of downtime, slow-down or any other issue. The
percentage of the memory of the two on-line nodes at the time was around 75-80%, with the
HE consuming the most.
The upgrade happened right after I had to replace a failed SAS HDD (hot-plug) which
contained the mirror of the HDD with the ovirt node OS. Everything went as my team and I
hoped with no problems on either side.
For the storage; the deployment we have is GlusterFS with one node as arbiter. When the
node rejoined the others, it took around 8-10 mins for the healing operations to be
completed and since then everything's going perfectly well.
Thank you all for the time you took to respond to me and the valuable information you
shared. It means a lot.
Best regards,
Maria
1354
days inactive
1446
days old
5 comments
4 participants
participants (4)
-
Gianluca Cecchi -
souvaliotimaria＠mail.com -
Strahil Nikolov -
thomas＠hoberg.net