Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version
Hi, no, fortunately, the bug didn't came back. It looks like, that the engine needs its time to settle after changing the cluster compatibility version. After about 2 hours, the CPU load was as low as before. The time to load the VM UI takes a little bit more time, but I think this is also normal, because all VMs still have the orange triangle and need a reboot. Sorry for the false positive, but I was really afraid about the high load. BR Florian Von: "p staniforth" <P.Staniforth@leedsbeckett.ac.uk> An: "Florian Schmid" <fschmid@ubimet.com>, "users" <users@ovirt.org> Gesendet: Donnerstag, 6. Februar 2020 18:22:51 Betreff: Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Has this bug come back engine.log flooded with "Field 'foo' can not be updated when status is 'Up'" [ https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUH... | https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUH... ] Regards, Paul S. From: Florian Schmid <fschmid@ubimet.com> Sent: 06 February 2020 15:53 To: users <users@ovirt.org> Subject: [ovirt-users] Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. Hi, I have upgraded yesterday one of our ovirt environments from 4.2.8 to 4.3.8. We have a hosted engine, 25 hosts and about 150 VMs running there. After upgrading the cluster compatibility version from 4.2 to 4.3, I have seen a huge increase of CPU load on the engine VM. The engine had 4 cores (16GB mem) and never had load issues, but after the upgrade, the system has a load of 6 and more. I have increased the CPUs to 8, because working in the UI was impossible. I have noticed, that the load always increase, when I load the VM UI tab, where now all VMs have the orange triangle with pending changes: Custom compatibility version The problem I have, is, that I can't reboot now all VMs, maybe in the next 3 or 4 weeks. Working now with the UI is quite problematic, because of the slowness. The engine VM was already rebooted, but didn't solve the problem. What logs do you need and is there a workaround available? I have some more oVirt environments to upgrade, all with even more VMs. BR Florian _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.... | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.... ] oVirt Code of Conduct: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.... | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.... ] List Archives: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovir... | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovir... ] To view the terms under which this email is distributed, please go to:- [ http://leedsbeckett.ac.uk/disclaimer/email/ | http://leedsbeckett.ac.uk/disclaimer/email/ ]
Good afternoon, after rebooting all the VMs, the CPU load of the engine is ok now. I think this high CPU usage is coming from this new feature, that you see on a VM with orange triangle, what will be changed after power cycle. Actually, this is a nice feature, but it brings down an engine, when all VMs need a reboot. After an upgrade for example. Is there a possibility to avoid this? How should I upgrade an oVirt environment with 1000s of VMs? The engine would be unusable. BR Florian Von: "Florian Schmid" <fschmid@ubimet.com> An: "users" <users@ovirt.org> Gesendet: Freitag, 7. Februar 2020 08:13:05 Betreff: [ovirt-users] Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Hi, no, fortunately, the bug didn't came back. It looks like, that the engine needs its time to settle after changing the cluster compatibility version. After about 2 hours, the CPU load was as low as before. The time to load the VM UI takes a little bit more time, but I think this is also normal, because all VMs still have the orange triangle and need a reboot. Sorry for the false positive, but I was really afraid about the high load. BR Florian Von: "p staniforth" <P.Staniforth@leedsbeckett.ac.uk> An: "Florian Schmid" <fschmid@ubimet.com>, "users" <users@ovirt.org> Gesendet: Donnerstag, 6. Februar 2020 18:22:51 Betreff: Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Has this bug come back engine.log flooded with "Field 'foo' can not be updated when status is 'Up'" [ https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUH... | https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUH... ] Regards, Paul S. From: Florian Schmid <fschmid@ubimet.com> Sent: 06 February 2020 15:53 To: users <users@ovirt.org> Subject: [ovirt-users] Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. Hi, I have upgraded yesterday one of our ovirt environments from 4.2.8 to 4.3.8. We have a hosted engine, 25 hosts and about 150 VMs running there. After upgrading the cluster compatibility version from 4.2 to 4.3, I have seen a huge increase of CPU load on the engine VM. The engine had 4 cores (16GB mem) and never had load issues, but after the upgrade, the system has a load of 6 and more. I have increased the CPUs to 8, because working in the UI was impossible. I have noticed, that the load always increase, when I load the VM UI tab, where now all VMs have the orange triangle with pending changes: Custom compatibility version The problem I have, is, that I can't reboot now all VMs, maybe in the next 3 or 4 weeks. Working now with the UI is quite problematic, because of the slowness. The engine VM was already rebooted, but didn't solve the problem. What logs do you need and is there a workaround available? I have some more oVirt environments to upgrade, all with even more VMs. BR Florian _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.... | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.... ] oVirt Code of Conduct: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.... | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.... ] List Archives: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovir... | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovir... ] To view the terms under which this email is distributed, please go to:- [ http://leedsbeckett.ac.uk/disclaimer/email/ | http://leedsbeckett.ac.uk/disclaimer/email/ ] _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/UKNO26UNMQOSBO...
On February 26, 2020 4:05:22 PM GMT+02:00, Florian Schmid <fschmid@ubimet.com> wrote:
Good afternoon,
after rebooting all the VMs, the CPU load of the engine is ok now.
I think this high CPU usage is coming from this new feature, that you see on a VM with orange triangle, what will be changed after power cycle. Actually, this is a nice feature, but it brings down an engine, when all VMs need a reboot. After an upgrade for example.
Is there a possibility to avoid this? How should I upgrade an oVirt environment with 1000s of VMs? The engine would be unusable.
BR Florian
Von: "Florian Schmid" <fschmid@ubimet.com> An: "users" <users@ovirt.org> Gesendet: Freitag, 7. Februar 2020 08:13:05 Betreff: [ovirt-users] Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version
Hi,
no, fortunately, the bug didn't came back.
It looks like, that the engine needs its time to settle after changing the cluster compatibility version. After about 2 hours, the CPU load was as low as before. The time to load the VM UI takes a little bit more time, but I think this is also normal, because all VMs still have the orange triangle and need a reboot.
Sorry for the false positive, but I was really afraid about the high load.
BR Florian
Von: "p staniforth" <P.Staniforth@leedsbeckett.ac.uk> An: "Florian Schmid" <fschmid@ubimet.com>, "users" <users@ovirt.org> Gesendet: Donnerstag, 6. Februar 2020 18:22:51 Betreff: Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version
Has this bug come back
engine.log flooded with "Field 'foo' can not be updated when status is 'Up'"
[ https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUH... | https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUH... ]
Regards,
Paul S.
From: Florian Schmid <fschmid@ubimet.com> Sent: 06 February 2020 15:53 To: users <users@ovirt.org> Subject: [ovirt-users] Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe.
Hi,
I have upgraded yesterday one of our ovirt environments from 4.2.8 to 4.3.8. We have a hosted engine, 25 hosts and about 150 VMs running there.
After upgrading the cluster compatibility version from 4.2 to 4.3, I have seen a huge increase of CPU load on the engine VM. The engine had 4 cores (16GB mem) and never had load issues, but after the upgrade, the system has a load of 6 and more. I have increased the CPUs to 8, because working in the UI was impossible.
I have noticed, that the load always increase, when I load the VM UI tab, where now all VMs have the orange triangle with pending changes: Custom compatibility version
The problem I have, is, that I can't reboot now all VMs, maybe in the next 3 or 4 weeks. Working now with the UI is quite problematic, because of the slowness.
The engine VM was already rebooted, but didn't solve the problem.
What logs do you need and is there a workaround available? I have some more oVirt environments to upgrade, all with even more VMs.
BR Florian _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.... | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.... ] oVirt Code of Conduct: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.... | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.... ] List Archives: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovir... | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovir... ] To view the terms under which this email is distributed, please go to:-
[ http://leedsbeckett.ac.uk/disclaimer/email/ | http://leedsbeckett.ac.uk/disclaimer/email/ ]
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/UKNO26UNMQOSBO...
Hi Florian, Does the API work during this high load status ? Best Regards, Strahil Nikolov
Hi Strahil, I can't say it 100 % sure, but I think yes. We don't use the api a lot, but we have a running job to get all VM data regularly. There is no gap there. You can also work with the UI, but it takes extremely long to load the VMs UI and system load increases a lot. The engine I had this issue, has 8 vCPUs and load increased to 10 or 12 and stayed there for several minutes. When you then close the UI and wait some time, the load goes down to normal, about ~ 0.4. Now, after power cycling all VMs, the engine is fast like before the upgrade. What I'm really afraid of, is, what happened, when the engine has too much load and it looses the connectivity to their nodes. Will it then restart all of them? Probably yes and this would take down the whole environment and we will have a huge outage. And yes, we are a 24/7 company, where it is not so easy to reboot all VMs immediately. BR Florian ----- Ursprüngliche Mail ----- Von: "Strahil Nikolov" <hunter86_bg@yahoo.com> An: "users" <users@ovirt.org>, "Florian Schmid" <fschmid@ubimet.com> Gesendet: Mittwoch, 26. Februar 2020 16:38:05 Betreff: Re: [ovirt-users] Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version On February 26, 2020 4:05:22 PM GMT+02:00, Florian Schmid <fschmid@ubimet.com> wrote:
Good afternoon,
after rebooting all the VMs, the CPU load of the engine is ok now.
I think this high CPU usage is coming from this new feature, that you see on a VM with orange triangle, what will be changed after power cycle. Actually, this is a nice feature, but it brings down an engine, when all VMs need a reboot. After an upgrade for example.
Is there a possibility to avoid this? How should I upgrade an oVirt environment with 1000s of VMs? The engine would be unusable.
BR Florian
Hi Florian, Does the API work during this high load status ? Best Regards, Strahil Nikolov
Hi, I have now updated a second engine. Only the engine is updated from 4.2.8 to 4.3.8. Nothing else is updated yet. I have attached two pictures with pg_top and htop of the engine, while I only browse on VM UI. Maybe someone can have a look on it, if this is normal. The engine is running on NFS datastore on which is on a lot of SSDs. 8 vCPUs 24 GB Memory. BR Florian ----- Ursprüngliche Mail ----- Von: "Florian Schmid" <fschmid@ubimet.com> An: "Strahil Nikolov" <hunter86_bg@yahoo.com> CC: "users" <users@ovirt.org> Gesendet: Donnerstag, 27. Februar 2020 11:18:43 Betreff: [ovirt-users] Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Hi Strahil, I can't say it 100 % sure, but I think yes. We don't use the api a lot, but we have a running job to get all VM data regularly. There is no gap there. You can also work with the UI, but it takes extremely long to load the VMs UI and system load increases a lot. The engine I had this issue, has 8 vCPUs and load increased to 10 or 12 and stayed there for several minutes. When you then close the UI and wait some time, the load goes down to normal, about ~ 0.4. Now, after power cycling all VMs, the engine is fast like before the upgrade. What I'm really afraid of, is, what happened, when the engine has too much load and it looses the connectivity to their nodes. Will it then restart all of them? Probably yes and this would take down the whole environment and we will have a huge outage. And yes, we are a 24/7 company, where it is not so easy to reboot all VMs immediately. BR Florian ----- Ursprüngliche Mail ----- Von: "Strahil Nikolov" <hunter86_bg@yahoo.com> An: "users" <users@ovirt.org>, "Florian Schmid" <fschmid@ubimet.com> Gesendet: Mittwoch, 26. Februar 2020 16:38:05 Betreff: Re: [ovirt-users] Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version On February 26, 2020 4:05:22 PM GMT+02:00, Florian Schmid <fschmid@ubimet.com> wrote:
Good afternoon,
after rebooting all the VMs, the CPU load of the engine is ok now.
I think this high CPU usage is coming from this new feature, that you see on a VM with orange triangle, what will be changed after power cycle. Actually, this is a nice feature, but it brings down an engine, when all VMs need a reboot. After an upgrade for example.
Is there a possibility to avoid this? How should I upgrade an oVirt environment with 1000s of VMs? The engine would be unusable.
BR Florian
Hi Florian, Does the API work during this high load status ? Best Regards, Strahil Nikolov _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EPFSQ24VR2KSX5...
participants (2)
-
Florian Schmid -
Strahil Nikolov