Cluster stuck in maintenance - Hosts not upgrading as expected
I have a 3-node hyperconverged cluster with Gluster filesystem running on RHEL 8.3 hosts. It's been stable on oVirt 4.5. Today, I just upgraded the Engine to v4.6. [Screenshot from 2021-05-22 20-29-23.png] I then logged into the oVirt manager, navigated to Compute -> Clusters, and clicked on Upgrade. The Status of the cluster does have an exclamation mark that says "Upgrade Cluster Compatibility Level", but my understanding is, I shouldn't do that until all 3 of my hosts have been upgraded. Looking at active tasks, I see that the first host in my cluster is stuck in "Upgrading". It's now been like this for 3 hours, without any visible progress. If I try to launch a VM that is not configured for high availability, the VM fails to launch, and I get a message about the cluster being in "Maintenance Mode" and that none of the hosts satisfies current scheduling restraints. How do I.... a) Cancel / clear the upgrade and take the cluster out of "maintenance mode" b) Properly upgrade the cluster, since clicking the "Upgrade" button clearly didn't work? Thanks, David Sent with ProtonMail Secure Email.
I'm saw something similar on a test cluster on CentOS 8.3. You can take it out of global maintenance mode by navigating the engine UI to edit cluster -> scheduling policy and turn off global maintenance there. Not sure what else is going on. It wants me to put all three hosts into maintenance mode which is impossible. On Sat, May 22, 2021 at 8:36 PM David White via Users <users@ovirt.org> wrote:
I have a 3-node hyperconverged cluster with Gluster filesystem running on RHEL 8.3 hosts.
It's been stable on oVirt 4.5. Today, I just upgraded the Engine to v4.6.
[image: Screenshot from 2021-05-22 20-29-23.png]
I then logged into the oVirt manager, navigated to Compute -> Clusters, and clicked on Upgrade. The Status of the cluster does have an exclamation mark that says "Upgrade Cluster Compatibility Level", but my understanding is, I shouldn't do that until all 3 of my hosts have been upgraded.
Looking at active tasks, I see that the first host in my cluster is stuck in "Upgrading". It's now been like this for 3 hours, without any visible progress.
If I try to launch a VM that is not configured for high availability, the VM fails to launch, and I get a message about the cluster being in "Maintenance Mode" and that none of the hosts satisfies current scheduling restraints. How do I.... a) Cancel / clear the upgrade and take the cluster out of "maintenance mode" b) Properly upgrade the cluster, since clicking the "Upgrade" button clearly didn't work?
Thanks, David
Sent with ProtonMail <https://protonmail.com> Secure Email.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4JIRYDSSKBPBJB...
Hello, Can you check at which task the upgrade has struck. Please share the relevant log at the time. To check at which tasks it failed ssh to ovirt engine and navigate to directory /var/log/ovirt-engine/host-deploy/ and also share the vdsm logs from host as well. On Sun, May 23, 2021 at 7:10 AM Edward Berger <edwberger@gmail.com> wrote:
I'm saw something similar on a test cluster on CentOS 8.3. You can take it out of global maintenance mode by navigating the engine UI to edit cluster -> scheduling policy and turn off global maintenance there.
Not sure what else is going on. It wants me to put all three hosts into maintenance mode which is impossible.
On Sat, May 22, 2021 at 8:36 PM David White via Users <users@ovirt.org> wrote:
I have a 3-node hyperconverged cluster with Gluster filesystem running on RHEL 8.3 hosts.
It's been stable on oVirt 4.5. Today, I just upgraded the Engine to v4.6.
[image: Screenshot from 2021-05-22 20-29-23.png]
I then logged into the oVirt manager, navigated to Compute -> Clusters, and clicked on Upgrade. The Status of the cluster does have an exclamation mark that says "Upgrade Cluster Compatibility Level", but my understanding is, I shouldn't do that until all 3 of my hosts have been upgraded.
Looking at active tasks, I see that the first host in my cluster is stuck in "Upgrading". It's now been like this for 3 hours, without any visible progress.
If I try to launch a VM that is not configured for high availability, the VM fails to launch, and I get a message about the cluster being in "Maintenance Mode" and that none of the hosts satisfies current scheduling restraints. How do I.... a) Cancel / clear the upgrade and take the cluster out of "maintenance mode" b) Properly upgrade the cluster, since clicking the "Upgrade" button clearly didn't work?
Thanks, David
Sent with ProtonMail <https://protonmail.com> Secure Email.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4JIRYDSSKBPBJB...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/CC4J7AGJ7TKSSL...
Thank you, Ritesh. Those logs were perfect, and I immediately found the problem. This was my fault. I had disabled Root Login in /etc/ssh/sshd_config (because let's face it, we all can agree that logging in as root over ssh is normally a bad idea!) I just updated that from "no" to "prohibit-password" and restarted sshd. I'll report back if I still have problems after this. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, May 24, 2021 1:53 AM, Ritesh Chikatwar <rchikatw@redhat.com> wrote:
Hello,
Can you check at which task the upgrade has struck. Please share the relevant log at the time.
To check at which tasks it failed ssh to ovirt engine and navigate to directory /var/log/ovirt-engine/host-deploy/ and also share the vdsm logs from host as well.
On Sun, May 23, 2021 at 7:10 AM Edward Berger <edwberger@gmail.com> wrote:
I'm saw something similar on a test cluster on CentOS 8.3. You can take it out of global maintenance mode by navigating the engine UI to edit cluster -> scheduling policy and turn off global maintenance there.
Not sure what else is going on. It wants me to put all three hosts into maintenance mode which is impossible.
On Sat, May 22, 2021 at 8:36 PM David White via Users <users@ovirt.org> wrote:
I have a 3-node hyperconverged cluster with Gluster filesystem running on RHEL 8.3 hosts.
It's been stable on oVirt 4.5. Today, I just upgraded the Engine to v4.6.
[Screenshot from 2021-05-22 20-29-23.png]
I then logged into the oVirt manager, navigated to Compute -> Clusters, and clicked on Upgrade. The Status of the cluster does have an exclamation mark that says "Upgrade Cluster Compatibility Level", but my understanding is, I shouldn't do that until all 3 of my hosts have been upgraded.
Looking at active tasks, I see that the first host in my cluster is stuck in "Upgrading". It's now been like this for 3 hours, without any visible progress.
If I try to launch a VM that is not configured for high availability, the VM fails to launch, and I get a message about the cluster being in "Maintenance Mode" and that none of the hosts satisfies current scheduling restraints. How do I.... a) Cancel / clear the upgrade and take the cluster out of "maintenance mode" b) Properly upgrade the cluster, since clicking the "Upgrade" button clearly didn't work?
Thanks, David
Sent with ProtonMail Secure Email.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4JIRYDSSKBPBJB...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/CC4J7AGJ7TKSSL...
Hi David, you can always prohibit root and also make an exclusion for specific hosts (like oVirt's Engine). Maybe something like : Match Address 192.168.2.5 PermitRootLogin yes Source: https://www.cyberciti.biz/faq/match-address-sshd_config-allow-root-loginfrom... Best Regards, Strahil Nikolov В понеделник, 24 май 2021 г., 23:07:40 ч. Гринуич+3, David White via Users <users@ovirt.org> написа: Thank you, Ritesh. Those logs were perfect, and I immediately found the problem. This was my fault. I had disabled Root Login in /etc/ssh/sshd_config (because let's face it, we all can agree that logging in as root over ssh is normally a bad idea!) I just updated that from "no" to "prohibit-password" and restarted sshd. I'll report back if I still have problems after this. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, May 24, 2021 1:53 AM, Ritesh Chikatwar <rchikatw@redhat.com> wrote:
Hello,
Can you check at which task the upgrade has struck. Please share the relevant log at the time.
To check at which tasks it failed ssh to ovirt engine and navigate to directory /var/log/ovirt-engine/host-deploy/ and also share the vdsm logs from host as well.
On Sun, May 23, 2021 at 7:10 AM Edward Berger <edwberger@gmail.com> wrote:
I'm saw something similar on a test cluster on CentOS 8.3. You can take it out of global maintenance mode by navigating the engine UI to edit cluster -> scheduling policy and turn off global maintenance there.
Not sure what else is going on. It wants me to put all three hosts into maintenance mode which is impossible.
On Sat, May 22, 2021 at 8:36 PM David White via Users <users@ovirt.org> wrote:
I have a 3-node hyperconverged cluster with Gluster filesystem running on RHEL 8.3 hosts.
It's been stable on oVirt 4.5. Today, I just upgraded the Engine to v4.6.
I then logged into the oVirt manager, navigated to Compute -> Clusters, and clicked on Upgrade. The Status of the cluster does have an exclamation mark that says "Upgrade Cluster Compatibility Level", but my understanding is, I shouldn't do that until all 3 of my hosts have been upgraded.
Looking at active tasks, I see that the first host in my cluster is stuck in "Upgrading". It's now been like this for 3 hours, without any visible progress.
If I try to launch a VM that is not configured for high availability, the VM fails to launch, and I get a message about the cluster being in "Maintenance Mode" and that none of the hosts satisfies current scheduling restraints. How do I.... a) Cancel / clear the upgrade and take the cluster out of "maintenance mode" b) Properly upgrade the cluster, since clicking the "Upgrade" button clearly didn't work?
Thanks, David
Sent with ProtonMail Secure Email.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4JIRYDSSKBPBJB...
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/CC4J7AGJ7TKSSL...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/P27FWUG43K4ZM2...
participants (4)
-
David White -
Edward Berger -
Ritesh Chikatwar -
Strahil Nikolov