Engine insists on running on 1 host in cluster when that host is online
About a month ago, I completely rebuilt my oVirt cluster, as I needed to move all my hardware from 1 data center to another with minimal downtime. All my hardware is in the new data center (yay for HIPAA compliance and 24/7 access, unlike the old place!) I originally built the cluster as a single-node hyperconverged. I then added nodes to it, and then, finally, I reconfigured gluster to run in a replica 2 / arbiter 1 configuration. As of now: - I have 4 compute hosts - Gluster is running fine and replicating fine between the two full replicas, along with the arbiter node - I need to revisit my recent email about gluster replication speed, though However, I'm worried about two things: - I think that my hosts, and hosted-engine, are all still configured to use the single mount point from the original single-node hyperconverged ... i.e. if I shutdown / reboot that host, then the storage "goes away" - How do I reconfigure oVirt to use the 2nd replica as a secondary mount point? - Currently, the Engine is deployed on two of the servers (a and c). But any time one of those servers is online (c), the hosted-engine insists on running on that server c. I cannot migrate the engine off of c. And if the engine is running on the other host a, then I turn host c back on, the engine shuts down from a, and comes back up on c. - How do I "fix" this so that the engine will run on host a, even when host c is turned on? - How do I deploy the hosted-engine to host b? - Is it as simple as logging into host b, and running "hosted-engine --deploy" ? Sent with ProtonMail Secure Email.
How do I reconfigure oVirt to use the 2nd replica as a secondary mount point? Verify that your engine's volume is really OK. gluster volume info enginegluster volume status enginegluster volume heal engine info summary I cannot migrate the engine off of c. And if the engine is running on the other host a, then I turn host c back on, the engine shuts down from a, and comes back up on c. Check the scores of the systems via 'hosted-engine --vm-status'.Check vdsm logs on both hosts.Check the logs in the engine itself. Best Regards,Strahil Nikolov
On Tue, Oct 12, 2021 at 13:34, David White via Users<users@ovirt.org> wrote: About a month ago, I completely rebuilt my oVirt cluster, as I needed to move all my hardware from 1 data center to another with minimal downtime. All my hardware is in the new data center (yay for HIPAA compliance and 24/7 access, unlike the old place!) I originally built the cluster as a single-node hyperconverged. I then added nodes to it, and then, finally, I reconfigured gluster to run in a replica 2 / arbiter 1 configuration. As of now: - I have 4 compute hosts - Gluster is running fine and replicating fine between the two full replicas, along with the arbiter node - I need to revisit my recent email about gluster replication speed, though However, I'm worried about two things: - I think that my hosts, and hosted-engine, are all still configured to use the single mount point from the original single-node hyperconverged ... i.e. if I shutdown / reboot that host, then the storage "goes away" - How do I reconfigure oVirt to use the 2nd replica as a secondary mount point? - Currently, the Engine is deployed on two of the servers (a and c). But any time one of those servers is online (c), the hosted-engine insists on running on that server c. I cannot migrate the engine off of c. And if the engine is running on the other host a, then I turn host c back on, the engine shuts down from a, and comes back up on c. - How do I "fix" this so that the engine will run on host a, even when host c is turned on? - How do I deploy the hosted-engine to host b? - Is it as simple as logging into host b, and running "hosted-engine --deploy" ? Sent with ProtonMail Secure Email. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/74UV4MC3VH2DSN...
Check the scores of the systems via 'hosted-engine --vm-status'. That must be the problem.
Host a has a score of 1800. Host c has a score of 3400. I was able to deploy the hosted-engine to Host B through the web UI. 1) Put it in maintenance mode 2) Installation -> Reinstall ... and make sure to select "Deploy" for the hosted-engine How do I fix / change the score on the first node? Separately, I'm still confused how I would update oVirt to use the 2nd replica as a mount point. Healing the volume seems to me like a completely different task than reconfiguring oVirt to mount the 2nd replica in its mount options? Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, October 12th, 2021 at 3:14 PM, Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
How do I reconfigure oVirt to use the 2nd replica as a secondary mount point?
Verify that your engine's volume is really OK.
gluster volume info enginegluster volume status enginegluster volume heal engine info summary
I cannot migrate the engine off of c. And if the engine is running on the other host a, then I turn host c back on, the engine shuts down from a, and comes back up on c.
Check the scores of the systems via 'hosted-engine --vm-status'.Check vdsm logs on both hosts.Check the logs in the engine itself.
Best Regards,Strahil Nikolov
On Tue, Oct 12, 2021 at 13:34, David White via Users<users@ovirt.org> wrote:About a month ago, I completely rebuilt my oVirt cluster, as I needed to move all my hardware from 1 data center to another with minimal downtime. All my hardware is in the new data center (yay for HIPAA compliance and 24/7 access, unlike the old place!)
I originally built the cluster as a single-node hyperconverged. I then added nodes to it, and then, finally, I reconfigured gluster to run in a replica 2 / arbiter 1 configuration.
As of now:
- I have 4 compute hosts - Gluster is running fine and replicating fine between the two full replicas, along with the arbiter node
- I need to revisit my recent email about gluster replication speed, though
However, I'm worried about two things:
- I think that my hosts, and hosted-engine, are all still configured to use the single mount point from the original single-node hyperconverged ... i.e. if I shutdown / reboot that host, then the storage "goes away"
- How do I reconfigure oVirt to use the 2nd replica as a secondary mount point?
- Currently, the Engine is deployed on two of the servers (a and c). But any time one of those servers is online (c), the hosted-engine insists on running on that server c. I cannot migrate the engine off of c.
And if the engine is running on the other host a, then I turn host c back on, the engine shuts down from a, and comes back up on c.
- How do I "fix" this so that the engine will run on host a, even when host c is turned on? - How do I deploy the hosted-engine to host b?
- Is it as simple as logging into host b, and running "hosted-engine --deploy" ?
Sent with ProtonMail Secure Email.
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/74UV4MC3VH2DSN...
I think I'm making progress. I had 1 storage domain that was hardly used, so in the oVirt UI, I migrated production data onto a different storage domain, and then I put that storage domain into maintenance mode. I was then able to edit the mount options for that storage domain. The path is still as it was originally: cha1-storage.my-domain.com:/data But I added the following as a mount option. Does this look right? backup-volfile-servers=cha2-storage.my-domain.com Essentially, my goal here is for high availability storage. I want to be able to reboot cha1-storage, and still have everything remain online. For the Hosted Storage, can I do this from the command line without putting that into maintenance mode? This is what I see right now: [root@cha2-storage dwhite]# hosted-engine --get-shared-config storage --type=he_local storage : cha1-storage.my-domain.com:/engine, type : he_local Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, October 12th, 2021 at 4:38 PM, David White via Users <users@ovirt.org> wrote:
Check the scores of the systems via 'hosted-engine --vm-status'. That must be the problem.
Host a has a score of 1800. Host c has a score of 3400.
I was able to deploy the hosted-engine to Host B through the web UI.
1) Put it in maintenance mode
2) Installation -> Reinstall ... and make sure to select "Deploy" for the hosted-engine
How do I fix / change the score on the first node?
Separately, I'm still confused how I would update oVirt to use the 2nd replica as a mount point. Healing the volume seems to me like a completely different task than reconfiguring oVirt to mount the 2nd replica in its mount options?
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, October 12th, 2021 at 3:14 PM, Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
How do I reconfigure oVirt to use the 2nd replica as a secondary mount point?
Verify that your engine's volume is really OK.
gluster volume info enginegluster volume status enginegluster volume heal engine info summary
I cannot migrate the engine off of c. And if the engine is running on the other host a, then I turn host c back on, the engine shuts down from a, and comes back up on c.
Check the scores of the systems via 'hosted-engine --vm-status'.Check vdsm logs on both hosts.Check the logs in the engine itself.
Best Regards,Strahil Nikolov
On Tue, Oct 12, 2021 at 13:34, David White via Users<users@ovirt.org> wrote:About a month ago, I completely rebuilt my oVirt cluster, as I needed to move all my hardware from 1 data center to another with minimal downtime. All my hardware is in the new data center (yay for HIPAA compliance and 24/7 access, unlike the old place!)
I originally built the cluster as a single-node hyperconverged. I then added nodes to it, and then, finally, I reconfigured gluster to run in a replica 2 / arbiter 1 configuration.
As of now:
- I have 4 compute hosts - Gluster is running fine and replicating fine between the two full replicas, along with the arbiter node
- I need to revisit my recent email about gluster replication speed, though
However, I'm worried about two things:
- I think that my hosts, and hosted-engine, are all still configured to use the single mount point from the original single-node hyperconverged ... i.e. if I shutdown / reboot that host, then the storage "goes away"
- How do I reconfigure oVirt to use the 2nd replica as a secondary mount point?
- Currently, the Engine is deployed on two of the servers (a and c). But any time one of those servers is online (c), the hosted-engine insists on running on that server c. I cannot migrate the engine off of c.
And if the engine is running on the other host a, then I turn host c back on, the engine shuts down from a, and comes back up on c.
- How do I "fix" this so that the engine will run on host a, even when host c is turned on? - How do I deploy the hosted-engine to host b?
- Is it as simple as logging into host b, and running "hosted-engine --deploy" ?
Sent with ProtonMail Secure Email.
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/74UV4MC3VH2DSN...
For the score issue you can check https://www.google.com/url?sa=t&source=web&rct=j&url=https://www.ovirt.org/images/Hosted-Engine-4.3-deep-dive.pdf&ved=2ahUKEwiWlZu5yMvzAhX0hf0HHaDACzQQFnoECAQQAQ&usg=AOvVaw2SJWnH5ghQoZq7CV_f-9hs and then identify your problem and fix it. For the he_local, you can use hosted-engine --set-shared-config mnt_options backup-volfile-servers=<server2>:<server3> But most probably you will need to set each host in maintenance and reboot it (there are other ways but that's the simplest) to take effect. Best Regards,Strahil Nikolov On Wed, Oct 13, 2021 at 2:49, David White via Users<users@ovirt.org> wrote: _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OOJAMPZK2P2HTZ...
Thank you very much. I was able to resolve the score issue. Turns out there were multiple static routes on this first host, including to a gateway address that I had temporarily setup right when I built the new network in the new datacenter for testing purposes. The symptoms were weird. I was able to dig and resolve public DNS perfectly fine. I was also able to ping public IP addresses perfectly fine. But I was not able to ping or access any content using DNS (i.e. yum couldn't download, for example). Once I identified the faulty route, fixed it, and re-synchronized the Network settings inside the oVirt UI, the score immediately jumped back up to 3400. I was also able to (successfully) test my high availability storage this morning, and confirmed that everything still worked when that host was completely rebooted. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Friday, October 15th, 2021 at 12:25 AM, Strahil Nikolov via Users <users@ovirt.org> wrote:
For the score issue you can check https://www.google.com/url?sa=t&source=web&rct=j&url=https://www.ovirt.org/images/Hosted-Engine-4.3-deep-dive.pdf&ved=2ahUKEwiWlZu5yMvzAhX0hf0HHaDACzQQFnoECAQQAQ&usg=AOvVaw2SJWnH5ghQoZq7CV_f-9hs and then identify your problem and fix it.
For the he_local, you can use hosted-engine --set-shared-config mnt_options
backup-volfile-servers=<server2>:<server3>
But most probably you will need to set each host in maintenance and reboot it (there are other ways but that's the simplest) to take effect.
Best Regards,Strahil Nikolov
On Wed, Oct 13, 2021 at 2:49, David White via Users<users@ovirt.org> wrote:_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OOJAMPZK2P2HTZ...
participants (2)
-
David White -
Strahil Nikolov