OK, where to begin. As for your Gluster issue, Gluster maintains it's own copy of the configuration for each brick outside of oVirt / VDSM. As you have changed the network config manually, you also needed to change the Gluster config to match as well. The fact that you haven't is the reason why Gluster failed to restart the volume. However, In a hyperconverged configuration, oVirt maintains the gluster configuration in it's database. Manually fixing Gluster's configuration on the bricks themselves won't fix the engine's copy. (Believe me, I had to fix this before myself because I didn't use hostnames initially for the bricks. It's a pain to manually fix the database.) That copy is used to connect the VM's to their storage. If the engine's copy doesn't match Gluster's config, you'll have a working Gluster volume but the hosts won't be able to start VMs. Essentially, in a hyperconverged configuration oVirt doesn't allow removal of host with a Gluster brick unless removal of that host won't break Gluster and prevent the volume from running. (I.e. you can't remove a host if doing so would cause the volume to loose quorum.) Your options for fixing Gluster are either: 1. Add enough new bricks to the Gluster volumes so that removal of an old host (brick) doesn't cause quorum loss. - OR - 2. Manually update the engine's database with the engine and all hosts offline to point to the correct hosts, after manually updating the bricks and bringing back up the volume. The first option is your safest bet. But that assumes that the volume is up and can accept new bricks in the first place. If not, you could potentially still do the first option but it would require reverting your network configuration changes on each host first. The second option is one of last resort. This is the reason why I said updating the interfaces manually instead of using the web interface was a bad idea. If possible, use the first option. If not, you'd be better off just hosing the oVirt installation and reinstalling from scratch. If you *really* need to use the second option, you'll need to follow these instructions on each brick: https://serverfault.com/questions/631365/rename-a-glusterfs-peer and then update the engine database manually to point to the correct hostnames for each brick. (Keep in mind I am *NOT* recommending that you do this. This information is provided for educational / experimental purposes only.) As for Matthew's solution, the only reason it worked at all was because you removed and re-added the host from the cluster. Had you not done that, VDSM would have overwritten your changes on the next host upgrade / reinstall, and as you have seen that solution won't completely fix a host in a hyperconverged configuration. As to the question about oVirt's Logical Networks, what I meant was that oVirt doesn't care what the IP configuration is for them, and that if you wanted to change which network the roles used you needed to do so elsewhere in the web interface. The only thing that does matter for each role is that all of the clients using or hosts providing that role can communicate with each other on that interface. (I.e. If you use "Network Bob" for storage and migration, then all hosts with a "Network Bob" interface must be able to communicate with each other over that interface. If you use "Network Alice" for VM consoles, then all end- user workstations must be able to commuicate with the "Network Alice" interface. The exact IPs, vlan IDs, routing tables, and firewall restrictions for a logical network don't matter as long as each role can still reach the role on other hosts over the assigned interface.) -Patrick Hibbs On Sun, 2022-02-20 at 01:17 +0000, Abe E wrote:

* gluster volume info all On Sun, Feb 20, 2022 at 14:46, Strahil Nikolov<hunter86_bg(a)yahoo.com> wrote: In lrder to have an idea how to help you provide the following from all nodes (separate the info per node): ip a sgluster pool listgluster peer statusgluster volume listgluster volume status allgluster volume all Best Regards,Strahil Nikolov On Sun, Feb 20, 2022 at 7:14, Patrick Hibbs<hibbsncc1701(a)gmail.com> wrote: OK, where to begin. As for your Gluster issue, Gluster maintains it's own copy of the configuration for each brick outside of oVirt / VDSM. As you have changed the network config manually, you also needed to change the Gluster config to match as well. The fact that you haven't is the reason why Gluster failed to restart the volume. However, In a hyperconverged configuration, oVirt maintains the gluster configuration in it's database. Manually fixing Gluster's configuration on the bricks themselves won't fix the engine's copy. (Believe me, I had to fix this before myself because I didn't use hostnames initially for the bricks. It's a pain to manually fix the database.) That copy is used to connect the VM's to their storage. If the engine's copy doesn't match Gluster's config, you'll have a working Gluster volume but the hosts won't be able to start VMs. Essentially, in a hyperconverged configuration oVirt doesn't allow removal of host with a Gluster brick unless removal of that host won't break Gluster and prevent the volume from running. (I.e. you can't remove a host if doing so would cause the volume to loose quorum.) Your options for fixing Gluster are either:     1. Add enough new bricks to the Gluster volumes so that removal of an old host (brick) doesn't cause quorum loss.     - OR -     2. Manually update the engine's database with the engine and all hosts offline to point to the correct hosts, after manually updating the bricks and bringing back up the volume. The first option is your safest bet. But that assumes that the volume is up and can accept new bricks in the first place. If not, you could potentially still do the first option but it would require reverting your network configuration changes on each host first. The second option is one of last resort. This is the reason why I said updating the interfaces manually instead of using the web interface was a bad idea. If possible, use the first option. If not, you'd be better off just hosing the oVirt installation and reinstalling from scratch. If you *really* need to use the second option, you'll need to follow these instructions on each brick: https://serverfault.com/questions/631365/rename-a-glusterfs-peer and then update the engine database manually to point to the correct hostnames for each brick. (Keep in mind I am *NOT* recommending that you do this. This information is provided for educational / experimental purposes only.) As for Matthew's solution, the only reason it worked at all was because you removed and re-added the host from the cluster. Had you not done that, VDSM would have overwritten your changes on the next host upgrade / reinstall, and as you have seen that solution won't completely fix a host in a hyperconverged configuration. As to the question about oVirt's Logical Networks, what I meant was that oVirt doesn't care what the IP configuration is for them, and that if you wanted to change which network the roles used you needed to do so elsewhere in the web interface. The only thing that does matter for each role is that all of the clients using or hosts providing that role can communicate with each other on that interface. (I.e. If you use "Network Bob" for storage and migration, then all hosts with a "Network Bob" interface must be able to communicate with each other over that interface. If you use "Network Alice" for VM consoles, then all end- user workstations must be able to commuicate with the "Network Alice" interface. The exact IPs, vlan IDs, routing tables, and firewall restrictions for a logical network don't matter as long as each role can still reach the role on other hosts over the assigned interface.) -Patrick Hibbs On Sun, 2022-02-20 at 01:17 +0000, Abe E wrote: _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6RDDDCPOQOQ...

Thats what I did, rebuilt the Ovirt. Im learning alot so I dont mind but I want to make sure its built and designed well by me before I ever let anyone get hands on it from my colleagues as I inherited an old system that was never maintained and its been a lot to figure out how they designed it with no documentation so its better to start clean.

I'm glad you made it work! My main lesson from oVirt from the last two years is: It's not a turnkey solution. Unless you are willing to dive deep and understand how it works (not so easy, because there is few up-to-date materials to explain the concepts) *AND* spend a significant amount of time to run it through its paces, you may not survive the first thing that goes wrong. And if you do, that may be even worse: Trying to fix a busy production farm that has a core problem is stuff for nightmares. So make sure you have a test environment or two to try things. And it doesn't have to be physical, especially for the more complex things, that require a lot of restart-from-scratch before you do the thing that breaks things. You can run oVirt on oVirt or any other hypervisor that supports nested virtualization and play with that before you deal with a "real patient". I've gone through through catastrophic hardware failures, which would have resulted in in a total loss without the resilience oVirt HCI with Gluster replicas provided, but I've also lost a lot of hair and nails just running it.