<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 19, 2016 at 12:29 PM, Nicolas Ecarnot <span dir="ltr"><<a href="mailto:nicolas@ecarnot.net" target="_blank">nicolas@ecarnot.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
I'm digging out this thread because I now had the time to work on this subject, and I'm stuck.<br>
<br>
This oVirt setup has a standalone engine, and 3 hosts.<br>
These 3 hosts are hypervisors and gluster nodes, each using one NIC for all the traffic, that is a very bad idea. (Well, it's working, but not recommended).<br>
<br>
I added 3 OTHER nodes, and so far, I only created the gluster setup and created a replica-3 volume.<br>
Each of these new nodes now have one NIC for management, one NIC for gluster, and other NICs for other things.<br>
Each NIC has an IP + DNS name in its dedicated VLAN : one for mgmt and one for gluster.<br>
The mgmt subnet is routed, though the gluster subnet is not.<br>
Every node can ping each other, either using the mgmt or the gluster subnets.<br>
<br>
The creation of the gluster subnet and volume went very well and seems to be perfect.<br>
<br>
Now, in the oVirt web gui, I'm trying to add these nodes as oVirt hosts.<br>
I'm using their mgmt DNS names, and I'm getting :<br>
"Error while executing action: Server xxxxxxxx is already part of another cluster."<br></blockquote><div><br></div><div>Did you peer probe the gluster cluster prior to adding the nodes to oVirt? What's the output of "gluster peer status"<br><br></div><div>If I understand correctly:<br></div><div>node1 - mgmt.ip.1 & gluster.ip.1<br></div><div>node2 - mgmt.ip.2 & gluster.ip.2<br></div><div>node3 - mgmt.ip.3 & gluster.ip.3<br><br></div><div>Did you create a network and assign "gluster" role to it in the cluster? Were you able to add the first node to cluster, and got this error on second node addition ?<br></div><div>From the error, it looks like oVirt does not understand the peer list returned from gluster is a match with node being added. <br></div><div>Please provide the log snippet of the failure (from engine.log as well as vdsm.log on node)<br><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I found no idea when googling, except something related to gluster (you bet!), telling this may be related to the fact that there is already a volume, managed with a different name.<br>
<br>
Obviously, using a different name and IP is what I needed!<br>
I used "transport.socket.bind-address<wbr>" to make sure the gluster traffic will only use the dedicated NICs.<br>
<br>
<br>
Well, I also tried to created a storage domain relying on the freshly created gluster volume, but as this subnet is not routed, it is not reachable from the manager nor the existing SPM.<br></blockquote><div><br></div><div>The existing SPM - isn't it one of the the 3 new nodes being added? Or are you adding the 3 nodes to your existing cluster? If so, I suggest you try adding this to a new cluster<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
I'm feeling I'm missing something here, so your help is warmly welcome.<br>
<br>
<br>
Nicolas ECARNOT<br>
<br>
PS : CentOS 7.2 everywhere, oVirt 3.6.7<div><div><br>
<br>
Le 27/11/2015 à 20:00, Ivan Bulatovic a écrit :<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Nicolas,<br>
<br>
what works for me in 3.6 is creating a new network for gluster within<br>
oVirt, marking it for gluster use only, optionally setting bonded<br>
interface upon NIC's that are dedicated for gluster traffic and<br>
providing it with an IP address without configuring a gateway, and then<br>
modifying /etc/hosts so that hostnames are resolvable between nodes.<br>
Every node should have two hostnames, one for ovirtmgmt network that is<br>
resolvable via DNS (or via /etc/hosts), and the other for gluster<br>
network that is resolvable purely via /etc/hosts (every node should<br>
contain entries for themselves and for each gluster node).<br>
<br>
Peers should be probed via their gluster hostnames, while ensuring that<br>
gluster peer status contains only addresses and hostnames that are<br>
dedicated for gluster on each node. Same goes for adding bricks,<br>
creating a volume etc.<br>
<br>
This way, no communication (except gluster one) should be allowed<br>
through gluster dedicated vlan. To be on the safe side, we can also<br>
force gluster to listen only on dedicated interfaces via<br>
transport.socket.bind-address option (haven't tried this one, will do).<br>
<br>
Separation of gluster (or in the future any storage network), live<br>
migration network, vm and management network is always a good thing.<br>
Perhaps, we could manage failover of those networks within oVirt, ie. in<br>
case lm network is down - use gluster network for lm and vice versa.<br>
Cool candidate for an RFE, but first we need this supported within<br>
gluster itself. This may prove useful when there is not enough NIC's<br>
available to do a bond beneath every defined network. But we can still<br>
separate traffic and provide failover by selecting multiple networks<br>
without actually doing any load balancing between the two.<br>
<br>
As Nathanaël mentioned, marking network for gluster use is only<br>
available in 3.6. I'm also interested if there is a better way around<br>
this procedure, or perharps enhancing it.<br>
<br>
Kind regards,<br>
<br>
Ivan<br>
<br>
On 11/27/2015 05:47 PM, Nathanaël Blanchet wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello Nicolas,<br>
<br>
Did you have a look to this :<br>
<a href="http://www.ovirt.org/Features/Select_Network_For_Gluster" rel="noreferrer" target="_blank">http://www.ovirt.org/Features/<wbr>Select_Network_For_Gluster</a> ?<br>
But it is only available from >=3.6...<br>
<br>
Le 27/11/2015 17:02, Nicolas Ecarnot a écrit :<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
[Here : oVirt 3.5.3, 3 x CentOS 7.0 hosts with replica-3 gluster SD<br>
on the hosts].<br>
<br>
On the switchs, I have created a dedicated VLAN to isolate the<br>
glusterFS traffic, but I'm not using it yet.<br>
I was thinking of creating a dedicated IP for each node's gluster<br>
NIC, and a DNS record by the way ("my_nodes_name_GL"), but I fear<br>
using this hostname or this ip in oVirt GUI host network interface<br>
tab, leading oVirt think this is a different host.<br>
<br>
Not being sure this fear is clearly described, let's say :<br>
- On each node, I create a second ip+(dns record in the soa) used by<br>
gluster, plugged on the correct VLAN<br>
- in oVirt gui, in the host network setting tab, the interface will<br>
be seen, with its ip, but reverse-dns-related to a different hostname.<br>
Here, I fear oVirt might check this reverse DNS and declare this NIC<br>
belongs to another host.<br>
<br>
I would also prefer not use a reverse pointing to the name of the<br>
host management ip, as this is evil and I'm a good guy.<br>
<br>
On your side, how do you cope with a dedicated storage network in<br>
case of storage+compute mixed hosts?<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
<br></div></div><span><font color="#888888">
-- <br>
Nicolas ECARNOT</font></span><div><div><br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
</div></div></blockquote></div><br></div></div>