[Users] Gluster network info
Hello, I remember in the past that could be a problem to have high usage of ovirtmgmt network because engine sometimes detects hosts as unresponsive. And it should this the reason about bandwith limitation on vm migration, until dedicated network for it has been released. SO the question is : what about ovirtmgmt network for gluster replication when gluster domain is provided by ovirt nodes? I suppose it could be a problem too, couldn't it? In case I have a dedicated network for gluster for the nodes, how can i configure it? Can I configure in this case Gluster storage domain from ovirt engine or does it require to be on this network too? Thanks in advance for any suggestion Gianluca
On 09/28/2013 08:33 PM, Gianluca Cecchi wrote:
Hello, I remember in the past that could be a problem to have high usage of ovirtmgmt network because engine sometimes detects hosts as unresponsive. And it should this the reason about bandwith limitation on vm migration, until dedicated network for it has been released. SO the question is : what about ovirtmgmt network for gluster replication when gluster domain is provided by ovirt nodes? I suppose it could be a problem too, couldn't it? In case I have a dedicated network for gluster for the nodes, how can i configure it? Can I configure in this case Gluster storage domain from ovirt engine or does it require to be on this network too?
Thanks in advance for any suggestion
Gianluca _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
livnat - would network QoS cover this? vijay/sahina - does it make sense to define a 'gluster replication network' or something like that?
Hi Gianluca, <snip> On 10/02/2013 08:52 PM, Itamar Heim wrote:
On 09/28/2013 08:33 PM, Gianluca Cecchi wrote:
Hello, I remember in the past that could be a problem to have high usage of ovirtmgmt network because engine sometimes detects hosts as unresponsive. And it should this the reason about bandwith limitation on vm migration, until dedicated network for it has been released.
We have a dedicated migration network in oVirt 3.3, which means all the migration network is using a dedicated network and not the management network. The caveat is that we still did not add support for capping it's bandwidth. The problem you detailed above can be bypassed by configuring the migration network on a dedicated interface.
SO the question is : what about ovirtmgmt network for gluster replication when gluster domain is provided by ovirt nodes? I suppose it could be a problem too, couldn't it?
yes that is correct, using ovirtmgmt for non management traffic is not a good idea.
In case I have a dedicated network for gluster for the nodes, how can i configure it?
Generally there two ways to go about this: 1. add a new network role 'gluster replication' and then adjust the engine code to pass the network with this role as parameter to the replication verb in VDSM. 2. in the replication verb in the UI let the user choose which network to use for the replication. I am not familiar with the replication verb so I might be missing something but otherwise I think that solution 2 is more simple and less invasive then requiring a new network role.
Can I configure in this case Gluster storage domain from ovirt engine or does it require to be on this network too?
Thanks in advance for any suggestion
Gianluca _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
livnat - would network QoS cover this?
Network Qos is needed in addition to what I have specified above. and it would be available in oVirt 3.4 hopefully, but if you use a dedicated network as suggested above you can bypass the traffic shaping issue by using a dedicated NIC. Livnat
vijay/sahina - does it make sense to define a 'gluster replication network' or something like that?
On Thu, Oct 03, 2013 at 12:44:14PM +0300, Livnat Peer wrote:
Hi Gianluca,
<snip>
On 10/02/2013 08:52 PM, Itamar Heim wrote:
On 09/28/2013 08:33 PM, Gianluca Cecchi wrote:
Hello, I remember in the past that could be a problem to have high usage of ovirtmgmt network because engine sometimes detects hosts as unresponsive. And it should this the reason about bandwith limitation on vm migration, until dedicated network for it has been released.
We have a dedicated migration network in oVirt 3.3, which means all the migration network is using a dedicated network and not the management network. The caveat is that we still did not add support for capping it's bandwidth. The problem you detailed above can be bypassed by configuring the migration network on a dedicated interface.
SO the question is : what about ovirtmgmt network for gluster replication when gluster domain is provided by ovirt nodes? I suppose it could be a problem too, couldn't it?
yes that is correct, using ovirtmgmt for non management traffic is not a good idea.
In case I have a dedicated network for gluster for the nodes, how can i configure it?
Generally there two ways to go about this:
1. add a new network role 'gluster replication' and then adjust the engine code to pass the network with this role as parameter to the replication verb in VDSM.
2. in the replication verb in the UI let the user choose which network to use for the replication.
I am not familiar with the replication verb so I might be missing something but otherwise I think that solution 2 is more simple and less invasive then requiring a new network role.
Bala, is there a means to select, for each node, which IP address is to be used for replication? AFAICT we use the host fqdn, which most probably resolves to the ovirtmgmt device.
Can I configure in this case Gluster storage domain from ovirt engine or does it require to be on this network too?
Thanks in advance for any suggestion
Gianluca _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
livnat - would network QoS cover this?
Network Qos is needed in addition to what I have specified above. and it would be available in oVirt 3.4 hopefully, but if you use a dedicated network as suggested above you can bypass the traffic shaping issue by using a dedicated NIC.
Livnat
vijay/sahina - does it make sense to define a 'gluster replication network' or something like that?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Thu, Oct 3, 2013 at 2:08 PM, Dan Kenigsberg wrote:
SO the question is : what about ovirtmgmt network for gluster replication when gluster domain is provided by ovirt nodes? I suppose it could be a problem too, couldn't it?
yes that is correct, using ovirtmgmt for non management traffic is not a good idea.
In case I have a dedicated network for gluster for the nodes, how can i configure it?
Generally there two ways to go about this:
1. add a new network role 'gluster replication' and then adjust the engine code to pass the network with this role as parameter to the replication verb in VDSM.
2. in the replication verb in the UI let the user choose which network to use for the replication.
I am not familiar with the replication verb so I might be missing something but otherwise I think that solution 2 is more simple and less invasive then requiring a new network role.
Bala, is there a means to select, for each node, which IP address is to be used for replication? AFAICT we use the host fqdn, which most probably resolves to the ovirtmgmt device.
what is "replication verb in the UI"? Inside UI when I click "create volume" and then "add Bricks" buttons I can only select "Server" that in my case is a drop down menu containing only ip addresses of my 2 hypervisors.... Or simply do you mean development steps to integrate this missing functionality in oVirt with steps 1. and 2.? Thanks, Gianluca
On 10/03/2013 03:45 PM, Gianluca Cecchi wrote:
On Thu, Oct 3, 2013 at 2:08 PM, Dan Kenigsberg wrote:
SO the question is : what about ovirtmgmt network for gluster replication when gluster domain is provided by ovirt nodes? I suppose it could be a problem too, couldn't it?
yes that is correct, using ovirtmgmt for non management traffic is not a good idea.
In case I have a dedicated network for gluster for the nodes, how can i configure it?
Generally there two ways to go about this:
1. add a new network role 'gluster replication' and then adjust the engine code to pass the network with this role as parameter to the replication verb in VDSM.
2. in the replication verb in the UI let the user choose which network to use for the replication.
I am not familiar with the replication verb so I might be missing something but otherwise I think that solution 2 is more simple and less invasive then requiring a new network role.
Bala, is there a means to select, for each node, which IP address is to be used for replication? AFAICT we use the host fqdn, which most probably resolves to the ovirtmgmt device.
what is "replication verb in the UI"? Inside UI when I click "create volume" and then "add Bricks" buttons I can only select "Server" that in my case is a drop down menu containing only ip addresses of my 2 hypervisors....
Or simply do you mean development steps to integrate this missing functionality in oVirt with steps 1. and 2.?
I am probably missing some background around Gluster, could you share what triggers the gluster replication?
Thanks, Gianluca
On 10/03/2013 09:27 PM, Livnat Peer wrote:
On 10/03/2013 03:45 PM, Gianluca Cecchi wrote:
On Thu, Oct 3, 2013 at 2:08 PM, Dan Kenigsberg wrote:
SO the question is : what about ovirtmgmt network for gluster replication when gluster domain is provided by ovirt nodes? I suppose it could be a problem too, couldn't it? yes that is correct, using ovirtmgmt for non management traffic is not a good idea.
In case I have a dedicated network for gluster for the nodes, how can i configure it? Generally there two ways to go about this:
1. add a new network role 'gluster replication' and then adjust the engine code to pass the network with this role as parameter to the replication verb in VDSM.
2. in the replication verb in the UI let the user choose which network to use for the replication.
I am not familiar with the replication verb so I might be missing something but otherwise I think that solution 2 is more simple and less invasive then requiring a new network role. Bala, is there a means to select, for each node, which IP address is to be used for replication? AFAICT we use the host fqdn, which most probably resolves to the ovirtmgmt device. what is "replication verb in the UI"? Inside UI when I click "create volume" and then "add Bricks" buttons I can only select "Server" that in my case is a drop down menu containing only ip addresses of my 2 hypervisors....
Or simply do you mean development steps to integrate this missing functionality in oVirt with steps 1. and 2.? I am probably missing some background around Gluster, could you share what triggers the gluster replication? [Adding Amar from gluster team]
There's no user interface way to trigger replication. Gianluca, I assume you do not mean geo-replication but replication of files when volume is of replicated type. Currently, we do not have a way to specify the network to be used when gluster communicates with peers. Like Joop mentioned in a separate mail, if you use FQDN to add host to the engine, and FQDN resolves differently for internal and external communication, gluster would then use the internal network for its communication thanks sahina
Thanks, Gianluca
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Fri, Oct 4, 2013 at 12:01 PM, Sahina Bose wrote:
There's no user interface way to trigger replication. Gianluca, I assume you do not mean geo-replication but replication of files when volume is of replicated type.
YesSahina, I mean replication of files when volume is of replicated type (in my case distributed replicated)
Currently, we do not have a way to specify the network to be used when gluster communicates with peers. Like Joop mentioned in a separate mail, if you use FQDN to add host to the engine, and FQDN resolves differently for internal and external communication, gluster would then use the internal network for its communication
OK, eventuallt I'm going to try this way. Do you think it would work if without dns at all for a test environment I set up this way mgmt lan 192.x gluster network 10.x server hostnames involved engine node01 node02 In /etc/hosts of engine 192.xxx engine 192.yyy node01 192.zzz node02 In /etc/hosts of nodes 192.xxx engine 10.yyy node01 10.zzz node02 and accordingly in sysconfig files of nodes for their hostnames/ip bindng And when I preconfigure gluster on nodes I use on node01 the command gluster peer probe node02 Gianluca
On 10/04/2013 03:40 PM, Gianluca Cecchi wrote:
On Fri, Oct 4, 2013 at 12:01 PM, Sahina Bose wrote:
There's no user interface way to trigger replication. Gianluca, I assume you do not mean geo-replication but replication of files when volume is of replicated type. YesSahina, I mean replication of files when volume is of replicated type (in my case distributed replicated)
Currently, we do not have a way to specify the network to be used when gluster communicates with peers. Like Joop mentioned in a separate mail, if you use FQDN to add host to the engine, and FQDN resolves differently for internal and external communication, gluster would then use the internal network for its communication OK, eventuallt I'm going to try this way. Do you think it would work if without dns at all for a test environment I set up this way
mgmt lan 192.x gluster network 10.x
server hostnames involved engine node01 node02
In /etc/hosts of engine
192.xxx engine 192.yyy node01 192.zzz node02
In /etc/hosts of nodes 192.xxx engine 10.yyy node01 10.zzz node02
and accordingly in sysconfig files of nodes for their hostnames/ip bindng
And when I preconfigure gluster on nodes I use on node01 the command gluster peer probe node02
I believe it should. When you add host to engine you would use node01. Please let us know how it goes, thanks again sahina
Gianluca
On Fri, Oct 4, 2013 at 12:15 PM, Sahina Bose wrote:
I believe it should. When you add host to engine you would use node01. Please let us know how it goes,
thanks again sahina
I tried and it works. I was able to maintain my data too. I stopped the only VM created and put both hosts in maintenance. Then from f18ovn01 (with 10.4.4.58 on ovirtmgmt and 192.168.3.1 on new replication network): please note f18 in the name but is fedora 19.... gluster volume stop gvdata gluster volume delete gvdata gluster peer detach 10.4.4.59 change /etc/hosts on both hosts and activate another interface with 192.168.3.1 and 192.168.3.3 to clear data about previous gluster volume and reuse it with same values: on both hosts setfattr -x trusted.glusterfs.volume-id /gluster/DATA_GLUSTER/brick1 setfattr -x trusted.gfid /gluster/DATA_GLUSTER/brick1 rm -rf rm -rf /gluster/DATA_GLUSTER/brick1/.glusterfs gluster peer probe f18ovn03.mydomain remake the manual steps to create gvdata volume and settings as before from oVirt webadmin activated both hosts and all came up transparently. able to start c6s VM (from run once only due to bug...) wget a netinst.iso for fedora 19 from inside VM and during transfer (VM is running on this f18ovn01 node): [root@f18ovn01 ]# bwm-ng v0.6 (probing every 1.000s), press 'h' for help input: libstatnet type: rate | iface Rx Tx Total ============================================================================== ovirtmgmt: 0.19 KB/s 0.24 KB/s 0.43 KB/s enp3s0: 214.34 KB/s 28794.59 KB/s 29008.93 KB/s ------------------------------------------------------------------------------ total: 214.53 KB/s 28794.82 KB/s 29009.35 KB/s Also restarted ovirt-engine service on engine. Note that from engine point of view when I go in volumes --> gvdata --> bricks, still I see 10.4.4.58 and 10.4.4.59, probably because for the engine those are the ip addresses of f18ovn01 and f18ovn03 nodes.... Or I don't know if it wrote anywhere in the db the gluster data or only gets them dinamically Gianluca
Gianluca Cecchi wrote:
Hello, I remember in the past that could be a problem to have high usage of ovirtmgmt network because engine sometimes detects hosts as unresponsive. And it should this the reason about bandwith limitation on vm migration, until dedicated network for it has been released. SO the question is : what about ovirtmgmt network for gluster replication when gluster domain is provided by ovirt nodes? I suppose it could be a problem too, couldn't it? In case I have a dedicated network for gluster for the nodes, how can i configure it? Can I configure in this case Gluster storage domain from ovirt engine or does it require to be on this network too?
No, it doesn't. I have a setup where the two gluster servers are using 10G and ovirtmgmt is 1G. We use split-dns but the poor mens variety. The engine server (mgmt01) sees both gluster servers on its own network (192.168.x.x) but the gluster servers itself have host entries for the same hostnames in the 172.19.x.x range, same holds for the vmhosts. What happens is that you add the storage servers by fqdn and the gluster peer probe commands resolve to the 10G network and all storage traffic is confined to this network. So managment commands arrive on the 192 network and the actual work is done on the storage network. Hope that I haven't confused you beyond hope:-) Joop
participants (6)
-
Dan Kenigsberg -
Gianluca Cecchi -
Itamar Heim -
Joop -
Livnat Peer -
Sahina Bose