[Users] oVirt 3.1 and Glusterfs how-to
I wrote a basic how to for using glusterfs under oVirt. If you have any questions or you find mistakes please let me know. Thanks Robert http://www.middleswarth.net/content/installing-ovirt-31-and-glusterfs-using-...
On 07/05/2012 12:20 AM, Robert Middleswarth wrote:
I wrote a basic how to for using glusterfs under oVirt. If you have any questions or you find mistakes please let me know.
Thanks Robert
http://www.middleswarth.net/content/installing-ovirt-31-and-glusterfs-using-...
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
oVirt 3.1 is currently in beta and compared to 3.0 it adds a several new features. One of the nice new features is glusterfs support. For it intended use it works well but I personally think it needs to be adjusted to for more generic use. The 2 main limits that effect me are. 1) Only oVirt nodes in the same cluster can be used as bricks to create clusters. So you can't add in storage only nodes in the same oVirt cluster as nodes. However you can create a cluster of just gluster only nodes.
the long term plan would be "dynamic clusters/services" - a subset of hosts with gluster, a subset of hosts with virt, some with both, in overlapping clusters. a more simple solution to this would be just allowing to specify at host level if to use the host for gluster service and/or virt service (so still same set of hosts in a cluster, just allowing to choose if they are to be used for the relevant service).
2) You can not change what interface / IP gluster uses, it will always use the ovirtmgmt network. This is a weakness as many people have an independent network just for storage and they can't use them with 3.1.
sorry, i don't understand this one - only the management of gluster is done via this interface. you can define a different logical network in the cluster for storage, configure ip addresses for them, and define the mount point that way. (well, apart from potential bugs on network definitions in 3.1 which may still exist)
If you can live with these limits then gluster intergration is for you. Before we start you should ask yourself whether you want to use glusterfsover NFS or posix (Native) fs. Both work and are very simlilar in setup but they do work slightly differently. Many prefer the native version and if you are using Fedora 17 with it latest kernal then native (posix) fs actually supports direct IO that will increase the throuput a lot. However I found on CentOS 6.2 nodes NFS ran faster because NFS shares are cached. My testing was over a 1G networks so a faster network will yield different results. Both install methods are pretty much the same I will add notes on there differences.
cache is supposed to be only relevant to read only images like templates, and i'm not even sure this optimization is in. so caching is supposed to be disabled (otherwise, shared disk won't work, etc.) maybe something else?
A quick word about OS. While there are plans to support EL6 and other Linux based distributions, support is currently limited to Fedora 17. However Andrey Gordeev ( http://www.dreyou.org/ovirt/ ) has created EL6 packages. Since Fedora 17 crashes on both my Dells every 10 to 12 hours I use the centos builds right now. Although the CentOS builds work really well there are a few missing features, such as live snapshots. I have done both installs and the steps outlined below work for both.
very nice!
On 07/05/2012 02:49 AM, Itamar Heim wrote:
On 07/05/2012 12:20 AM, Robert Middleswarth wrote:
I wrote a basic how to for using glusterfs under oVirt. If you have any questions or you find mistakes please let me know.
Thanks Robert
http://www.middleswarth.net/content/installing-ovirt-31-and-glusterfs-using-...
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
oVirt 3.1 is currently in beta and compared to 3.0 it adds a several new features. One of the nice new features is glusterfs support. For it intended use it works well but I personally think it needs to be adjusted to for more generic use. The 2 main limits that effect me are. 1) Only oVirt nodes in the same cluster can be used as bricks to create clusters. So you can't add in storage only nodes in the same oVirt cluster as nodes. However you can create a cluster of just gluster only nodes.
the long term plan would be "dynamic clusters/services" - a subset of hosts with gluster, a subset of hosts with virt, some with both, in overlapping clusters. a more simple solution to this would be just allowing to specify at host level if to use the host for gluster service and/or virt service (so still same set of hosts in a cluster, just allowing to choose if they are to be used for the relevant service).
A host level option would fix the issue.
2) You can not change what interface / IP gluster uses, it will always use the ovirtmgmt network. This is a weakness as many people have an independent network just for storage and they can't use them with 3.1.
sorry, i don't understand this one - only the management of gluster is done via this interface. you can define a different logical network in the cluster for storage, configure ip addresses for them, and define the mount point that way. (well, apart from potential bugs on network definitions in 3.1 which may still exist)
Unless I am missing something if I setup Gluster to use anything other then the ovirtmgmt network then I can't create volumes because the engine uses the ovirtmgmt networks IP's and Gluster doesn't recognize it.
If you can live with these limits then gluster intergration is for you. Before we start you should ask yourself whether you want to use glusterfsover NFS or posix (Native) fs. Both work and are very simlilar in setup but they do work slightly differently. Many prefer the native version and if you are using Fedora 17 with it latest kernal then native (posix) fs actually supports direct IO that will increase the throuput a lot. However I found on CentOS 6.2 nodes NFS ran faster because NFS shares are cached. My testing was over a 1G networks so a faster network will yield different results. Both install methods are pretty much the same I will add notes on there differences.
cache is supposed to be only relevant to read only images like templates, and i'm not even sure this optimization is in. so caching is supposed to be disabled (otherwise, shared disk won't work, etc.) maybe something else?
NFS is faster on my network. When I looked around to see why it suggested that reads were cached. It was real world testing you never know.
A quick word about OS. While there are plans to support EL6 and other Linux based distributions, support is currently limited to Fedora 17. However Andrey Gordeev ( http://www.dreyou.org/ovirt/ ) has created EL6 packages. Since Fedora 17 crashes on both my Dells every 10 to 12 hours I use the centos builds right now. Although the CentOS builds work really well there are a few missing features, such as live snapshots. I have done both installs and the steps outlined below work for both.
very nice!
On 07/05/2012 10:06 AM, Robert Middleswarth wrote:
2) You can not change what interface / IP gluster uses, it will always use the ovirtmgmt network. This is a weakness as many people have an independent network just for storage and they can't use them with 3.1.
sorry, i don't understand this one - only the management of gluster is done via this interface. you can define a different logical network in the cluster for storage, configure ip addresses for them, and define the mount point that way. (well, apart from potential bugs on network definitions in 3.1 which may still exist)
Unless I am missing something if I setup Gluster to use anything other then the ovirtmgmt network then I can't create volumes because the engine uses the ovirtmgmt networks IP's and Gluster doesn't recognize it.
engine uses ovirtmgmt to create them. you can define another network to consume them (assuming you have more than a single nic. otherwise, i'm not sure why it matters).
On 07/05/2012 02:19 PM, Itamar Heim wrote:
On 07/05/2012 10:06 AM, Robert Middleswarth wrote:
2) You can not change what interface / IP gluster uses, it will always use the ovirtmgmt network. This is a weakness as many people have an independent network just for storage and they can't use them with 3.1.
sorry, i don't understand this one - only the management of gluster is done via this interface. you can define a different logical network in the cluster for storage, configure ip addresses for them, and define the mount point that way. (well, apart from potential bugs on network definitions in 3.1 which may still exist)
Unless I am missing something if I setup Gluster to use anything other then the ovirtmgmt network then I can't create volumes because the engine uses the ovirtmgmt networks IP's and Gluster doesn't recognize it.
engine uses ovirtmgmt to create them. you can define another network to consume them (assuming you have more than a single nic. otherwise, i'm not sure why it matters).
Are you suggesting that I add both Network IP into Gluster. That would work but how would I know what network Gluster would use to sync up with? Thanks Robert
On 07/05/2012 10:08 PM, Robert Middleswarth wrote:
On 07/05/2012 02:19 PM, Itamar Heim wrote:
On 07/05/2012 10:06 AM, Robert Middleswarth wrote:
2) You can not change what interface / IP gluster uses, it will always use the ovirtmgmt network. This is a weakness as many people have an independent network just for storage and they can't use them with 3.1.
sorry, i don't understand this one - only the management of gluster is done via this interface. you can define a different logical network in the cluster for storage, configure ip addresses for them, and define the mount point that way. (well, apart from potential bugs on network definitions in 3.1 which may still exist)
Unless I am missing something if I setup Gluster to use anything other then the ovirtmgmt network then I can't create volumes because the engine uses the ovirtmgmt networks IP's and Gluster doesn't recognize it.
engine uses ovirtmgmt to create them. you can define another network to consume them (assuming you have more than a single nic. otherwise, i'm not sure why it matters).
Are you suggesting that I add both Network IP into Gluster. That would work but how would I know what network Gluster would use to sync up with?
I can think of two items: 1. network for hosts running VMs to communicate over - I assume will be based on the url of the export you provide (well, for nfs. for native/posix, not sure how redirection will work). maybe worth adding something similar to the 'display network' - telling gluster which interface should be provided for clients to communicate over 2. for communication between gluster nodes for replication, etc. maybe worth defining something like 'live migration network' or 'storage network'[1] - telling nodes which interface they should use for replication between nodes. [1] both storage and live migration network are still not available for ovirt today, just a concept. display network is available today.
Thanks Robert
On 07/05/2012 10:08 PM, Robert Middleswarth wrote:
On 07/05/2012 02:19 PM, Itamar Heim wrote:
On 07/05/2012 10:06 AM, Robert Middleswarth wrote:
2) You can not change what interface / IP gluster uses, it will always use the ovirtmgmt network. This is a weakness as many people have an independent network just for storage and they can't use them with 3.1.
sorry, i don't understand this one - only the management of gluster is done via this interface. you can define a different logical network in the cluster for storage, configure ip addresses for them, and define the mount point that way. (well, apart from potential bugs on network definitions in 3.1 which may still exist)
Unless I am missing something if I setup Gluster to use anything other then the ovirtmgmt network then I can't create volumes because the engine uses the ovirtmgmt networks IP's and Gluster doesn't recognize it.
engine uses ovirtmgmt to create them. you can define another network to consume them (assuming you have more than a single nic. otherwise, i'm not sure why it matters).
Are you suggesting that I add both Network IP into Gluster. That would work but how would I know what network Gluster would use to sync up with?
I can think of two items: 1. network for hosts running VMs to communicate over - I assume will be based on the url of the export you provide (well, for nfs. for native/posix, not sure how redirection will work).
maybe worth adding something similar to the 'display network' - telling gluster which interface should be provided for clients to communicate over I use localhost:/volumename so they talk to the localhost no network
On 07/06/2012 08:19 AM, Itamar Heim wrote: traffic between the oVirt node and the gluster systems all the traffice is between the gluster nodes.
2. for communication between gluster nodes for replication, etc.
maybe worth defining something like 'live migration network' or 'storage network'[1] - telling nodes which interface they should use for replication between nodes.
I think in the end that is the proper fix. Defining a storage network. Then when you go to create gluster volumes it would create them using the IP of the storage network. Also once you get the gluster peer setup cleaned up it would allow the nodes to be joined that way as well.
[1] both storage and live migration network are still not available for ovirt today, just a concept. display network is available today.
Thanks Robert
On 07/06/2012 05:16 PM, Robert Middleswarth wrote:
On 07/05/2012 10:08 PM, Robert Middleswarth wrote:
On 07/05/2012 02:19 PM, Itamar Heim wrote:
On 07/05/2012 10:06 AM, Robert Middleswarth wrote:
> 2) You can not change what interface / IP gluster uses, it will > always use the ovirtmgmt network. This is a weakness as many people > have > an independent network just for storage and they can't use them with > 3.1.
sorry, i don't understand this one - only the management of gluster is done via this interface. you can define a different logical network in the cluster for storage, configure ip addresses for them, and define the mount point that way. (well, apart from potential bugs on network definitions in 3.1 which may still exist)
Unless I am missing something if I setup Gluster to use anything other then the ovirtmgmt network then I can't create volumes because the engine uses the ovirtmgmt networks IP's and Gluster doesn't recognize it.
engine uses ovirtmgmt to create them. you can define another network to consume them (assuming you have more than a single nic. otherwise, i'm not sure why it matters).
Are you suggesting that I add both Network IP into Gluster. That would work but how would I know what network Gluster would use to sync up with?
I can think of two items: 1. network for hosts running VMs to communicate over - I assume will be based on the url of the export you provide (well, for nfs. for native/posix, not sure how redirection will work).
maybe worth adding something similar to the 'display network' - telling gluster which interface should be provided for clients to communicate over I use localhost:/volumename so they talk to the localhost no network
On 07/06/2012 08:19 AM, Itamar Heim wrote: traffic between the oVirt node and the gluster systems all the traffice is between the gluster nodes.
I don't think so. it only means the host uses the local gluster service to fetch the data, then uses the network to directly access the node which has the needed volume to run the VM.
2. for communication between gluster nodes for replication, etc.
maybe worth defining something like 'live migration network' or 'storage network'[1] - telling nodes which interface they should use for replication between nodes. I think in the end that is the proper fix. Defining a storage network. Then when you go to create gluster volumes it would create them using the IP of the storage network. Also once you get the gluster peer setup cleaned up it would allow the nodes to be joined that way as well.
I think there are two things here: - client to gluster communication (for both metadata and data) - replication/sync between gluster nodes
[1] both storage and live migration network are still not available for ovirt today, just a concept. display network is available today.
Thanks Robert
On 07/07/2012 08:44 AM, Itamar Heim wrote:
On 07/06/2012 05:16 PM, Robert Middleswarth wrote:
On 07/05/2012 10:08 PM, Robert Middleswarth wrote:
On 07/05/2012 02:19 PM, Itamar Heim wrote:
On 07/05/2012 10:06 AM, Robert Middleswarth wrote:
> > 2) You can not change what interface / IP gluster uses, it will >> always use the ovirtmgmt network. This is a weakness as many >> people >> have >> an independent network just for storage and they can't use them >> with >> 3.1. > > sorry, i don't understand this one - only the management of > gluster is > done via this interface. you can define a different logical > network in > the cluster for storage, configure ip addresses for them, and > define > the mount point that way. > (well, apart from potential bugs on network definitions in 3.1 > which > may still exist) >
Unless I am missing something if I setup Gluster to use anything other then the ovirtmgmt network then I can't create volumes because the engine uses the ovirtmgmt networks IP's and Gluster doesn't recognize it.
engine uses ovirtmgmt to create them. you can define another network to consume them (assuming you have more than a single nic. otherwise, i'm not sure why it matters).
Are you suggesting that I add both Network IP into Gluster. That would work but how would I know what network Gluster would use to sync up with?
I can think of two items: 1. network for hosts running VMs to communicate over - I assume will be based on the url of the export you provide (well, for nfs. for native/posix, not sure how redirection will work).
maybe worth adding something similar to the 'display network' - telling gluster which interface should be provided for clients to communicate over I use localhost:/volumename so they talk to the localhost no network
On 07/06/2012 08:19 AM, Itamar Heim wrote: traffic between the oVirt node and the gluster systems all the traffice is between the gluster nodes.
I don't think so. it only means the host uses the local gluster service to fetch the data, then uses the network to directly access the node which has the needed volume to run the VM. I may not have made that clear but that is what I was saying the oVirt node talks to gluster either using NFS or Posix FS (Similar code) on the same host and allows Gluster to handle what ever network communication Gluster needs to do.
2. for communication between gluster nodes for replication, etc.
maybe worth defining something like 'live migration network' or 'storage network'[1] - telling nodes which interface they should use for replication between nodes. I think in the end that is the proper fix. Defining a storage network. Then when you go to create gluster volumes it would create them using the IP of the storage network. Also once you get the gluster peer setup cleaned up it would allow the nodes to be joined that way as well.
I think there are two things here: - client to gluster communication (for both metadata and data) - replication/sync between gluster nodes
Correct. In my setup all the client to gluster communication is done over local host. Gluster node's sync must happen over the ovirtmgnt network or I loss the ability to create volumes inside oVirt. If you don't under stand what I mean create a 2 node network. Setup the 2 nodes to use Glusterfs. Configure Gluster using the 2nd set of nics not using the ovirtmgmt and then try and create a volume using the volume tab. It will error out because the blocks use the ovirtmgmt interfaces IP for creating blocks not the 2nd nic IP so Gluster will error out on the block creation. That is why I said that create of an option for a Gluster Network would work. Since the code would adjust the add brink dialog box to allow a diff IP address to be used in Volume creation. Thanks Robert
[1] both storage and live migration network are still not available for ovirt today, just a concept. display network is available today.
Thanks Robert
On Wed, Jul 04, 2012 at 05:20:36PM -0400, Robert Middleswarth wrote:
I wrote a basic how to for using glusterfs under oVirt. If you have any questions or you find mistakes please let me know.
Thanks Robert
http://www.middleswarth.net/content/installing-ovirt-31-and-glusterfs-using-...
Nice blog entry(ies)! You mentions that "We need to open some ports. There are several ways to do this..." But actually I think it would be better if the iptables rules stored in the Engine db are to be tweaked. Frankly, I hate storing files in databases - that's what filesystems were designed for! So it would be nicer if the rules are moved to vds_bootstrap, where you could easily put two flavors thereof: one for gluster and one for virt-only.
On 07/05/2012 09:55 AM, Dan Kenigsberg wrote:
On Wed, Jul 04, 2012 at 05:20:36PM -0400, Robert Middleswarth wrote:
I wrote a basic how to for using glusterfs under oVirt. If you have any questions or you find mistakes please let me know.
Thanks Robert
http://www.middleswarth.net/content/installing-ovirt-31-and-glusterfs-using-...
Nice blog entry(ies)!
You mentions that "We need to open some ports. There are several ways to do this..." But actually I think it would be better if the iptables rules stored in the Engine db are to be tweaked. Frankly, I hate storing files in databases - that's what filesystems were designed for! So it would be nicer if the rules are moved to vds_bootstrap, where you could easily put two flavors thereof: one for gluster and one for virt-only.
I believe the reason the iptable rules are in the db is that we can configure some of the ports. gluster bootstrapping is needed for both installing the rpms, configuring the host, adding iptables, etc. shireesh/bala - any ETA from your side to push this? thanks, Itamar
On Jul 5, 2012, at 10:01 , Itamar Heim wrote:
On 07/05/2012 09:55 AM, Dan Kenigsberg wrote:
On Wed, Jul 04, 2012 at 05:20:36PM -0400, Robert Middleswarth wrote:
I wrote a basic how to for using glusterfs under oVirt. If you have any questions or you find mistakes please let me know.
Thanks Robert
http://www.middleswarth.net/content/installing-ovirt-31-and-glusterfs-using-...
Nice blog entry(ies)!
You mentions that "We need to open some ports. There are several ways to do this..." But actually I think it would be better if the iptables rules stored in the Engine db are to be tweaked. Frankly, I hate storing files in databases - that's what filesystems were designed for! So it would be nicer if the rules are moved to vds_bootstrap, where you could easily put two flavors thereof: one for gluster and one for virt-only.
I believe the reason the iptable rules are in the db is that we can configure some of the ports. sounds to me similar to other issues like sysprep customizations… we do want to have user frriendly interface and provide a nice config of the most useful parameters yet we want to often provide better control for more advanced uses/users Would it perhaps make sense to have a simple config with a nice GUI for few of the ports we do configure often, and an option of editing/uploading of iptables rules file? And I guess similarly for sysprep ...
gluster bootstrapping is needed for both installing the rpms, configuring the host, adding iptables, etc.
shireesh/bala - any ETA from your side to push this?
thanks, Itamar _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 07/05/2012 03:16 AM, Michal Skrivanek wrote:
On Jul 5, 2012, at 10:01 , Itamar Heim wrote:
On 07/05/2012 09:55 AM, Dan Kenigsberg wrote:
On Wed, Jul 04, 2012 at 05:20:36PM -0400, Robert Middleswarth wrote:
I wrote a basic how to for using glusterfs under oVirt. If you have any questions or you find mistakes please let me know.
Thanks Robert
http://www.middleswarth.net/content/installing-ovirt-31-and-glusterfs-using-... Nice blog entry(ies)!
You mentions that "We need to open some ports. There are several ways to do this..." But actually I think it would be better if the iptables rules stored in the Engine db are to be tweaked. Frankly, I hate storing files in databases - that's what filesystems were designed for! So it would be nicer if the rules are moved to vds_bootstrap, where you could easily put two flavors thereof: one for gluster and one for virt-only. I believe the reason the iptable rules are in the db is that we can configure some of the ports. sounds to me similar to other issues like sysprep customizations… we do want to have user frriendly interface and provide a nice config of the most useful parameters yet we want to often provide better control for more advanced uses/users Would it perhaps make sense to have a simple config with a nice GUI for few of the ports we do configure often, and an option of editing/uploading of iptables rules file? And I guess similarly for sysprep ...
Simple config would make since. Would allow for a more secure structure as admin could for instance define a list of IP's that VDSM could listen to improving security. Thanks Robert
gluster bootstrapping is needed for both installing the rpms, configuring the host, adding iptables, etc.
shireesh/bala - any ETA from your side to push this?
thanks, Itamar _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 07/05/2012 02:55 AM, Dan Kenigsberg wrote:
On Wed, Jul 04, 2012 at 05:20:36PM -0400, Robert Middleswarth wrote:
I wrote a basic how to for using glusterfs under oVirt. If you have any questions or you find mistakes please let me know.
Thanks Robert
http://www.middleswarth.net/content/installing-ovirt-31-and-glusterfs-using-... Nice blog entry(ies)!
You mentions that "We need to open some ports. There are several ways to do this..." But actually I think it would be better if the iptables rules stored in the Engine db are to be tweaked. Frankly, I hate storing files in databases - that's what filesystems were designed for! So it would be nicer if the rules are moved to vds_bootstrap, where you could easily put two flavors thereof: one for gluster and one for virt-only. I am more then happy to adjust the steps. Do you have the command for me to insert or change those?
Thanks Robert
participants (4)
-
Dan Kenigsberg -
Itamar Heim -
Michal Skrivanek -
Robert Middleswarth