[Users] Creation of preallocated disk with Gluster replication
Hello, I have a Gluster volume in distributed/replicated mode. I have 2 hosts. When I try to create a VM with a preallocated disk, it uses 100% of the available CPU and bandwidth (I have 1 Gigabit network card). The result is I can't even create a preallocated disk because the engine detects a network failure. I get that kind of messages in /var/log/messages : " Jan 2 14:13:54 localhost sanlock[3811]: 2014-01-02 14:13:54+0100 167737 [3811]: s4 kill 21114 sig 15 count 1 Jan 2 14:13:54 localhost wdmd[3800]: test failed rem 51 now 167737 ping 167718 close 167728 renewal 167657 expire 167737 client 3811 sanlock_ef4978d6-5711-4e01-a0ec-7ffbd9 cdbe5d:1 " And that in the Ovirt Gui : " 2014-janv.-02, 15:35 Operation Add-Disk failed to complete. 2014-janv.-02, 15:35 Storage Pool Manager runs on Host HOST2 (Address: X.X.X.X). 2014-janv.-02, 15:35 Invalid status on Data Center GlusterSewan. Setting Data Center status to Non Responsive (On host HOST2, Error: done). 2014-janv.-02, 15:35 State was set to Up for host HOST2. 2014-janv.-02, 15:33 Used Network resources of host HOST2 [98%] exceeded defined threshold [95%]. 2014-janv.-02, 15:33 Add-Disk operation of test_Disk1 was initiated on VM test by admin@internal. I understand that the creation of a 10 Go disk image generates a lot of traffic, but is there a way to limit it so that it doesn't have an impact on the production ? Furthermore, Why does it use so much CPU ressources ? I can see on my monitoring graph a big peak of CPU usage when I launched the operation (probably until 100%). Thank you, Regards, Grégoire Leroy
Adding gluster-users. On 01/02/2014 08:50 PM, gregoire.leroy@retenodus.net wrote:
Hello,
I have a Gluster volume in distributed/replicated mode. I have 2 hosts. When I try to create a VM with a preallocated disk, it uses 100% of the available CPU and bandwidth (I have 1 Gigabit network card). The result is I can't even create a preallocated disk because the engine detects a network failure.
I get that kind of messages in /var/log/messages : " Jan 2 14:13:54 localhost sanlock[3811]: 2014-01-02 14:13:54+0100 167737 [3811]: s4 kill 21114 sig 15 count 1 Jan 2 14:13:54 localhost wdmd[3800]: test failed rem 51 now 167737 ping 167718 close 167728 renewal 167657 expire 167737 client 3811 sanlock_ef4978d6-5711-4e01-a0ec-7ffbd9 cdbe5d:1 "
And that in the Ovirt Gui : " 2014-janv.-02, 15:35 Operation Add-Disk failed to complete. 2014-janv.-02, 15:35 Storage Pool Manager runs on Host HOST2 (Address: X.X.X.X). 2014-janv.-02, 15:35 Invalid status on Data Center GlusterSewan. Setting Data Center status to Non Responsive (On host HOST2, Error: done). 2014-janv.-02, 15:35 State was set to Up for host HOST2. 2014-janv.-02, 15:33 Used Network resources of host HOST2 [98%] exceeded defined threshold [95%]. 2014-janv.-02, 15:33 Add-Disk operation of test_Disk1 was initiated on VM test by admin@internal.
I understand that the creation of a 10 Go disk image generates a lot of traffic, but is there a way to limit it so that it doesn't have an impact on the production ? Furthermore, Why does it use so much CPU ressources ? I can see on my monitoring graph a big peak of CPU usage when I launched the operation (probably until 100%).
Do you happen to notice what is consuming CPU? Since the same cluster does both virtualization and storage, a GigE network might get saturated very quickly. Is it possible to separate out the management and data/gluster traffic in this setup? Regards, Vijay
Do you happen to notice what is consuming CPU?
When I check with top, glusterfs and glusterfsd are the only process who use a significant amount of CPU. Load average is between 5 and 6, and I don't have any started VM.
Since the same cluster does both virtualization and storage, a GigE network might get saturated very quickly. Is it possible to separate out the management and data/gluster traffic in this setup?
Unfortunately, it's not possible. I only have two hosts, both for virtualization and storage. Why does glusterfs use so much CPU ? About the network traffic, the creation of a pre-allocated disk is not something which absolutely needs to be fast : if it takes 5 minutes, well... it takes 5 minutes. I guess that if there's a way to limit the bandwidth available for glusterfs, it'll still have an impact on production because glusterfs and VM will be in competition for bandwidth, right ? Is there a way to limit the bandwidth only for the creation of a pre-allocated disk ? Thank you, Regards, Grégoire Leroy
Hello, Do you need more informations about this issue ? Do you think this problem is likely to show up in other cases ? I mean, is that an expected behaviour with my environment, or is it unexpected ? Is there a way to limit the bandwidth usage for creation of pre-allocated disk so that it doesn't impact production ? Thank you, Regards, Grégoire Le 2014-01-02 17:42, Vijay Bellur a écrit :
Adding gluster-users.
On 01/02/2014 08:50 PM, gregoire.leroy@retenodus.net wrote:
Hello,
I have a Gluster volume in distributed/replicated mode. I have 2 hosts. When I try to create a VM with a preallocated disk, it uses 100% of the available CPU and bandwidth (I have 1 Gigabit network card). The result is I can't even create a preallocated disk because the engine detects a network failure.
I get that kind of messages in /var/log/messages : " Jan 2 14:13:54 localhost sanlock[3811]: 2014-01-02 14:13:54+0100 167737 [3811]: s4 kill 21114 sig 15 count 1 Jan 2 14:13:54 localhost wdmd[3800]: test failed rem 51 now 167737 ping 167718 close 167728 renewal 167657 expire 167737 client 3811 sanlock_ef4978d6-5711-4e01-a0ec-7ffbd9 cdbe5d:1 "
And that in the Ovirt Gui : " 2014-janv.-02, 15:35 Operation Add-Disk failed to complete. 2014-janv.-02, 15:35 Storage Pool Manager runs on Host HOST2 (Address: X.X.X.X). 2014-janv.-02, 15:35 Invalid status on Data Center GlusterSewan. Setting Data Center status to Non Responsive (On host HOST2, Error: done). 2014-janv.-02, 15:35 State was set to Up for host HOST2. 2014-janv.-02, 15:33 Used Network resources of host HOST2 [98%] exceeded defined threshold [95%]. 2014-janv.-02, 15:33 Add-Disk operation of test_Disk1 was initiated on VM test by admin@internal.
I understand that the creation of a 10 Go disk image generates a lot of traffic, but is there a way to limit it so that it doesn't have an impact on the production ? Furthermore, Why does it use so much CPU ressources ? I can see on my monitoring graph a big peak of CPU usage when I launched the operation (probably until 100%).
Do you happen to notice what is consuming CPU? Since the same cluster does both virtualization and storage, a GigE network might get saturated very quickly. Is it possible to separate out the management and data/gluster traffic in this setup?
Regards, Vijay
Grégoire- I think this is expected behavior. Well, at least the high glusterfsd CPU use during disk creation, anyway. I tried creating a 10 G disk on my test environment and observed similar high CPU usage by glusterfsd. Did the creation on the i5 system, it showed 95%-105% cpu for glusterfsd during creation, with the core2 system running ~35-65% glusterfsd utilization during the creation. Minor disk wait was observed on both systems, < 10% peak and generally < 5%. I imagine my ZFS cached backends helped a lot here. Took about 3 minutes, roughly what I’d expect for the i5’s disk system. Network usage was about 45% of the 1G link. No errors or messages logged to /var/log/messages. Depending on what your test setup looks like, I’d check my network for packet loss or errors first. Then look at my storage setup and test pure throughput on the disks to see what you’ve got, maybe see what else is running. Did you use an NFS cluster or a PosixFS cluster for this? My test setup, running a version of the nightly self-hosted setup w/ gluster distributed/replicated disks as shared storage, in a NFS cluster: Core i5 3570K @ 3.4Ghz, 16G Ram Boot disks: 2x 32G SATA SSDs in raid-1 Storage system: 4x500G Seagate RE3s in a ZFS raid-10 w/ 1GB ZIL & ~22G L2ARC caching from boot drives 1 1G ethernet 2 VMs running Core2 Duo E8500 @ 3.16GHz, 8G Ram Boot disks: 2x 32G SATA SSDS in raid-1 Storage system: 2x1500G WD Green drives in a ZFS Raid w/ 1GB ZIL & ~22G L2ARC cache from boot drives 1 1G ethernet They are connected through a Netgear Prosafe+ workgroup style switch, not much going on between them. -Darrell On Jan 8, 2014, at 7:49 AM, gregoire.leroy@retenodus.net wrote:
Hello,
Do you need more informations about this issue ? Do you think this problem is likely to show up in other cases ? I mean, is that an expected behaviour with my environment, or is it unexpected ?
Is there a way to limit the bandwidth usage for creation of pre-allocated disk so that it doesn't impact production ?
Thank you, Regards, Grégoire
Le 2014-01-02 17:42, Vijay Bellur a écrit :
Adding gluster-users. On 01/02/2014 08:50 PM, gregoire.leroy@retenodus.net wrote:
Hello, I have a Gluster volume in distributed/replicated mode. I have 2 hosts. When I try to create a VM with a preallocated disk, it uses 100% of the available CPU and bandwidth (I have 1 Gigabit network card). The result is I can't even create a preallocated disk because the engine detects a network failure. I get that kind of messages in /var/log/messages : " Jan 2 14:13:54 localhost sanlock[3811]: 2014-01-02 14:13:54+0100 167737 [3811]: s4 kill 21114 sig 15 count 1 Jan 2 14:13:54 localhost wdmd[3800]: test failed rem 51 now 167737 ping 167718 close 167728 renewal 167657 expire 167737 client 3811 sanlock_ef4978d6-5711-4e01-a0ec-7ffbd9 cdbe5d:1 " And that in the Ovirt Gui : " 2014-janv.-02, 15:35 Operation Add-Disk failed to complete. 2014-janv.-02, 15:35 Storage Pool Manager runs on Host HOST2 (Address: X.X.X.X). 2014-janv.-02, 15:35 Invalid status on Data Center GlusterSewan. Setting Data Center status to Non Responsive (On host HOST2, Error: done). 2014-janv.-02, 15:35 State was set to Up for host HOST2. 2014-janv.-02, 15:33 Used Network resources of host HOST2 [98%] exceeded defined threshold [95%]. 2014-janv.-02, 15:33 Add-Disk operation of test_Disk1 was initiated on VM test by admin@internal. I understand that the creation of a 10 Go disk image generates a lot of traffic, but is there a way to limit it so that it doesn't have an impact on the production ? Furthermore, Why does it use so much CPU ressources ? I can see on my monitoring graph a big peak of CPU usage when I launched the operation (probably until 100%). Do you happen to notice what is consuming CPU? Since the same cluster does both virtualization and storage, a GigE network might get saturated very quickly. Is it possible to separate out the management and data/gluster traffic in this setup? Regards, Vijay
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Skickat från min iPhone
8 jan 2014 kl. 18:47 skrev "Darrell Budic" <darrell.budic@zenfire.com>:
Grégoire-
I think this is expected behavior. Well, at least the high glusterfsd CPU use during disk creation, anyway. I tried creating a 10 G disk on my test environment and observed similar high CPU usage by glusterfsd. Did the creation on the i5 system, it showed 95%-105% cpu for glusterfsd during creation, with the core2 system running ~35-65% glusterfsd utilization during the creation. Minor disk wait was observed on both systems, < 10% peak and generally < 5%. I imagine my ZFS cached backends helped a lot here. Took about 3 minutes, roughly what I’d expect for the i5’s disk system. Network usage was about 45% of the 1G link. No errors or messages logged to /var/log/messages.
Depending on what your test setup looks like, I’d check my network for packet loss or errors first. Then look at my storage setup and test pure throughput on the disks to see what you’ve got, maybe see what else is running. Did you use an NFS cluster or a PosixFS cluster for this?
My test setup, running a version of the nightly self-hosted setup w/ gluster distributed/replicated disks as shared storage, in a NFS cluster:
Core i5 3570K @ 3.4Ghz, 16G Ram Boot disks: 2x 32G SATA SSDs in raid-1 Storage system: 4x500G Seagate RE3s in a ZFS raid-10 w/ 1GB ZIL & ~22G L2ARC caching from boot drives 1 1G ethernet 2 VMs running
Core2 Duo E8500 @ 3.16GHz, 8G Ram Boot disks: 2x 32G SATA SSDS in raid-1 Storage system: 2x1500G WD Green drives in a ZFS Raid w/ 1GB ZIL & ~22G L2ARC cache from boot drives 1 1G ethernet
They are connected through a Netgear Prosafe+ workgroup style switch, not much going on between them.
-Darrell
Just curious, are you doing ZFS in Linux? /K
On Jan 8, 2014, at 7:49 AM, gregoire.leroy@retenodus.net wrote:
Hello,
Do you need more informations about this issue ? Do you think this problem is likely to show up in other cases ? I mean, is that an expected behaviour with my environment, or is it unexpected ?
Is there a way to limit the bandwidth usage for creation of pre-allocated disk so that it doesn't impact production ?
Thank you, Regards, Grégoire
Le 2014-01-02 17:42, Vijay Bellur a écrit :
Adding gluster-users. On 01/02/2014 08:50 PM, gregoire.leroy@retenodus.net wrote:
Hello, I have a Gluster volume in distributed/replicated mode. I have 2 hosts. When I try to create a VM with a preallocated disk, it uses 100% of the available CPU and bandwidth (I have 1 Gigabit network card). The result is I can't even create a preallocated disk because the engine detects a network failure. I get that kind of messages in /var/log/messages : " Jan 2 14:13:54 localhost sanlock[3811]: 2014-01-02 14:13:54+0100 167737 [3811]: s4 kill 21114 sig 15 count 1 Jan 2 14:13:54 localhost wdmd[3800]: test failed rem 51 now 167737 ping 167718 close 167728 renewal 167657 expire 167737 client 3811 sanlock_ef4978d6-5711-4e01-a0ec-7ffbd9 cdbe5d:1 " And that in the Ovirt Gui : " 2014-janv.-02, 15:35 Operation Add-Disk failed to complete. 2014-janv.-02, 15:35 Storage Pool Manager runs on Host HOST2 (Address: X.X.X.X). 2014-janv.-02, 15:35 Invalid status on Data Center GlusterSewan. Setting Data Center status to Non Responsive (On host HOST2, Error: done). 2014-janv.-02, 15:35 State was set to Up for host HOST2. 2014-janv.-02, 15:33 Used Network resources of host HOST2 [98%] exceeded defined threshold [95%]. 2014-janv.-02, 15:33 Add-Disk operation of test_Disk1 was initiated on VM test by admin@internal. I understand that the creation of a 10 Go disk image generates a lot of traffic, but is there a way to limit it so that it doesn't have an impact on the production ? Furthermore, Why does it use so much CPU ressources ? I can see on my monitoring graph a big peak of CPU usage when I launched the operation (probably until 100%). Do you happen to notice what is consuming CPU? Since the same cluster does both virtualization and storage, a GigE network might get saturated very quickly. Is it possible to separate out the management and data/gluster traffic in this setup? Regards, Vijay
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Jan 8, 2014, at 11:55 AM, Karli Sjöberg <Karli.Sjoberg@slu.se> wrote:
Skickat från min iPhone
8 jan 2014 kl. 18:47 skrev "Darrell Budic" <darrell.budic@zenfire.com>:
Grégoire-
My test setup, running a version of the nightly self-hosted setup w/ gluster distributed/replicated disks as shared storage, in a NFS cluster:
Core i5 3570K @ 3.4Ghz, 16G Ram Boot disks: 2x 32G SATA SSDs in raid-1 Storage system: 4x500G Seagate RE3s in a ZFS raid-10 w/ 1GB ZIL & ~22G L2ARC caching from boot drives 1 1G ethernet 2 VMs running
Core2 Duo E8500 @ 3.16GHz, 8G Ram Boot disks: 2x 32G SATA SSDS in raid-1 Storage system: 2x1500G WD Green drives in a ZFS Raid w/ 1GB ZIL & ~22G L2ARC cache from boot drives 1 1G ethernet
Just curious, are you doing ZFS in Linux?
/K
Yes, forgot to mention those are freshly built Centos 6.5 systems with zfs 0.6.2, and glusterfs-3.4.1-3.el6.x86_64, vdsm-gluster-4.13.2-1.el6.noarch for testing/experimenting. Bought some cheap SSDs and just grabbed systems and platters I had around for it. Testbedding and getting some experience with the self hosted engine, since I’d like to move to it once it’s released. Also looking forward to testing native gluster on this setup. I have a production ovirt cluster with a linux zfs based NFS storage server, the backend has been very stable since I got rid of Nextenta and went to linux. Sounds odd, I know, but couldn’t get good support for a community nextenta server I inherited. I was having driver level box lockup issues with openSolaris that I couldn’t resolve. So I rebuilt it with linux, imported the pool, and haven’t looked back or had a storage failure since. -Darrell
Skickat från min iPhone
8 jan 2014 kl. 19:11 skrev "Darrell Budic" <darrell.budic@zenfire.com>:
On Jan 8, 2014, at 11:55 AM, Karli Sjöberg <Karli.Sjoberg@slu.se> wrote:
Skickat från min iPhone
8 jan 2014 kl. 18:47 skrev "Darrell Budic" <darrell.budic@zenfire.com>:
Grégoire-
My test setup, running a version of the nightly self-hosted setup w/ gluster distributed/replicated disks as shared storage, in a NFS cluster:
Core i5 3570K @ 3.4Ghz, 16G Ram Boot disks: 2x 32G SATA SSDs in raid-1 Storage system: 4x500G Seagate RE3s in a ZFS raid-10 w/ 1GB ZIL & ~22G L2ARC caching from boot drives 1 1G ethernet 2 VMs running
Core2 Duo E8500 @ 3.16GHz, 8G Ram Boot disks: 2x 32G SATA SSDS in raid-1 Storage system: 2x1500G WD Green drives in a ZFS Raid w/ 1GB ZIL & ~22G L2ARC cache from boot drives 1 1G ethernet
Just curious, are you doing ZFS in Linux?
/K
Yes, forgot to mention those are freshly built Centos 6.5 systems with zfs 0.6.2, and glusterfs-3.4.1-3.el6.x86_64, vdsm-gluster-4.13.2-1.el6.noarch for testing/experimenting. Bought some cheap SSDs and just grabbed systems and platters I had around for it. Testbedding and getting some experience with the self hosted engine, since I’d like to move to it once it’s released. Also looking forward to testing native gluster on this setup.
I have a production ovirt cluster with a linux zfs based NFS storage server, the backend has been very stable since I got rid of Nextenta and went to linux. Sounds odd, I know, but couldn’t get good support for a community nextenta server I inherited. I was having driver level box lockup issues with openSolaris that I couldn’t resolve. So I rebuilt it with linux, imported the pool, and haven’t looked back or had a storage failure since.
-Darrell
Glad to hear you got it working, inheritance is a *****:) Never considered joining the daemon side of the force (FreeBSD) ? Oh, and are you running it native with ZoL or through FUSE? /K
On Jan 8, 2014, at 3:29 PM, Karli Sjöberg <Karli.Sjoberg@slu.se> wrote:
Skickat från min iPhone
8 jan 2014 kl. 19:11 skrev "Darrell Budic" <darrell.budic@zenfire.com>:
On Jan 8, 2014, at 11:55 AM, Karli Sjöberg <Karli.Sjoberg@slu.se> wrote:
Skickat från min iPhone
8 jan 2014 kl. 18:47 skrev "Darrell Budic" <darrell.budic@zenfire.com>:
Grégoire-
My test setup, running a version of the nightly self-hosted setup w/ gluster distributed/replicated disks as shared storage, in a NFS cluster:
Core i5 3570K @ 3.4Ghz, 16G Ram Boot disks: 2x 32G SATA SSDs in raid-1 Storage system: 4x500G Seagate RE3s in a ZFS raid-10 w/ 1GB ZIL & ~22G L2ARC caching from boot drives 1 1G ethernet 2 VMs running
Core2 Duo E8500 @ 3.16GHz, 8G Ram Boot disks: 2x 32G SATA SSDS in raid-1 Storage system: 2x1500G WD Green drives in a ZFS Raid w/ 1GB ZIL & ~22G L2ARC cache from boot drives 1 1G ethernet
Just curious, are you doing ZFS in Linux?
/K
Yes, forgot to mention those are freshly built Centos 6.5 systems with zfs 0.6.2, and glusterfs-3.4.1-3.el6.x86_64, vdsm-gluster-4.13.2-1.el6.noarch for testing/experimenting. Bought some cheap SSDs and just grabbed systems and platters I had around for it. Testbedding and getting some experience with the self hosted engine, since I’d like to move to it once it’s released. Also looking forward to testing native gluster on this setup.
I have a production ovirt cluster with a linux zfs based NFS storage server, the backend has been very stable since I got rid of Nextenta and went to linux. Sounds odd, I know, but couldn’t get good support for a community nextenta server I inherited. I was having driver level box lockup issues with openSolaris that I couldn’t resolve. So I rebuilt it with linux, imported the pool, and haven’t looked back or had a storage failure since.
-Darrell
Glad to hear you got it working, inheritance is a *****:) Never considered joining the daemon side of the force (FreeBSD) ? Oh, and are you running it native with ZoL or through FUSE?
/K
Native ZoL, it’s just a big fat 10g NFS server for the host nodes. It’s been solid since Ovirt 3.1 or so. Been using linux (old Slackware hand here) too long to turn back to the daemon side ;) I use macs and OS X for the desktop to get my mach Kernel bsd-ish fix, so the even darker side? :) -Darrell
On Wed, 2014-01-08 at 16:05 -0600, Darrell Budic wrote:
On Jan 8, 2014, at 3:29 PM, Karli Sjöberg <Karli.Sjoberg@slu.se> wrote:
Skickat från min iPhone
8 jan 2014 kl. 19:11 skrev "Darrell Budic" <darrell.budic@zenfire.com>:
On Jan 8, 2014, at 11:55 AM, Karli Sjöberg <Karli.Sjoberg@slu.se> wrote:
Skickat från min iPhone
8 jan 2014 kl. 18:47 skrev "Darrell Budic" <darrell.budic@zenfire.com>:
Grégoire-
My test setup, running a version of the nightly self-hosted setup w/ gluster distributed/replicated disks as shared storage, in a NFS cluster:
Core i5 3570K @ 3.4Ghz, 16G Ram Boot disks: 2x 32G SATA SSDs in raid-1 Storage system: 4x500G Seagate RE3s in a ZFS raid-10 w/ 1GB ZIL & ~22G L2ARC caching from boot drives 1 1G ethernet 2 VMs running
Core2 Duo E8500 @ 3.16GHz, 8G Ram Boot disks: 2x 32G SATA SSDS in raid-1 Storage system: 2x1500G WD Green drives in a ZFS Raid w/ 1GB ZIL & ~22G L2ARC cache from boot drives 1 1G ethernet
Just curious, are you doing ZFS in Linux?
/K
Yes, forgot to mention those are freshly built Centos 6.5 systems with zfs 0.6.2, and glusterfs-3.4.1-3.el6.x86_64, vdsm-gluster-4.13.2-1.el6.noarch for testing/experimenting. Bought some cheap SSDs and just grabbed systems and platters I had around for it. Testbedding and getting some experience with the self hosted engine, since I’d like to move to it once it’s released. Also looking forward to testing native gluster on this setup.
I have a production ovirt cluster with a linux zfs based NFS storage server, the backend has been very stable since I got rid of Nextenta and went to linux. Sounds odd, I know, but couldn’t get good support for a community nextenta server I inherited. I was having driver level box lockup issues with openSolaris that I couldn’t resolve. So I rebuilt it with linux, imported the pool, and haven’t looked back or had a storage failure since.
-Darrell
Glad to hear you got it working, inheritance is a *****:) Never considered joining the daemon side of the force (FreeBSD) ? Oh, and are you running it native with ZoL or through FUSE?
/K
Native ZoL, it’s just a big fat 10g NFS server for the host nodes. It’s been solid since Ovirt 3.1 or so. Been using linux (old Slackware hand here) too long to turn back to the daemon side ;) I use macs and OS X for the desktop to get my mach Kernel bsd-ish fix, so the even darker side? :)
-Darrell
Funny, I used to be a Slacker as well, before I met BSD:) The leap is actually quite small, which probably isn´t that surprising since Slack is (or at least was, last time I worked with it) supposed to be the closest to Unix you get, while running Linux:) /K
Hello Darell, Le 2014-01-08 18:47, Darrell Budic a écrit :
Grégoire-
I think this is expected behavior. Well, at least the high glusterfsd CPU use during disk creation, anyway. I tried creating a 10 G disk on my test environment and observed similar high CPU usage by glusterfsd. Did the creation on the i5 system, it showed 95%-105% cpu for glusterfsd during creation, with the core2 system running ~35-65% glusterfsd utilization during the creation. Minor disk wait was observed on both systems, < 10% peak and generally < 5%. I imagine my ZFS cached backends helped a lot here. Took about 3 minutes, roughly what I’d expect for the i5’s disk system.
Does that mean that glusterfs + ovirt absolutely need to be separated so that changes on glusterfs have no negative impact on VM in production ? Here I got the problem with the creation of a preallocated disk but if tomorrow I want to change the way I replicate glusterfs bricks, I guess I'll have the same issue.
Network usage was about 45% of the 1G link. No errors or messages logged to /var/log/messages.
If checked with iftop to be more accurate, and I can see it uses more than 95% with my setup.
Depending on what your test setup looks like, I’d check my network for packet loss or errors first.
I did, I have 0 network error and 0% packet loss (for this latter, I just used ping with the ovirtmgmt interface, which showed 0% packet lost while my server was considered as down by Ovirt).
Then look at my storage setup and test pure throughput on the disks to see what you’ve got, maybe see what else is running. Did you use an NFS cluster or a PosixFS cluster for this?
I use PosixFS cluster for this. My detailed setup is : Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz (*12), 65GB Ram Disk (both boot and storage) : Perc H710 2To, hardware RAID 1 2 1G ethernet in bonding (failover) Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz (*8), 64GB Ram Disk (boot + storage) : Perc H310 1G ethernet I'm pretty sure there isn't any problem with the switch between them. To conclude : 1) About the network issue, I think it could be possible to use iptables with QoS rules on specific ports to limit GlusterFS throughput. 2) However, the CPU issue seems to be more difficult to avoid. I guess I just have to review my architecture... Thank you, Regards, Grégoire Leroy
participants (4)
-
Darrell Budic -
gregoire.leroy＠retenodus.net -
Karli Sjöberg -
Vijay Bellur