5:20 p.m.
Hello,
I have a Gluster volume in distributed/replicated mode. I have 2 hosts.
When I try to create a VM with a preallocated disk, it uses 100% of the
available CPU and bandwidth (I have 1 Gigabit network card).
The result is I can't even create a preallocated disk because the engine
detects a network failure.
I get that kind of messages in /var/log/messages :
"
Jan 2 14:13:54 localhost sanlock[3811]: 2014-01-02 14:13:54+0100 167737
[3811]: s4 kill 21114 sig 15 count 1
Jan 2 14:13:54 localhost wdmd[3800]: test failed rem 51 now 167737 ping
167718 close 167728 renewal 167657 expire 167737 client 3811
sanlock_ef4978d6-5711-4e01-a0ec-7ffbd9 cdbe5d:1
"
And that in the Ovirt Gui :
"
2014-janv.-02, 15:35 Operation Add-Disk failed to complete.
2014-janv.-02, 15:35 Storage Pool Manager runs on Host HOST2 (Address:
X.X.X.X).
2014-janv.-02, 15:35 Invalid status on Data Center GlusterSewan. Setting
Data Center status to Non Responsive (On host HOST2, Error: done).
2014-janv.-02, 15:35 State was set to Up for host HOST2.
2014-janv.-02, 15:33 Used Network resources of host HOST2 [98%] exceeded
defined threshold [95%].
2014-janv.-02, 15:33 Add-Disk operation of test_Disk1 was initiated on
VM test by admin@internal.
I understand that the creation of a 10 Go disk image generates a lot of
traffic, but is there a way to limit it so that it doesn't have an
impact on the production ? Furthermore, Why does it use so much CPU
ressources ? I can see on my monitoring graph a big peak of CPU usage
when I launched the operation (probably until 100%).
Thank you,
Regards,
Grégoire Leroy
6:42 p.m.
Adding gluster-users.
On 01/02/2014 08:50 PM, gregoire.leroy(a)retenodus.net wrote:
Hello,
I have a Gluster volume in distributed/replicated mode. I have 2 hosts.
When I try to create a VM with a preallocated disk, it uses 100% of the
available CPU and bandwidth (I have 1 Gigabit network card).
The result is I can't even create a preallocated disk because the engine
detects a network failure.
I get that kind of messages in /var/log/messages :
"
Jan 2 14:13:54 localhost sanlock[3811]: 2014-01-02 14:13:54+0100 167737
[3811]: s4 kill 21114 sig 15 count 1
Jan 2 14:13:54 localhost wdmd[3800]: test failed rem 51 now 167737 ping
167718 close 167728 renewal 167657 expire 167737 client 3811
sanlock_ef4978d6-5711-4e01-a0ec-7ffbd9 cdbe5d:1
"
And that in the Ovirt Gui :
"
2014-janv.-02, 15:35 Operation Add-Disk failed to complete.
2014-janv.-02, 15:35 Storage Pool Manager runs on Host HOST2 (Address:
X.X.X.X).
2014-janv.-02, 15:35 Invalid status on Data Center GlusterSewan. Setting
Data Center status to Non Responsive (On host HOST2, Error: done).
2014-janv.-02, 15:35 State was set to Up for host HOST2.
2014-janv.-02, 15:33 Used Network resources of host HOST2 [98%] exceeded
defined threshold [95%].
2014-janv.-02, 15:33 Add-Disk operation of test_Disk1 was initiated on
VM test by admin@internal.
I understand that the creation of a 10 Go disk image generates a lot of
traffic, but is there a way to limit it so that it doesn't have an
impact on the production ? Furthermore, Why does it use so much CPU
ressources ? I can see on my monitoring graph a big peak of CPU usage
when I launched the operation (probably until 100%).
Do you happen to notice what is consuming CPU? Since the same cluster
does both virtualization and storage, a GigE network might get saturated
very quickly. Is it possible to separate out the management and
data/gluster traffic in this setup?
Regards,
Vijay
7:38 p.m.
Do you happen to notice what is consuming CPU?
When I check with top, glusterfs and glusterfsd are the only process who
use a significant amount of CPU. Load average is between 5 and 6, and I
don't have any started VM.
Since the same cluster
does both virtualization and storage, a GigE network might get
saturated very quickly. Is it possible to separate out the management
and data/gluster traffic in this setup?
Unfortunately, it's not possible. I only have two hosts, both for
virtualization and storage. Why does glusterfs use so much CPU ?
About the network traffic, the creation of a pre-allocated disk is not
something which absolutely needs to be fast : if it takes 5 minutes,
well... it takes 5 minutes. I guess that if there's a way to limit the
bandwidth available for glusterfs, it'll still have an impact on
production because glusterfs and VM will be in competition for
bandwidth, right ? Is there a way to limit the bandwidth only for the
creation of a pre-allocated disk ?
Thank you,
Regards,
Grégoire Leroy
3:49 p.m.
Hello,
Do you need more informations about this issue ? Do you think this
problem is likely to show up in other cases ? I mean, is that an
expected behaviour with my environment, or is it unexpected ?
Is there a way to limit the bandwidth usage for creation of
pre-allocated disk so that it doesn't impact production ?
Thank you,
Regards,
Grégoire
Le 2014-01-02 17:42, Vijay Bellur a écrit :
Adding gluster-users.
On 01/02/2014 08:50 PM, gregoire.leroy(a)retenodus.net wrote:
> Hello,
>
> I have a Gluster volume in distributed/replicated mode. I have 2
> hosts.
> When I try to create a VM with a preallocated disk, it uses 100% of
> the
> available CPU and bandwidth (I have 1 Gigabit network card).
> The result is I can't even create a preallocated disk because the
> engine
> detects a network failure.
>
> I get that kind of messages in /var/log/messages :
> "
> Jan 2 14:13:54 localhost sanlock[3811]: 2014-01-02 14:13:54+0100
> 167737
> [3811]: s4 kill 21114 sig 15 count 1
> Jan 2 14:13:54 localhost wdmd[3800]: test failed rem 51 now 167737
> ping
> 167718 close 167728 renewal 167657 expire 167737 client 3811
> sanlock_ef4978d6-5711-4e01-a0ec-7ffbd9 cdbe5d:1
> "
>
> And that in the Ovirt Gui :
> "
> 2014-janv.-02, 15:35 Operation Add-Disk failed to complete.
> 2014-janv.-02, 15:35 Storage Pool Manager runs on Host HOST2 (Address:
> X.X.X.X).
> 2014-janv.-02, 15:35 Invalid status on Data Center GlusterSewan.
> Setting
> Data Center status to Non Responsive (On host HOST2, Error: done).
> 2014-janv.-02, 15:35 State was set to Up for host HOST2.
> 2014-janv.-02, 15:33 Used Network resources of host HOST2 [98%]
> exceeded
> defined threshold [95%].
> 2014-janv.-02, 15:33 Add-Disk operation of test_Disk1 was initiated on
> VM test by admin@internal.
>
> I understand that the creation of a 10 Go disk image generates a lot
> of
> traffic, but is there a way to limit it so that it doesn't have an
> impact on the production ? Furthermore, Why does it use so much CPU
> ressources ? I can see on my monitoring graph a big peak of CPU usage
> when I launched the operation (probably until 100%).
Do you happen to notice what is consuming CPU? Since the same cluster
does both virtualization and storage, a GigE network might get
saturated very quickly. Is it possible to separate out the management
and data/gluster traffic in this setup?
Regards,
Vijay
7:47 p.m.
Grégoire-
I think this is expected behavior. Well, at least the high glusterfsd CPU use during disk
creation, anyway. I tried creating a 10 G disk on my test environment and observed similar
high CPU usage by glusterfsd. Did the creation on the i5 system, it showed 95%-105% cpu
for glusterfsd during creation, with the core2 system running ~35-65% glusterfsd
utilization during the creation. Minor disk wait was observed on both systems, < 10%
peak and generally < 5%. I imagine my ZFS cached backends helped a lot here. Took about
3 minutes, roughly what I’d expect for the i5’s disk system. Network usage was about 45%
of the 1G link. No errors or messages logged to /var/log/messages.
Depending on what your test setup looks like, I’d check my network for packet loss or
errors first. Then look at my storage setup and test pure throughput on the disks to see
what you’ve got, maybe see what else is running. Did you use an NFS cluster or a PosixFS
cluster for this?
My test setup, running a version of the nightly self-hosted setup w/ gluster
distributed/replicated disks as shared storage, in a NFS cluster:
Core i5 3570K @ 3.4Ghz, 16G Ram
Boot disks: 2x 32G SATA SSDs in raid-1
Storage system: 4x500G Seagate RE3s in a ZFS raid-10 w/ 1GB ZIL & ~22G L2ARC caching
from boot drives
1 1G ethernet
2 VMs running
Core2 Duo E8500 @ 3.16GHz, 8G Ram
Boot disks: 2x 32G SATA SSDS in raid-1
Storage system: 2x1500G WD Green drives in a ZFS Raid w/ 1GB ZIL & ~22G L2ARC cache
from boot drives
1 1G ethernet
They are connected through a Netgear Prosafe+ workgroup style switch, not much going on
between them.
-Darrell
On Jan 8, 2014, at 7:49 AM, gregoire.leroy(a)retenodus.net wrote:
Hello,
Do you need more informations about this issue ? Do you think this problem is likely to
show up in other cases ? I mean, is that an expected behaviour with my environment, or is
it unexpected ?
Is there a way to limit the bandwidth usage for creation of pre-allocated disk so that it
doesn't impact production ?
Thank you,
Regards,
Grégoire
Le 2014-01-02 17:42, Vijay Bellur a écrit :
> Adding gluster-users.
> On 01/02/2014 08:50 PM, gregoire.leroy(a)retenodus.net wrote:
>> Hello,
>> I have a Gluster volume in distributed/replicated mode. I have 2 hosts.
>> When I try to create a VM with a preallocated disk, it uses 100% of the
>> available CPU and bandwidth (I have 1 Gigabit network card).
>> The result is I can't even create a preallocated disk because the engine
>> detects a network failure.
>> I get that kind of messages in /var/log/messages :
>> "
>> Jan 2 14:13:54 localhost sanlock[3811]: 2014-01-02 14:13:54+0100 167737
>> [3811]: s4 kill 21114 sig 15 count 1
>> Jan 2 14:13:54 localhost wdmd[3800]: test failed rem 51 now 167737 ping
>> 167718 close 167728 renewal 167657 expire 167737 client 3811
>> sanlock_ef4978d6-5711-4e01-a0ec-7ffbd9 cdbe5d:1
>> "
>> And that in the Ovirt Gui :
>> "
>> 2014-janv.-02, 15:35 Operation Add-Disk failed to complete.
>> 2014-janv.-02, 15:35 Storage Pool Manager runs on Host HOST2 (Address:
>> X.X.X.X).
>> 2014-janv.-02, 15:35 Invalid status on Data Center GlusterSewan. Setting
>> Data Center status to Non Responsive (On host HOST2, Error: done).
>> 2014-janv.-02, 15:35 State was set to Up for host HOST2.
>> 2014-janv.-02, 15:33 Used Network resources of host HOST2 [98%] exceeded
>> defined threshold [95%].
>> 2014-janv.-02, 15:33 Add-Disk operation of test_Disk1 was initiated on
>> VM test by admin@internal.
>> I understand that the creation of a 10 Go disk image generates a lot of
>> traffic, but is there a way to limit it so that it doesn't have an
>> impact on the production ? Furthermore, Why does it use so much CPU
>> ressources ? I can see on my monitoring graph a big peak of CPU usage
>> when I launched the operation (probably until 100%).
> Do you happen to notice what is consuming CPU? Since the same cluster
> does both virtualization and storage, a GigE network might get
> saturated very quickly. Is it possible to separate out the management
> and data/gluster traffic in this setup?
> Regards,
> Vijay
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
7:55 p.m.
Skickat från min iPhone
8 jan 2014 kl. 18:47 skrev "Darrell Budic"
<darrell.budic(a)zenfire.com>:
Grégoire-
I think this is expected behavior. Well, at least the high glusterfsd CPU use during disk
creation, anyway. I tried creating a 10 G disk on my test environment and observed similar
high CPU usage by glusterfsd. Did the creation on the i5 system, it showed 95%-105% cpu
for glusterfsd during creation, with the core2 system running ~35-65% glusterfsd
utilization during the creation. Minor disk wait was observed on both systems, < 10%
peak and generally < 5%. I imagine my ZFS cached backends helped a lot here. Took about
3 minutes, roughly what I’d expect for the i5’s disk system. Network usage was about 45%
of the 1G link. No errors or messages logged to /var/log/messages.
Depending on what your test setup looks like, I’d check my network for packet loss or
errors first. Then look at my storage setup and test pure throughput on the disks to see
what you’ve got, maybe see what else is running. Did you use an NFS cluster or a PosixFS
cluster for this?
My test setup, running a version of the nightly self-hosted setup w/ gluster
distributed/replicated disks as shared storage, in a NFS cluster:
Core i5 3570K @ 3.4Ghz, 16G Ram
Boot disks: 2x 32G SATA SSDs in raid-1
Storage system: 4x500G Seagate RE3s in a ZFS raid-10 w/ 1GB ZIL & ~22G L2ARC caching
from boot drives
1 1G ethernet
2 VMs running
Core2 Duo E8500 @ 3.16GHz, 8G Ram
Boot disks: 2x 32G SATA SSDS in raid-1
Storage system: 2x1500G WD Green drives in a ZFS Raid w/ 1GB ZIL & ~22G L2ARC cache
from boot drives
1 1G ethernet
They are connected through a Netgear Prosafe+ workgroup style switch, not much going on
between them.
-Darrell
Just curious, are you doing ZFS in Linux?
/K
> On Jan 8, 2014, at 7:49 AM, gregoire.leroy(a)retenodus.net wrote:
>
> Hello,
>
> Do you need more informations about this issue ? Do you think this problem is likely
to show up in other cases ? I mean, is that an expected behaviour with my environment, or
is it unexpected ?
>
> Is there a way to limit the bandwidth usage for creation of pre-allocated disk so
that it doesn't impact production ?
>
> Thank you,
> Regards,
> Grégoire
>
> Le 2014-01-02 17:42, Vijay Bellur a écrit :
>> Adding gluster-users.
>> On 01/02/2014 08:50 PM, gregoire.leroy(a)retenodus.net wrote:
>>> Hello,
>>> I have a Gluster volume in distributed/replicated mode. I have 2 hosts.
>>> When I try to create a VM with a preallocated disk, it uses 100% of the
>>> available CPU and bandwidth (I have 1 Gigabit network card).
>>> The result is I can't even create a preallocated disk because the engine
>>> detects a network failure.
>>> I get that kind of messages in /var/log/messages :
>>> "
>>> Jan 2 14:13:54 localhost sanlock[3811]: 2014-01-02 14:13:54+0100 167737
>>> [3811]: s4 kill 21114 sig 15 count 1
>>> Jan 2 14:13:54 localhost wdmd[3800]: test failed rem 51 now 167737 ping
>>> 167718 close 167728 renewal 167657 expire 167737 client 3811
>>> sanlock_ef4978d6-5711-4e01-a0ec-7ffbd9 cdbe5d:1
>>> "
>>> And that in the Ovirt Gui :
>>> "
>>> 2014-janv.-02, 15:35 Operation Add-Disk failed to complete.
>>> 2014-janv.-02, 15:35 Storage Pool Manager runs on Host HOST2 (Address:
>>> X.X.X.X).
>>> 2014-janv.-02, 15:35 Invalid status on Data Center GlusterSewan. Setting
>>> Data Center status to Non Responsive (On host HOST2, Error: done).
>>> 2014-janv.-02, 15:35 State was set to Up for host HOST2.
>>> 2014-janv.-02, 15:33 Used Network resources of host HOST2 [98%] exceeded
>>> defined threshold [95%].
>>> 2014-janv.-02, 15:33 Add-Disk operation of test_Disk1 was initiated on
>>> VM test by admin@internal.
>>> I understand that the creation of a 10 Go disk image generates a lot of
>>> traffic, but is there a way to limit it so that it doesn't have an
>>> impact on the production ? Furthermore, Why does it use so much CPU
>>> ressources ? I can see on my monitoring graph a big peak of CPU usage
>>> when I launched the operation (probably until 100%).
>> Do you happen to notice what is consuming CPU? Since the same cluster
>> does both virtualization and storage, a GigE network might get
>> saturated very quickly. Is it possible to separate out the management
>> and data/gluster traffic in this setup?
>> Regards,
>> Vijay
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
8:11 p.m.
On Jan 8, 2014, at 11:55 AM, Karli Sjöberg <Karli.Sjoberg(a)slu.se> wrote:
Skickat från min iPhone
> 8 jan 2014 kl. 18:47 skrev "Darrell Budic"
<darrell.budic(a)zenfire.com>:
>
> Grégoire-
>
> My test setup, running a version of the nightly self-hosted setup w/ gluster
distributed/replicated disks as shared storage, in a NFS cluster:
>
> Core i5 3570K @ 3.4Ghz, 16G Ram
> Boot disks: 2x 32G SATA SSDs in raid-1
> Storage system: 4x500G Seagate RE3s in a ZFS raid-10 w/ 1GB ZIL & ~22G L2ARC
caching from boot drives
> 1 1G ethernet
> 2 VMs running
>
> Core2 Duo E8500 @ 3.16GHz, 8G Ram
> Boot disks: 2x 32G SATA SSDS in raid-1
> Storage system: 2x1500G WD Green drives in a ZFS Raid w/ 1GB ZIL & ~22G L2ARC
cache from boot drives
> 1 1G ethernet
Just curious, are you doing ZFS in Linux?
/K
Yes, forgot to mention those are freshly built Centos 6.5 systems with zfs 0.6.2, and
glusterfs-3.4.1-3.el6.x86_64, vdsm-gluster-4.13.2-1.el6.noarch for testing/experimenting.
Bought some cheap SSDs and just grabbed systems and platters I had around for it.
Testbedding and getting some experience with the self hosted engine, since I’d like to
move to it once it’s released. Also looking forward to testing native gluster on this
setup.
I have a production ovirt cluster with a linux zfs based NFS storage server, the backend
has been very stable since I got rid of Nextenta and went to linux. Sounds odd, I know,
but couldn’t get good support for a community nextenta server I inherited. I was having
driver level box lockup issues with openSolaris that I couldn’t resolve. So I rebuilt it
with linux, imported the pool, and haven’t looked back or had a storage failure since.
-Darrell
11:29 p.m.
Skickat från min iPhone
8 jan 2014 kl. 19:11 skrev "Darrell Budic"
<darrell.budic(a)zenfire.com>:
> On Jan 8, 2014, at 11:55 AM, Karli Sjöberg <Karli.Sjoberg(a)slu.se> wrote:
>
>
>
> Skickat från min iPhone
>
>> 8 jan 2014 kl. 18:47 skrev "Darrell Budic"
<darrell.budic(a)zenfire.com>:
>>
>> Grégoire-
>>
>> My test setup, running a version of the nightly self-hosted setup w/ gluster
distributed/replicated disks as shared storage, in a NFS cluster:
>>
>> Core i5 3570K @ 3.4Ghz, 16G Ram
>> Boot disks: 2x 32G SATA SSDs in raid-1
>> Storage system: 4x500G Seagate RE3s in a ZFS raid-10 w/ 1GB ZIL & ~22G L2ARC
caching from boot drives
>> 1 1G ethernet
>> 2 VMs running
>>
>> Core2 Duo E8500 @ 3.16GHz, 8G Ram
>> Boot disks: 2x 32G SATA SSDS in raid-1
>> Storage system: 2x1500G WD Green drives in a ZFS Raid w/ 1GB ZIL & ~22G
L2ARC cache from boot drives
>> 1 1G ethernet
>
> Just curious, are you doing ZFS in Linux?
>
> /K
Yes, forgot to mention those are freshly built Centos 6.5 systems with zfs 0.6.2, and
glusterfs-3.4.1-3.el6.x86_64, vdsm-gluster-4.13.2-1.el6.noarch for testing/experimenting.
Bought some cheap SSDs and just grabbed systems and platters I had around for it.
Testbedding and getting some experience with the self hosted engine, since I’d like to
move to it once it’s released. Also looking forward to testing native gluster on this
setup.
I have a production ovirt cluster with a linux zfs based NFS storage server, the backend
has been very stable since I got rid of Nextenta and went to linux. Sounds odd, I know,
but couldn’t get good support for a community nextenta server I inherited. I was having
driver level box lockup issues with openSolaris that I couldn’t resolve. So I rebuilt it
with linux, imported the pool, and haven’t looked back or had a storage failure since.
-Darrell
Glad to hear you got it working, inheritance is a *****:) Never considered joining the
daemon side of the force (FreeBSD) ? Oh, and are you running it native with ZoL or through
FUSE?
/K
12:05 a.m.
On Jan 8, 2014, at 3:29 PM, Karli Sjöberg <Karli.Sjoberg(a)slu.se> wrote:
Skickat från min iPhone
> 8 jan 2014 kl. 19:11 skrev "Darrell Budic"
<darrell.budic(a)zenfire.com>:
>
>
>> On Jan 8, 2014, at 11:55 AM, Karli Sjöberg <Karli.Sjoberg(a)slu.se> wrote:
>>
>>
>>
>> Skickat från min iPhone
>>
>>> 8 jan 2014 kl. 18:47 skrev "Darrell Budic"
<darrell.budic(a)zenfire.com>:
>>>
>>> Grégoire-
>>>
>>> My test setup, running a version of the nightly self-hosted setup w/ gluster
distributed/replicated disks as shared storage, in a NFS cluster:
>>>
>>> Core i5 3570K @ 3.4Ghz, 16G Ram
>>> Boot disks: 2x 32G SATA SSDs in raid-1
>>> Storage system: 4x500G Seagate RE3s in a ZFS raid-10 w/ 1GB ZIL & ~22G
L2ARC caching from boot drives
>>> 1 1G ethernet
>>> 2 VMs running
>>>
>>> Core2 Duo E8500 @ 3.16GHz, 8G Ram
>>> Boot disks: 2x 32G SATA SSDS in raid-1
>>> Storage system: 2x1500G WD Green drives in a ZFS Raid w/ 1GB ZIL & ~22G
L2ARC cache from boot drives
>>> 1 1G ethernet
>>
>> Just curious, are you doing ZFS in Linux?
>>
>> /K
>
> Yes, forgot to mention those are freshly built Centos 6.5 systems with zfs 0.6.2, and
glusterfs-3.4.1-3.el6.x86_64, vdsm-gluster-4.13.2-1.el6.noarch for testing/experimenting.
Bought some cheap SSDs and just grabbed systems and platters I had around for it.
Testbedding and getting some experience with the self hosted engine, since I’d like to
move to it once it’s released. Also looking forward to testing native gluster on this
setup.
>
> I have a production ovirt cluster with a linux zfs based NFS storage server, the
backend has been very stable since I got rid of Nextenta and went to linux. Sounds odd, I
know, but couldn’t get good support for a community nextenta server I inherited. I was
having driver level box lockup issues with openSolaris that I couldn’t resolve. So I
rebuilt it with linux, imported the pool, and haven’t looked back or had a storage failure
since.
>
> -Darrell
>
Glad to hear you got it working, inheritance is a *****:) Never considered joining the
daemon side of the force (FreeBSD) ? Oh, and are you running it native with ZoL or through
FUSE?
/K
Native ZoL, it’s just a big fat 10g NFS server for the host nodes. It’s been solid since
Ovirt 3.1 or so. Been using linux (old Slackware hand here) too long to turn back to the
daemon side ;) I use macs and OS X for the desktop to get my mach Kernel bsd-ish fix, so
the even darker side? :)
-Darrell
9:35 a.m.
On Wed, 2014-01-08 at 16:05 -0600, Darrell Budic wrote:
On Jan 8, 2014, at 3:29 PM, Karli Sjöberg
<Karli.Sjoberg(a)slu.se> wrote:
>
>
> Skickat från min iPhone
>
>> 8 jan 2014 kl. 19:11 skrev "Darrell Budic"
<darrell.budic(a)zenfire.com>:
>>
>>
>>> On Jan 8, 2014, at 11:55 AM, Karli Sjöberg <Karli.Sjoberg(a)slu.se>
wrote:
>>>
>>>
>>>
>>> Skickat från min iPhone
>>>
>>>> 8 jan 2014 kl. 18:47 skrev "Darrell Budic"
<darrell.budic(a)zenfire.com>:
>>>>
>>>> Grégoire-
>>>>
>>>> My test setup, running a version of the nightly self-hosted setup w/
gluster distributed/replicated disks as shared storage, in a NFS cluster:
>>>>
>>>> Core i5 3570K @ 3.4Ghz, 16G Ram
>>>> Boot disks: 2x 32G SATA SSDs in raid-1
>>>> Storage system: 4x500G Seagate RE3s in a ZFS raid-10 w/ 1GB ZIL &
~22G L2ARC caching from boot drives
>>>> 1 1G ethernet
>>>> 2 VMs running
>>>>
>>>> Core2 Duo E8500 @ 3.16GHz, 8G Ram
>>>> Boot disks: 2x 32G SATA SSDS in raid-1
>>>> Storage system: 2x1500G WD Green drives in a ZFS Raid w/ 1GB ZIL &
~22G L2ARC cache from boot drives
>>>> 1 1G ethernet
>>>
>>> Just curious, are you doing ZFS in Linux?
>>>
>>> /K
>>
>> Yes, forgot to mention those are freshly built Centos 6.5 systems with zfs
0.6.2, and glusterfs-3.4.1-3.el6.x86_64, vdsm-gluster-4.13.2-1.el6.noarch for
testing/experimenting. Bought some cheap SSDs and just grabbed systems and platters I had
around for it. Testbedding and getting some experience with the self hosted engine, since
I’d like to move to it once it’s released. Also looking forward to testing native gluster
on this setup.
>>
>> I have a production ovirt cluster with a linux zfs based NFS storage server, the
backend has been very stable since I got rid of Nextenta and went to linux. Sounds odd, I
know, but couldn’t get good support for a community nextenta server I inherited. I was
having driver level box lockup issues with openSolaris that I couldn’t resolve. So I
rebuilt it with linux, imported the pool, and haven’t looked back or had a storage failure
since.
>>
>> -Darrell
>>
>
> Glad to hear you got it working, inheritance is a *****:) Never considered joining
the daemon side of the force (FreeBSD) ? Oh, and are you running it native with ZoL or
through FUSE?
>
> /K
Native ZoL, it’s just a big fat 10g NFS server for the host nodes. It’s been solid since
Ovirt 3.1 or so. Been using linux (old Slackware hand here) too long to turn back to the
daemon side ;) I use macs and OS X for the desktop to get my mach Kernel bsd-ish fix, so
the even darker side? :)
-Darrell
Funny, I used to be a Slacker as well, before I met BSD:) The leap is
actually quite small, which probably isn´t that surprising since Slack
is (or at least was, last time I worked with it) supposed to be the
closest to Unix you get, while running Linux:)
/K
7:34 p.m.
New subject: [Users] [Gluster-users] Creation of preallocated disk with Gluster replication
Hello Darell,
Le 2014-01-08 18:47, Darrell Budic a écrit :
Grégoire-
I think this is expected behavior. Well, at least the high glusterfsd
CPU use during disk creation, anyway. I tried creating a 10 G disk on
my test environment and observed similar high CPU usage by glusterfsd.
Did the creation on the i5 system, it showed 95%-105% cpu for
glusterfsd during creation, with the core2 system running ~35-65%
glusterfsd utilization during the creation. Minor disk wait was
observed on both systems, < 10% peak and generally < 5%. I imagine my
ZFS cached backends helped a lot here. Took about 3 minutes, roughly
what I’d expect for the i5’s disk system.
Does that mean that glusterfs + ovirt absolutely need to be separated so
that changes on glusterfs have no negative impact on VM in production ?
Here I got the problem with the creation of a preallocated disk but if
tomorrow I want to change the way I replicate glusterfs bricks, I guess
I'll have the same issue.
Network usage was about 45%
of the 1G link. No errors or messages logged to /var/log/messages.
If checked with iftop to be more accurate, and I can see it uses more
than 95% with my setup.
Depending on what your test setup looks like, I’d check my network
for
packet loss or errors first.
I did, I have 0 network error and 0% packet loss (for this latter, I
just used ping with the ovirtmgmt interface, which showed 0% packet lost
while my server was considered as down by Ovirt).
Then look at my storage setup and test
pure throughput on the disks to see what you’ve got, maybe see what
else is running. Did you use an NFS cluster or a PosixFS cluster for
this?
I use PosixFS cluster for this.
My detailed setup is :
Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz (*12), 65GB Ram
Disk (both boot and storage) : Perc H710 2To, hardware RAID 1
2 1G ethernet in bonding (failover)
Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz (*8), 64GB Ram
Disk (boot + storage) : Perc H310
1G ethernet
I'm pretty sure there isn't any problem with the switch between them.
To conclude :
1) About the network issue, I think it could be possible to use iptables
with QoS rules on specific ports to limit GlusterFS throughput.
2) However, the CPU issue seems to be more difficult to avoid. I guess I
just have to review my architecture...
Thank you,
Regards,
Grégoire Leroy
3768
days inactive
3775
days old
10 comments
4 participants
participants (4)
-
Darrell Budic -
gregoire.leroy＠retenodus.net -
Karli Sjöberg -
Vijay Bellur