
3 Mar
2017
3 Mar
'17
7 p.m.
--Apple-Mail=_33A5E632-5864-4B33-9FED-1901AA57DEBD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Why are you using an arbitrator if all your HW configs are identical? = I=E2=80=99d use a true replica 3 in this case. Also in my experience with gluster and vm hosting, the ZIL/slog degrades = write performance unless it=E2=80=99s a truly dedicated disk. But I have = 8 spinners backing my ZFS volumes, so trying to share a sata disk = wasn=E2=80=99t a good zil. If yours is dedicated SAS, keep it, if it=E2=80= =99s SATA, try testing without it. You don=E2=80=99t have compression enabled on your zfs volume, and I=E2=80= =99d recommend enabling relatime on it. Depending on the amount of RAM = in these boxes, you probably want to limit your zfs arc size to 8G or so = (1/4 total ram or less). Gluster just works volumes hard during a = rebuild, what=E2=80=99s the problem you=E2=80=99re seeing? If it=E2=80=99s= affecting your VMs, using shading and tuning client & server threads = can help avoid interruptions to your VMs while repairs are running. If = you really need to limit it, you can use cgroups to keep it from hogging = all the CPU, but it takes longer to heal, of course. There are a couple = older posts and blogs about it, if you go back a while. > On Mar 3, 2017, at 9:02 AM, Arman Khalatyan <arm2arm@gmail.com> wrote: >=20 > The problem itself is not the streaming data performance., and also dd = zero does not help much in the production zfs running with compression. > the main problem comes when the gluster is starting to do something = with that, it is using xattrs, probably accessing extended attributes = inside the zfs is slower than XFS. > Also primitive find file or ls -l in the (dot)gluster folders takes = ages:=20 >=20 > now I can see that arbiter host has almost 100% cache miss during the = rebuild, which is actually natural while he is reading always the new = datasets: > [root@clei26 ~]# arcstat.py 1 > time read miss miss% dmis dm% pmis pm% mmis mm% arcsz = c =20 > 15:57:31 29 29 100 29 100 0 0 29 100 685M = 31G =20 > 15:57:32 530 476 89 476 89 0 0 457 89 685M = 31G =20 > 15:57:33 480 467 97 467 97 0 0 463 97 685M = 31G =20 > 15:57:34 452 443 98 443 98 0 0 435 97 685M = 31G =20 > 15:57:35 582 547 93 547 93 0 0 536 94 685M = 31G =20 > 15:57:36 439 417 94 417 94 0 0 393 94 685M = 31G =20 > 15:57:38 435 392 90 392 90 0 0 374 89 685M = 31G =20 > 15:57:39 364 352 96 352 96 0 0 352 96 685M = 31G =20 > 15:57:40 408 375 91 375 91 0 0 360 91 685M = 31G =20 > 15:57:41 552 539 97 539 97 0 0 539 97 685M = 31G =20 >=20 > It looks like we cannot have in the same system performance and = reliability :( > Simply final conclusion is with the single disk+ssd even zfs doesnot = help to speedup the glusterfs healing. > I will stop here:) >=20 >=20 >=20 >=20 > On Fri, Mar 3, 2017 at 3:35 PM, Juan Pablo <pablo.localhost@gmail.com = <mailto:pablo.localhost@gmail.com>> wrote: > cd to inside the pool path > then dd if=3D/dev/zero of=3Dtest.tt <http://test.tt/> bs=3D1M=20 > leave it runing 5/10 minutes. > do ctrl+c paste result here. > etc. >=20 > 2017-03-03 11:30 GMT-03:00 Arman Khalatyan <arm2arm@gmail.com = <mailto:arm2arm@gmail.com>>: > No, I have one pool made of the one disk and ssd as a cache and log = device. > I have 3 Glusterfs bricks- separate 3 hosts:Volume type Replicate = (Arbiter)=3D replica 2+1! > That how much you can push into compute nodes(they have only 3 disk = slots). >=20 >=20 > On Fri, Mar 3, 2017 at 3:19 PM, Juan Pablo <pablo.localhost@gmail.com = <mailto:pablo.localhost@gmail.com>> wrote: > ok, you have 3 pools, zclei22, logs and cache, thats wrong. you should = have 1 pool, with zlog+cache if you are looking for performance. > also, dont mix drives.=20 > whats the performance issue you are facing?=20 >=20 >=20 > regards, >=20 > 2017-03-03 11:00 GMT-03:00 Arman Khalatyan <arm2arm@gmail.com = <mailto:arm2arm@gmail.com>>: > This is CentOS 7.3 ZoL version 0.6.5.9-1 >=20 > [root@clei22 ~]# lsscsi >=20 > [2:0:0:0] disk ATA INTEL SSDSC2CW24 400i /dev/sda >=20 > [3:0:0:0] disk ATA HGST HUS724040AL AA70 /dev/sdb >=20 > [4:0:0:0] disk ATA WDC WD2002FYPS-0 1G01 /dev/sdc >=20 >=20 >=20 > [root@clei22 ~]# pvs ;vgs;lvs >=20 > PV VG Fmt = Attr PSize PFree >=20 > /dev/mapper/INTEL_SSDSC2CW240A3_CVCV306302RP240CGN vg_cache = lvm2 a-- 223.57g 0 >=20 > /dev/sdc2 centos_clei22 = lvm2 a-- 1.82t 64.00m >=20 > VG #PV #LV #SN Attr VSize VFree >=20 > centos_clei22 1 3 0 wz--n- 1.82t 64.00m >=20 > vg_cache 1 2 0 wz--n- 223.57g 0 >=20 > LV VG Attr LSize Pool Origin Data% Meta% = Move Log Cpy%Sync Convert >=20 > home centos_clei22 -wi-ao---- 1.74t = =20 >=20 > root centos_clei22 -wi-ao---- 50.00g = =20 >=20 > swap centos_clei22 -wi-ao---- 31.44g = =20 >=20 > lv_cache vg_cache -wi-ao---- 213.57g = =20 >=20 > lv_slog vg_cache -wi-ao---- 10.00g =20 >=20 >=20 >=20 > [root@clei22 ~]# zpool status -v >=20 > pool: zclei22 >=20 > state: ONLINE >=20 > scan: scrub repaired 0 in 0h0m with 0 errors on Tue Feb 28 14:16:07 = 2017 >=20 > config: >=20 >=20 >=20 > NAME STATE READ WRITE CKSUM >=20 > zclei22 ONLINE 0 0 0 >=20 > HGST_HUS724040ALA640_PN2334PBJ4SV6T1 ONLINE 0 0 0 >=20 > logs >=20 > lv_slog ONLINE 0 0 0 >=20 > cache >=20 > lv_cache ONLINE 0 0 0 >=20 >=20 >=20 > errors: No known data errors >=20 >=20 > ZFS config: >=20 > [root@clei22 ~]# zfs get all zclei22/01 >=20 > NAME PROPERTY VALUE SOURCE >=20 > zclei22/01 type filesystem - >=20 > zclei22/01 creation Tue Feb 28 14:06 2017 - >=20 > zclei22/01 used 389G - >=20 > zclei22/01 available 3.13T - >=20 > zclei22/01 referenced 389G - >=20 > zclei22/01 compressratio 1.01x - >=20 > zclei22/01 mounted yes - >=20 > zclei22/01 quota none default >=20 > zclei22/01 reservation none default >=20 > zclei22/01 recordsize 128K local >=20 > zclei22/01 mountpoint /zclei22/01 default >=20 > zclei22/01 sharenfs off default >=20 > zclei22/01 checksum on default >=20 > zclei22/01 compression off local >=20 > zclei22/01 atime on default >=20 > zclei22/01 devices on default >=20 > zclei22/01 exec on default >=20 > zclei22/01 setuid on default >=20 > zclei22/01 readonly off default >=20 > zclei22/01 zoned off default >=20 > zclei22/01 snapdir hidden default >=20 > zclei22/01 aclinherit restricted default >=20 > zclei22/01 canmount on default >=20 > zclei22/01 xattr sa local >=20 > zclei22/01 copies 1 default >=20 > zclei22/01 version 5 - >=20 > zclei22/01 utf8only off - >=20 > zclei22/01 normalization none - >=20 > zclei22/01 casesensitivity sensitive - >=20 > zclei22/01 vscan off default >=20 > zclei22/01 nbmand off default >=20 > zclei22/01 sharesmb off default >=20 > zclei22/01 refquota none default >=20 > zclei22/01 refreservation none default >=20 > zclei22/01 primarycache metadata local >=20 > zclei22/01 secondarycache metadata local >=20 > zclei22/01 usedbysnapshots 0 - >=20 > zclei22/01 usedbydataset 389G - >=20 > zclei22/01 usedbychildren 0 - >=20 > zclei22/01 usedbyrefreservation 0 - >=20 > zclei22/01 logbias latency default >=20 > zclei22/01 dedup off default >=20 > zclei22/01 mlslabel none default >=20 > zclei22/01 sync disabled local >=20 > zclei22/01 refcompressratio 1.01x - >=20 > zclei22/01 written 389G - >=20 > zclei22/01 logicalused 396G - >=20 > zclei22/01 logicalreferenced 396G - >=20 > zclei22/01 filesystem_limit none default >=20 > zclei22/01 snapshot_limit none default >=20 > zclei22/01 filesystem_count none default >=20 > zclei22/01 snapshot_count none default >=20 > zclei22/01 snapdev hidden default >=20 > zclei22/01 acltype off default >=20 > zclei22/01 context none default >=20 > zclei22/01 fscontext none default >=20 > zclei22/01 defcontext none default >=20 > zclei22/01 rootcontext none default >=20 > zclei22/01 relatime off default >=20 > zclei22/01 redundant_metadata all default >=20 > zclei22/01 overlay off default >=20 >=20 >=20 >=20 >=20 >=20 > On Fri, Mar 3, 2017 at 2:52 PM, Juan Pablo <pablo.localhost@gmail.com = <mailto:pablo.localhost@gmail.com>> wrote: > Which operating system version are you using for your zfs storage?=20 > do: > zfs get all your-pool-name > use arc_summary.py from freenas git repo if you wish. >=20 >=20 > 2017-03-03 10:33 GMT-03:00 Arman Khalatyan <arm2arm@gmail.com = <mailto:arm2arm@gmail.com>>: > Pool load: > [root@clei21 ~]# zpool iostat -v 1=20 > capacity operations = bandwidth > pool alloc free read write = read write > -------------------------------------- ----- ----- ----- ----- = ----- ----- > zclei21 10.1G 3.62T 0 112 = 823 8.82M > HGST_HUS724040ALA640_PN2334PBJ52XWT1 10.1G 3.62T 0 46 = 626 4.40M > logs - - - - = - - > lv_slog 225M 9.72G 0 66 = 198 4.45M > cache - - - - = - - > lv_cache 9.81G 204G 0 46 = 56 4.13M > -------------------------------------- ----- ----- ----- ----- = ----- ----- >=20 > capacity operations = bandwidth > pool alloc free read write = read write > -------------------------------------- ----- ----- ----- ----- = ----- ----- > zclei21 10.1G 3.62T 0 191 = 0 12.8M > HGST_HUS724040ALA640_PN2334PBJ52XWT1 10.1G 3.62T 0 0 = 0 0 > logs - - - - = - - > lv_slog 225M 9.72G 0 191 = 0 12.8M > cache - - - - = - - > lv_cache 9.83G 204G 0 218 = 0 20.0M > -------------------------------------- ----- ----- ----- ----- = ----- ----- >=20 > capacity operations = bandwidth > pool alloc free read write = read write > -------------------------------------- ----- ----- ----- ----- = ----- ----- > zclei21 10.1G 3.62T 0 191 = 0 12.7M > HGST_HUS724040ALA640_PN2334PBJ52XWT1 10.1G 3.62T 0 0 = 0 0 > logs - - - - = - - > lv_slog 225M 9.72G 0 191 = 0 12.7M > cache - - - - = - - > lv_cache 9.83G 204G 0 72 = 0 7.68M > -------------------------------------- ----- ----- ----- ----- = ----- ----- >=20 >=20 > On Fri, Mar 3, 2017 at 2:32 PM, Arman Khalatyan <arm2arm@gmail.com = <mailto:arm2arm@gmail.com>> wrote: > Glusterfs now in healing mode: > Receiver: > [root@clei21 ~]# arcstat.py 1 > time read miss miss% dmis dm% pmis pm% mmis mm% arcsz = c =20 > 13:24:49 0 0 0 0 0 0 0 0 0 4.6G = 31G =20 > 13:24:50 154 80 51 80 51 0 0 80 51 4.6G = 31G =20 > 13:24:51 179 62 34 62 34 0 0 62 42 4.6G = 31G =20 > 13:24:52 148 68 45 68 45 0 0 68 45 4.6G = 31G =20 > 13:24:53 140 64 45 64 45 0 0 64 45 4.6G = 31G =20 > 13:24:54 124 48 38 48 38 0 0 48 38 4.6G = 31G =20 > 13:24:55 157 80 50 80 50 0 0 80 50 4.7G = 31G =20 > 13:24:56 202 68 33 68 33 0 0 68 41 4.7G = 31G =20 > 13:24:57 127 54 42 54 42 0 0 54 42 4.7G = 31G =20 > 13:24:58 126 50 39 50 39 0 0 50 39 4.7G = 31G =20 > 13:24:59 116 40 34 40 34 0 0 40 34 4.7G = 31G =20 >=20 >=20 > Sender > [root@clei22 ~]# arcstat.py 1 > time read miss miss% dmis dm% pmis pm% mmis mm% arcsz = c =20 > 13:28:37 8 2 25 2 25 0 0 2 25 468M = 31G =20 > 13:28:38 1.2K 727 62 727 62 0 0 525 54 469M = 31G =20 > 13:28:39 815 508 62 508 62 0 0 376 55 469M = 31G =20 > 13:28:40 994 624 62 624 62 0 0 450 54 469M = 31G =20 > 13:28:41 783 456 58 456 58 0 0 338 50 470M = 31G =20 > 13:28:42 916 541 59 541 59 0 0 390 50 470M = 31G =20 > 13:28:43 768 437 56 437 57 0 0 313 48 471M = 31G =20 > 13:28:44 877 534 60 534 60 0 0 393 53 470M = 31G =20 > 13:28:45 957 630 65 630 65 0 0 450 57 470M = 31G =20 > 13:28:46 819 479 58 479 58 0 0 357 51 471M = 31G =20 >=20 >=20 > On Thu, Mar 2, 2017 at 7:18 PM, Juan Pablo <pablo.localhost@gmail.com = <mailto:pablo.localhost@gmail.com>> wrote: > hey, > what are you using for zfs? get an arc status and show please >=20 >=20 > 2017-03-02 9:57 GMT-03:00 Arman Khalatyan <arm2arm@gmail.com = <mailto:arm2arm@gmail.com>>: > no,=20 > ZFS itself is not on top of lvm. only ssd was spitted by lvm for = slog(10G) and cache (the rest) > but in any-case the ssd does not help much on glusterfs/ovirt load it = has almost 100% cache misses....:( (terrible performance compare with = nfs) >=20 >=20 >=20 >=20 >=20 > On Thu, Mar 2, 2017 at 1:47 PM, FERNANDO FREDIANI = <fernando.frediani@upx.com <mailto:fernando.frediani@upx.com>> wrote: > Am I understanding correctly, but you have Gluster on the top of ZFS = which is on the top of LVM ? If so, why the usage of LVM was necessary ? = I have ZFS with any need of LVM. >=20 > Fernando >=20 > On 02/03/2017 06:19, Arman Khalatyan wrote: >> Hi,=20 >> I use 3 nodes with zfs and glusterfs. >> Are there any suggestions to optimize it? >>=20 >> host zfs config 4TB-HDD+250GB-SSD: >> [root@clei22 ~]# zpool status=20 >> pool: zclei22 >> state: ONLINE >> scan: scrub repaired 0 in 0h0m with 0 errors on Tue Feb 28 14:16:07 = 2017 >> config: >>=20 >> NAME STATE READ WRITE = CKSUM >> zclei22 ONLINE 0 0 = 0 >> HGST_HUS724040ALA640_PN2334PBJ4SV6T1 ONLINE 0 0 = 0 >> logs >> lv_slog ONLINE 0 0 = 0 >> cache >> lv_cache ONLINE 0 0 = 0 >>=20 >> errors: No known data errors >>=20 >> Name: >> GluReplica >> Volume ID: >> ee686dfe-203a-4caa-a691-26353460cc48 >> Volume Type: >> Replicate (Arbiter) >> Replica Count: >> 2 + 1 >> Number of Bricks: >> 3 >> Transport Types: >> TCP, RDMA >> Maximum no of snapshots: >> 256 >> Capacity: >> 3.51 TiB total, 190.56 GiB used, 3.33 TiB free >>=20 >>=20 >> _______________________________________________ >> Users mailing list >> Users@ovirt.org <mailto:Users@ovirt.org> >> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users> >=20 >=20 > _______________________________________________ > Users mailing list > Users@ovirt.org <mailto:Users@ovirt.org> > http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users> >=20 >=20 >=20 > _______________________________________________ > Users mailing list > Users@ovirt.org <mailto:Users@ovirt.org> > http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users> >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users --Apple-Mail=_33A5E632-5864-4B33-9FED-1901AA57DEBD Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">Why are you using an arbitrator if all your HW configs are = identical? I=E2=80=99d use a true replica 3 in this case.<div = class=3D""><br class=3D""></div><div class=3D"">Also in my experience = with gluster and vm hosting, the ZIL/slog degrades write performance = unless it=E2=80=99s a truly dedicated disk. But I have 8 spinners = backing my ZFS volumes, so trying to share a sata disk wasn=E2=80=99t a = good zil. If yours is dedicated SAS, keep it, if it=E2=80=99s SATA, try = testing without it.</div><div class=3D""><br class=3D""></div><div = class=3D"">You don=E2=80=99t have compression enabled on your zfs = volume, and I=E2=80=99d recommend enabling relatime on it. Depending on = the amount of RAM in these boxes, you probably want to limit your zfs = arc size to 8G or so (1/4 total ram or less). Gluster just works volumes = hard during a rebuild, what=E2=80=99s the problem you=E2=80=99re seeing? = If it=E2=80=99s affecting your VMs, using shading and tuning client = & server threads can help avoid interruptions to your VMs while = repairs are running. If you really need to limit it, you can use cgroups = to keep it from hogging all the CPU, but it takes longer to heal, of = course. There are a couple older posts and blogs about it, if you go = back a while.</div><div class=3D""><br class=3D""></div><div = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Mar 3, 2017, at 9:02 AM, Arman Khalatyan <<a = href=3D"mailto:arm2arm@gmail.com" class=3D"">arm2arm@gmail.com</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D""><div class=3D""><div = class=3D""><div class=3D""><div class=3D""><div class=3D"">The problem = itself is not the streaming data performance., and also dd zero does not = help much in the production zfs running with compression.<br = class=3D""></div>the main problem comes when the gluster is starting to = do something with that, it is using xattrs, probably accessing extended = attributes inside the zfs is slower than XFS.<br class=3D""></div>Also = primitive find file or ls -l in the (dot)gluster folders takes ages: <br = class=3D""><br class=3D""></div>now I can see that arbiter host has = almost 100% cache miss during the rebuild, which is actually natural = while he is reading always the new datasets:<br class=3D"">[root@clei26 = ~]# arcstat.py 1<br class=3D""> time read = miss miss% dmis dm% pmis pm% = mmis mm% arcsz c <br = class=3D"">15:57:31 29 = 29 100 29 = 100 0 0 = 29 100 685M 31G <br = class=3D"">15:57:32 530 = 476 89 476 = 89 0 0 = 457 89 685M 31G <br = class=3D"">15:57:33 480 = 467 97 467 = 97 0 0 = 463 97 685M 31G <br = class=3D"">15:57:34 452 = 443 98 443 = 98 0 0 = 435 97 685M 31G <br = class=3D"">15:57:35 582 = 547 93 547 = 93 0 0 = 536 94 685M 31G <br = class=3D"">15:57:36 439 = 417 94 417 = 94 0 0 = 393 94 685M 31G <br = class=3D"">15:57:38 435 = 392 90 392 = 90 0 0 = 374 89 685M 31G <br = class=3D"">15:57:39 364 = 352 96 352 = 96 0 0 = 352 96 685M 31G <br = class=3D"">15:57:40 408 = 375 91 375 = 91 0 0 = 360 91 685M 31G <br = class=3D"">15:57:41 552 = 539 97 539 = 97 0 0 = 539 97 685M 31G <br = class=3D""><br class=3D""></div>It looks like we cannot have in the same = system performance and reliability :(<br class=3D""></div>Simply final = conclusion is with the single disk+ssd even zfs doesnot help to speedup = the glusterfs healing.<br class=3D""></div>I will stop here:)<br = class=3D""><br class=3D""><div class=3D""><div class=3D""><div = class=3D""><br class=3D""><div class=3D""><br class=3D""><br = class=3D""><div class=3D"gmail_extra"><div class=3D"gmail_quote">On Fri, = Mar 3, 2017 at 3:35 PM, Juan Pablo <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:pablo.localhost@gmail.com" target=3D"_blank" = class=3D"">pablo.localhost@gmail.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div = dir=3D"ltr" class=3D""><div class=3D""><div class=3D""><div = class=3D""><div class=3D"">cd to inside the pool path<br = class=3D""></div>then dd if=3D/dev/zero of=3D<a href=3D"http://test.tt/" = target=3D"_blank" class=3D"">test.tt</a> bs=3D1M <br = class=3D""></div>leave it runing 5/10 minutes.<br class=3D""></div>do = ctrl+c paste result here.<br class=3D""></div>etc.<br = class=3D""></div><div class=3D"gmail-HOEnZb"><div class=3D"gmail-h5"><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">2017-03-03= 11:30 GMT-03:00 Arman Khalatyan <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:arm2arm@gmail.com" target=3D"_blank" = class=3D"">arm2arm@gmail.com</a>></span>:<br class=3D""><blockquote = class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr" class=3D""><div = class=3D""><div class=3D"">No, I have one pool made of the one disk and = ssd as a cache and log device.<br class=3D""></div>I have 3 Glusterfs = bricks- separate 3 hosts:Volume type Replicate (Arbiter)=3D replica = 2+1!<br class=3D""></div>That how much you can push into compute = nodes(they have only 3 disk slots).<br class=3D""><br = class=3D""></div><div class=3D"gmail-m_6281861324822600694HOEnZb"><div = class=3D"gmail-m_6281861324822600694h5"><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On Fri, Mar 3, 2017 at 3:19 PM, = Juan Pablo <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:pablo.localhost@gmail.com" target=3D"_blank" = class=3D"">pablo.localhost@gmail.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div = dir=3D"ltr" class=3D""><div class=3D""><div class=3D"">ok, you have 3 = pools, zclei22, logs and cache, thats wrong. you should have 1 pool, = with zlog+cache if you are looking for performance.<br = class=3D""></div>also, dont mix drives. <br class=3D""></div>whats the = performance issue you are facing? <br class=3D""><div class=3D""><div = class=3D""><br class=3D""><br class=3D""></div><div = class=3D"">regards,</div></div></div><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465HOEnZb"><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465h5"><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">2017-03-03= 11:00 GMT-03:00 Arman Khalatyan <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:arm2arm@gmail.com" target=3D"_blank" = class=3D"">arm2arm@gmail.com</a>></span>:<br class=3D""><blockquote = class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr" class=3D"">This = is CentOS 7.3 ZoL version 0.6.5.9-1<br class=3D""><br = class=3D""><blockquote class=3D""><p class=3D"">[root@clei22 ~]# lsscsi = </p><p class=3D"">[2:0:0:0] disk = ATA INTEL SSDSC2CW24 400i /dev/sda = </p><p class=3D"">[3:0:0:0] disk = ATA HGST HUS724040AL AA70 /dev/sdb = </p><p class=3D"">[4:0:0:0] disk = ATA WDC WD2002FYPS-0 1G01 /dev/sdc = </p><p class=3D""><br class=3D""></p><p class=3D"">[root@clei22 ~]# pvs = ;vgs;lvs</p><p class=3D""> = PV = &n= bsp; <wbr = class=3D""> &nb= sp; = VG = Fmt Attr PSize PFree </p><p class=3D""> = /dev/mapper/INTEL_SSDSC2CW240A<wbr class=3D"">3_CVCV306302RP240CGN = vg_cache lvm2 a-- = 223.57g 0 </p><p class=3D""> = /dev/sdc2  = ; <wbr = class=3D""> &nb= sp; centos_clei22 = lvm2 a-- 1.82t 64.00m</p><p class=3D""> = VG #PV = #LV #SN Attr VSize VFree </p><p class=3D""> = centos_clei22 1 3 0 = wz--n- 1.82t 64.00m</p><p class=3D""> = vg_cache 1 = 2 0 wz--n- 223.57g 0 </p><p = class=3D""> LV = VG = Attr LSize Pool Origin = Data% Meta% Move Log Cpy%Sync Convert</p><p class=3D""> = home centos_clei22 -wi-ao---- = 1.74t &nb= sp;  = ; <wbr = class=3D""> &nb= sp;  = ; </p><p class=3D""> = root centos_clei22 -wi-ao---- = 50.00g &n= bsp; &nbs= p;<wbr = class=3D""> &nb= sp;  = ; </p><p class=3D""> = swap centos_clei22 -wi-ao---- = 31.44g &n= bsp; &nbs= p;<wbr = class=3D""> &nb= sp;  = ; </p><p class=3D""> lv_cache = vg_cache -wi-ao---- = 213.57g &= nbsp; <wb= r = class=3D""> &nb= sp;  = ; </p><p class=3D""> lv_slog = vg_cache -wi-ao---- = 10.00g </p><p class=3D""><br class=3D""></p><p = class=3D"">[root@clei22 ~]# zpool status -v</p><span class=3D""><p = class=3D""> pool: zclei22</p><p class=3D""> state: = ONLINE</p><p class=3D""> scan: scrub repaired 0 in 0h0m with 0 = errors on Tue Feb 28 14:16:07 2017</p><p class=3D"">config:</p><p = class=3D""><br class=3D""></p><p class=3D""> = NAME &nbs= p; = <wbr = class=3D""> = STATE READ WRITE CKSUM</p><p = class=3D""> = zclei22 &= nbsp; <wb= r class=3D""> = ONLINE 0 = 0 0</p><p class=3D""> = HGST_HUS724040ALA640_PN2334PBJ<wbr class=3D"">4SV6T1 = ONLINE 0 = 0 0</p><p class=3D""> = logs</p><p class=3D""> = lv_slog &= nbsp; <wb= r class=3D""> = ONLINE 0 = 0 0</p><p class=3D""> = cache</p><p class=3D""> = lv_cache = <wbr = class=3D""> = ONLINE 0 = 0 0</p><p class=3D""><br class=3D""></p><p = class=3D"">errors: No known data errors</p></span></blockquote><p = class=3D""><font size=3D"3" class=3D""><b class=3D""><br class=3D"">ZFS = config:</b></font></p><blockquote = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294gmail-tr_bq"><p class=3D"">[root@clei22 ~]# zfs = get all zclei22/01</p><p = class=3D"">NAME = PROPERTY = = VALUE &nb= sp; SOURCE</p><p class=3D"">zclei22/01 = = type &nbs= p; = filesystem &nbs= p; -</p><p class=3D"">zclei22/01 = creation = Tue Feb 28 14:06 2017 -</p><p = class=3D"">zclei22/01 = used &nbs= p; = 389G &nbs= p; -</p><p class=3D"">zclei22/01 = = available  = ; = 3.13T &nb= sp; -</p><p class=3D"">zclei22/01 = referenced &nbs= p; = 389G &nbs= p; -</p><p class=3D"">zclei22/01 = compressratio = 1.01x &nb= sp; -</p><p class=3D"">zclei22/01 = mounted &= nbsp; = yes  = ; -</p><p = class=3D"">zclei22/01 = quota &nb= sp; = none &nbs= p; default</p><p = class=3D"">zclei22/01 = reservation = none &nbs= p; default</p><p = class=3D"">zclei22/01 = recordsize &nbs= p; = 128K &nbs= p; local</p><p = class=3D"">zclei22/01 = mountpoint &nbs= p; = /zclei22/01 &nb= sp; default</p><p class=3D"">zclei22/01 = sharenfs = = off  = ; default</p><p = class=3D"">zclei22/01 = checksum = = on = default</p><p = class=3D"">zclei22/01 = compression = off  = ; local</p><p = class=3D"">zclei22/01 = atime &nb= sp; = on = default</p><p = class=3D"">zclei22/01 = devices &= nbsp; = on = default</p><p = class=3D"">zclei22/01 = exec &nbs= p; = on = default</p><p = class=3D"">zclei22/01 = setuid &n= bsp; = on = default</p><p = class=3D"">zclei22/01 = readonly = = off  = ; default</p><p = class=3D"">zclei22/01 = zoned &nb= sp; = off  = ; default</p><p = class=3D"">zclei22/01 = snapdir &= nbsp; = hidden &n= bsp; default</p><p class=3D"">zclei22/01 = aclinherit &nbs= p; = restricted &nbs= p; default</p><p class=3D"">zclei22/01 = canmount = = on = default</p><p = class=3D"">zclei22/01 = xattr &nb= sp; = sa = local</p><p = class=3D"">zclei22/01 = copies &n= bsp; = 1 &= nbsp; default</p><p = class=3D"">zclei22/01 = version &= nbsp; = 5 &= nbsp; -</p><p = class=3D"">zclei22/01 = utf8only = = off  = ; -</p><p = class=3D"">zclei22/01 = normalization = none &nbs= p; -</p><p class=3D"">zclei22/01 = casesensitivity = sensitive  = ; -</p><p class=3D"">zclei22/01 = vscan &nb= sp; = off  = ; default</p><p = class=3D"">zclei22/01 = nbmand &n= bsp; = off  = ; default</p><p = class=3D"">zclei22/01 = sharesmb = = off  = ; default</p><p = class=3D"">zclei22/01 = refquota = = none &nbs= p; default</p><p = class=3D"">zclei22/01 = refreservation = none &nbs= p; default</p><p = class=3D"">zclei22/01 = primarycache = metadata = local</p><p class=3D"">zclei22/01 = secondarycache = metadata = local</p><p class=3D"">zclei22/01 = usedbysnapshots = 0 &= nbsp; -</p><p = class=3D"">zclei22/01 = usedbydataset = 389G &nbs= p; -</p><p class=3D"">zclei22/01 = usedbychildren = 0 &= nbsp; -</p><p = class=3D"">zclei22/01 usedbyrefreservation = 0 &= nbsp; -</p><p = class=3D"">zclei22/01 = logbias &= nbsp; = latency &= nbsp; default</p><p class=3D"">zclei22/01 = dedup &nb= sp; = off  = ; default</p><p = class=3D"">zclei22/01 = mlslabel = = none &nbs= p; default</p><p = class=3D"">zclei22/01 = sync &nbs= p; = disabled = local</p><p class=3D"">zclei22/01 = refcompressratio = 1.01x &nb= sp; -</p><p class=3D"">zclei22/01 = written &= nbsp; = 389G &nbs= p; -</p><p class=3D"">zclei22/01 = logicalused = 396G &nbs= p; -</p><p class=3D"">zclei22/01 = logicalreferenced = 396G &nbs= p; -</p><p class=3D"">zclei22/01 = filesystem_limit = none &nbs= p; default</p><p = class=3D"">zclei22/01 = snapshot_limit = none &nbs= p; default</p><p = class=3D"">zclei22/01 = filesystem_count = none &nbs= p; default</p><p = class=3D"">zclei22/01 = snapshot_count = none &nbs= p; default</p><p = class=3D"">zclei22/01 = snapdev &= nbsp; = hidden &n= bsp; default</p><p class=3D"">zclei22/01 = acltype &= nbsp; = off  = ; default</p><p = class=3D"">zclei22/01 = context &= nbsp; = none &nbs= p; default</p><p = class=3D"">zclei22/01 = fscontext  = ; = none &nbs= p; default</p><p = class=3D"">zclei22/01 = defcontext &nbs= p; = none &nbs= p; default</p><p = class=3D"">zclei22/01 = rootcontext = none &nbs= p; default</p><p = class=3D"">zclei22/01 = relatime = = off  = ; default</p><p = class=3D"">zclei22/01 redundant_metadata = all  = ; default</p><p = class=3D"">zclei22/01 = overlay &= nbsp; = off  = ; default</p></blockquote><p = class=3D""><br class=3D""></p><br class=3D""><br class=3D""></div><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878HOEnZb"><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878h5"><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote">On Fri, Mar 3, 2017 at 2:52 PM, Juan Pablo <span = dir=3D"ltr" class=3D""><<a href=3D"mailto:pablo.localhost@gmail.com" = target=3D"_blank" class=3D"">pablo.localhost@gmail.com</a>></span> = wrote:<br class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px= 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr" class=3D""><div = class=3D""><div class=3D""><div class=3D"">Which operating system = version are you using for your zfs storage? <br class=3D""></div>do:<br = class=3D""></div>zfs get all your-pool-name<br class=3D""></div>use = arc_summary.py from freenas git repo if you wish.<br class=3D""><br = class=3D""></div><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294HOEnZb"><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294h5"><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">2017-03-03 10:33 GMT-03:00 Arman = Khalatyan <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:arm2arm@gmail.com" target=3D"_blank" = class=3D"">arm2arm@gmail.com</a>></span>:<br class=3D""><blockquote = class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr" class=3D"">Pool = load:<br class=3D"">[root@clei21 ~]# zpool iostat -v 1 <br = class=3D""> &nb= sp;  = ; <wbr = class=3D""> &nb= sp; capacity operations = bandwidth<br = class=3D"">pool  = ; &= nbsp; <wbr = class=3D""> = alloc free read write = read write<br class=3D"">------------------------------<wbr = class=3D"">-------- ----- ----- ----- = ----- ----- -----<br = class=3D"">zclei21 &n= bsp; &nbs= p; <wbr = class=3D""> = 10.1G 3.62T 0 = 112 823 8.82M<br class=3D""> = HGST_HUS724040ALA640_PN2334PBJ<wbr class=3D"">52XWT1 10.1G = 3.62T 0 = 46 626 4.40M<br = class=3D"">logs  = ; &= nbsp; <wbr = class=3D""> &nb= sp; - = - - = - - -<br = class=3D""> = lv_slog &= nbsp; <wb= r class=3D""> 225M = 9.72G 0 = 66 198 4.45M<br = class=3D"">cache &nbs= p; = <wbr = class=3D""> &nb= sp; - = - - = - - -<br = class=3D""> = lv_cache = <wbr = class=3D""> 9.81G = 204G 0 = 46 56 4.13M<br = class=3D"">------------------------------<wbr class=3D"">-------- = ----- ----- ----- ----- ----- -----<br = class=3D""><br = class=3D""> &nb= sp;  = ; <wbr = class=3D""> &nb= sp; capacity operations = bandwidth<br = class=3D"">pool  = ; &= nbsp; <wbr = class=3D""> = alloc free read write = read write<br class=3D"">------------------------------<wbr = class=3D"">-------- ----- ----- ----- = ----- ----- -----<br = class=3D"">zclei21 &n= bsp; &nbs= p; <wbr = class=3D""> = 10.1G 3.62T 0 = 191 0 12.8M<br class=3D""> = HGST_HUS724040ALA640_PN2334PBJ<wbr class=3D"">52XWT1 10.1G = 3.62T 0 = 0 0 0<br = class=3D"">logs  = ; &= nbsp; <wbr = class=3D""> &nb= sp; - = - - = - - -<br = class=3D""> = lv_slog &= nbsp; <wb= r class=3D""> 225M = 9.72G 0 = 191 0 12.8M<br = class=3D"">cache &nbs= p; = <wbr = class=3D""> &nb= sp; - = - - = - - -<br = class=3D""> = lv_cache = <wbr = class=3D""> 9.83G = 204G 0 = 218 0 20.0M<br = class=3D"">------------------------------<wbr class=3D"">-------- = ----- ----- ----- ----- ----- -----<br = class=3D""><br = class=3D""> &nb= sp;  = ; <wbr = class=3D""> &nb= sp; capacity operations = bandwidth<br = class=3D"">pool  = ; &= nbsp; <wbr = class=3D""> = alloc free read write = read write<br class=3D"">------------------------------<wbr = class=3D"">-------- ----- ----- ----- = ----- ----- -----<br = class=3D"">zclei21 &n= bsp; &nbs= p; <wbr = class=3D""> = 10.1G 3.62T 0 = 191 0 12.7M<br class=3D""> = HGST_HUS724040ALA640_PN2334PBJ<wbr class=3D"">52XWT1 10.1G = 3.62T 0 = 0 0 0<br = class=3D"">logs  = ; &= nbsp; <wbr = class=3D""> &nb= sp; - = - - = - - -<br = class=3D""> = lv_slog &= nbsp; <wb= r class=3D""> 225M = 9.72G 0 = 191 0 12.7M<br = class=3D"">cache &nbs= p; = <wbr = class=3D""> &nb= sp; - = - - = - - -<br = class=3D""> = lv_cache = <wbr = class=3D""> 9.83G = 204G 0 = 72 0 7.68M<br = class=3D"">------------------------------<wbr class=3D"">-------- = ----- ----- ----- ----- ----- -----<br = class=3D""><br class=3D""></div><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647HOEnZb"><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647h5"><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On Fri, = Mar 3, 2017 at 2:32 PM, Arman Khalatyan <span dir=3D"ltr" = class=3D""><<a href=3D"mailto:arm2arm@gmail.com" target=3D"_blank" = class=3D"">arm2arm@gmail.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div = dir=3D"ltr" class=3D""><div class=3D"">Glusterfs now in healing mode:<br = class=3D""></div>Receiver:<br class=3D"">[root@clei21 ~]# arcstat.py = 1<br class=3D""> time read miss = miss% dmis dm% pmis pm% mmis = mm% arcsz c <br = class=3D"">13:24:49 0 = 0 0 = 0 0 0 = 0 0 0 = 4.6G 31G <br class=3D"">13:24:50 = 154 80 51 = 80 51 0 = 0 80 51 4.6G = 31G <br class=3D"">13:24:51 179 = 62 34 62 = 34 0 0 = 62 42 4.6G 31G <br = class=3D"">13:24:52 148 = 68 45 68 = 45 0 0 = 68 45 4.6G 31G <br = class=3D"">13:24:53 140 = 64 45 64 = 45 0 0 = 64 45 4.6G 31G <br = class=3D"">13:24:54 124 = 48 38 48 = 38 0 0 = 48 38 4.6G 31G <br = class=3D"">13:24:55 157 = 80 50 80 = 50 0 0 = 80 50 4.7G 31G <br = class=3D"">13:24:56 202 = 68 33 68 = 33 0 0 = 68 41 4.7G 31G <br = class=3D"">13:24:57 127 = 54 42 54 = 42 0 0 = 54 42 4.7G 31G <br = class=3D"">13:24:58 126 = 50 39 50 = 39 0 0 = 50 39 4.7G 31G <br = class=3D"">13:24:59 116 = 40 34 40 = 34 0 0 = 40 34 4.7G 31G <br = class=3D""><br class=3D""><div class=3D""><br class=3D"">Sender<br = class=3D"">[root@clei22 ~]# arcstat.py 1<br class=3D""> = time read miss miss% dmis dm% = pmis pm% mmis mm% arcsz = c <br class=3D"">13:28:37 = 8 2 = 25 2 25 = 0 0 2 = 25 468M 31G <br class=3D"">13:28:38 = 1.2K 727 62 = 727 62 0 = 0 525 54 469M 31G = <br class=3D"">13:28:39 815 = 508 62 508 = 62 0 0 = 376 55 469M 31G <br = class=3D"">13:28:40 994 = 624 62 624 = 62 0 0 = 450 54 469M 31G <br = class=3D"">13:28:41 783 = 456 58 456 = 58 0 0 = 338 50 470M 31G <br = class=3D"">13:28:42 916 = 541 59 541 = 59 0 0 = 390 50 470M 31G <br = class=3D"">13:28:43 768 = 437 56 437 = 57 0 0 = 313 48 471M 31G <br = class=3D"">13:28:44 877 = 534 60 534 = 60 0 0 = 393 53 470M 31G <br = class=3D"">13:28:45 957 = 630 65 630 = 65 0 0 = 450 57 470M 31G <br = class=3D"">13:28:46 819 = 479 58 479 = 58 0 0 = 357 51 471M 31G <br = class=3D""><br class=3D""></div></div><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713HOEnZ= b"><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713h5"><= div class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On = Thu, Mar 2, 2017 at 7:18 PM, Juan Pablo <span dir=3D"ltr" = class=3D""><<a href=3D"mailto:pablo.localhost@gmail.com" = target=3D"_blank" class=3D"">pablo.localhost@gmail.com</a>></span> = wrote:<br class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px= 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr" class=3D"">hey,<br = class=3D"">what are you using for zfs? get an arc status and show = please<br class=3D""><br class=3D""></div><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713m_690= 6981569320781HOEnZb"><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713m_690= 6981569320781h5"><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote">2017-03-02 9:57 GMT-03:00 Arman Khalatyan <span = dir=3D"ltr" class=3D""><<a href=3D"mailto:arm2arm@gmail.com" = target=3D"_blank" class=3D"">arm2arm@gmail.com</a>></span>:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div = dir=3D"ltr" class=3D""><div class=3D""><div class=3D"">no, <br = class=3D""></div>ZFS itself is not on top of lvm. only ssd was spitted = by lvm for slog(10G) and cache (the rest)<br class=3D""></div><div = class=3D"">but in any-case the ssd does not help much on = glusterfs/ovirt load it has almost 100% cache misses....:( = (terrible performance compare with nfs)<br class=3D""><br = class=3D""></div><div class=3D""><br class=3D""></div><br class=3D""><br = class=3D""></div><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713m_690= 6981569320781m_349397128160904570HOEnZb"><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713m_690= 6981569320781m_349397128160904570h5"><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On Thu, Mar 2, 2017 at 1:47 PM, = FERNANDO FREDIANI <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:fernando.frediani@upx.com" target=3D"_blank" = class=3D"">fernando.frediani@upx.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> =20 =20 =20 <div bgcolor=3D"#FFFFFF" class=3D""><p class=3D"">Am I understanding = correctly, but you have Gluster on the top of ZFS which is on the top of LVM ? If so, why the usage of LVM was necessary ? I have ZFS with any need of LVM.</p><span = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713m_690= 6981569320781m_349397128160904570m_-6203522006917276901HOEnZb"><font = color=3D"#888888" class=3D""><p class=3D"">Fernando<br class=3D""> </p></font></span><div class=3D""><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713m_690= 6981569320781m_349397128160904570m_-6203522006917276901h5"> <br class=3D""> <div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713m_690= 6981569320781m_349397128160904570m_-6203522006917276901m_35422971570659266= 81moz-cite-prefix">On 02/03/2017 06:19, Arman Khalatyan wrote:<br class=3D""> </div> </div></div><blockquote type=3D"cite" class=3D""><div class=3D""><div = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713m_690= 6981569320781m_349397128160904570m_-6203522006917276901h5"> <div dir=3D"ltr" class=3D""> <div class=3D""> <div class=3D"">Hi, <br class=3D""> </div> I use 3 nodes with zfs and glusterfs.<br class=3D""> </div> Are there any suggestions to optimize it?<br class=3D""> <div class=3D""><br class=3D""> host zfs config 4TB-HDD+250GB-SSD:<br class=3D""> [root@clei22 ~]# zpool status <br class=3D""> pool: zclei22<br class=3D""> state: ONLINE<br class=3D""> scan: scrub repaired 0 in 0h0m with 0 errors on Tue Feb = 28 14:16:07 2017<br class=3D""> config:<br class=3D""> <br class=3D""> = NAME &nbs= p; = <wbr = class=3D""> = STATE READ WRITE CKSUM<br class=3D""> = zclei22 &= nbsp; <wb= r class=3D""> = ONLINE 0 0 0<br class=3D""> HGST_HUS724040ALA640_PN2334PBJ<wbr = class=3D"">4SV6T1 ONLINE = 0 0 0<br class=3D""> logs<br class=3D""> = lv_slog &= nbsp; <wb= r class=3D""> = ONLINE 0 0 0<br class=3D""> cache<br class=3D""> = lv_cache = <wbr = class=3D""> = ONLINE 0 0 0<br class=3D""> <br class=3D""> errors: No known data errors<br class=3D""> <br class=3D""> Name:<br class=3D""> GluReplica<br class=3D""> Volume ID:<br class=3D""> ee686dfe-203a-4caa-a691-263534<wbr class=3D"">60cc48<br = class=3D""> Volume Type:<br class=3D""> Replicate (Arbiter)<br class=3D""> Replica Count:<br class=3D""> 2 + 1<br class=3D""> Number of Bricks:<br class=3D""> 3<br class=3D""> Transport Types:<br class=3D""> TCP, RDMA<br class=3D""> Maximum no of snapshots:<br class=3D""> 256<br class=3D""> Capacity:<br class=3D""> 3.51 TiB total, 190.56 GiB used, 3.33 TiB free<br class=3D""> </div> </div> <br class=3D""> <fieldset = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713m_690= 6981569320781m_349397128160904570m_-6203522006917276901m_35422971570659266= 81mimeAttachmentHeader"></fieldset> <br class=3D""> </div></div><span class=3D""><pre = class=3D"">______________________________<wbr class=3D"">_________________= Users mailing list <a = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713m_690= 6981569320781m_349397128160904570m_-6203522006917276901m_35422971570659266= 81moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org" = target=3D"_blank">Users@ovirt.org</a> <a = class=3D"gmail-m_6281861324822600694m_9102919535904979465m_-77235589057693= 4878m_8401411119881083294m_-2168826882466388647m_-4808390406438333713m_690= 6981569320781m_349397128160904570m_-6203522006917276901m_35422971570659266= 81moz-txt-link-freetext" = href=3D"http://lists.ovirt.org/mailman/listinfo/users" = target=3D"_blank">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/users</a> </pre> </span></blockquote> <br class=3D""> </div> <br class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" target=3D"_blank" = class=3D"">Users@ovirt.org</a><br class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/users</a><br class=3D""> <br class=3D""></blockquote></div><br class=3D""></div> </div></div><br class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" target=3D"_blank" = class=3D"">Users@ovirt.org</a><br class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/users</a><br class=3D""> <br class=3D""></blockquote></div><br class=3D""></div> </div></div></blockquote></div><br class=3D""></div> </div></div></blockquote></div><br class=3D""></div> </div></div></blockquote></div><br class=3D""></div> </div></div></blockquote></div><br class=3D""></div> </div></div></blockquote></div><br class=3D""></div> </div></div></blockquote></div><br class=3D""></div> </div></div></blockquote></div><br class=3D""></div> </div></div></blockquote></div><br = class=3D""></div></div></div></div></div></div> _______________________________________________<br class=3D"">Users = mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org" = class=3D"">Users@ovirt.org</a><br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br = class=3D""></div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_33A5E632-5864-4B33-9FED-1901AA57DEBD--