[ovirt-users] Replicated Glusterfs on top of ZFS

Juan Pablo pablo.localhost at gmail.com
Fri Mar 3 13:52:15 UTC 2017


Which operating system version are you using for your zfs storage?
do:
zfs get all your-pool-name
use arc_summary.py from freenas git repo if you wish.


2017-03-03 10:33 GMT-03:00 Arman Khalatyan <arm2arm at gmail.com>:

> Pool load:
> [root at clei21 ~]# zpool iostat -v 1
>                                            capacity     operations
> bandwidth
> pool                                    alloc   free   read  write
> read  write
> --------------------------------------  -----  -----  -----  -----
> -----  -----
> zclei21                                 10.1G  3.62T      0    112
> 823  8.82M
>   HGST_HUS724040ALA640_PN2334PBJ52XWT1  10.1G  3.62T      0     46
> 626  4.40M
> logs                                        -      -      -      -
> -      -
>   lv_slog                                225M  9.72G      0     66
> 198  4.45M
> cache                                       -      -      -      -
> -      -
>   lv_cache                              9.81G   204G      0     46
> 56  4.13M
> --------------------------------------  -----  -----  -----  -----
> -----  -----
>
>                                            capacity     operations
> bandwidth
> pool                                    alloc   free   read  write
> read  write
> --------------------------------------  -----  -----  -----  -----
> -----  -----
> zclei21                                 10.1G  3.62T      0    191
> 0  12.8M
>   HGST_HUS724040ALA640_PN2334PBJ52XWT1  10.1G  3.62T      0      0
> 0      0
> logs                                        -      -      -      -
> -      -
>   lv_slog                                225M  9.72G      0    191
> 0  12.8M
> cache                                       -      -      -      -
> -      -
>   lv_cache                              9.83G   204G      0    218
> 0  20.0M
> --------------------------------------  -----  -----  -----  -----
> -----  -----
>
>                                            capacity     operations
> bandwidth
> pool                                    alloc   free   read  write
> read  write
> --------------------------------------  -----  -----  -----  -----
> -----  -----
> zclei21                                 10.1G  3.62T      0    191
> 0  12.7M
>   HGST_HUS724040ALA640_PN2334PBJ52XWT1  10.1G  3.62T      0      0
> 0      0
> logs                                        -      -      -      -
> -      -
>   lv_slog                                225M  9.72G      0    191
> 0  12.7M
> cache                                       -      -      -      -
> -      -
>   lv_cache                              9.83G   204G      0     72
> 0  7.68M
> --------------------------------------  -----  -----  -----  -----
> -----  -----
>
>
> On Fri, Mar 3, 2017 at 2:32 PM, Arman Khalatyan <arm2arm at gmail.com> wrote:
>
>> Glusterfs now in healing mode:
>> Receiver:
>> [root at clei21 ~]# arcstat.py 1
>>     time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz
>> c
>> 13:24:49     0     0      0     0    0     0    0     0    0   4.6G
>> 31G
>> 13:24:50   154    80     51    80   51     0    0    80   51   4.6G
>> 31G
>> 13:24:51   179    62     34    62   34     0    0    62   42   4.6G
>> 31G
>> 13:24:52   148    68     45    68   45     0    0    68   45   4.6G
>> 31G
>> 13:24:53   140    64     45    64   45     0    0    64   45   4.6G
>> 31G
>> 13:24:54   124    48     38    48   38     0    0    48   38   4.6G
>> 31G
>> 13:24:55   157    80     50    80   50     0    0    80   50   4.7G
>> 31G
>> 13:24:56   202    68     33    68   33     0    0    68   41   4.7G
>> 31G
>> 13:24:57   127    54     42    54   42     0    0    54   42   4.7G
>> 31G
>> 13:24:58   126    50     39    50   39     0    0    50   39   4.7G
>> 31G
>> 13:24:59   116    40     34    40   34     0    0    40   34   4.7G
>> 31G
>>
>>
>> Sender
>> [root at clei22 ~]# arcstat.py 1
>>     time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz
>> c
>> 13:28:37     8     2     25     2   25     0    0     2   25   468M
>> 31G
>> 13:28:38  1.2K   727     62   727   62     0    0   525   54   469M
>> 31G
>> 13:28:39   815   508     62   508   62     0    0   376   55   469M
>> 31G
>> 13:28:40   994   624     62   624   62     0    0   450   54   469M
>> 31G
>> 13:28:41   783   456     58   456   58     0    0   338   50   470M
>> 31G
>> 13:28:42   916   541     59   541   59     0    0   390   50   470M
>> 31G
>> 13:28:43   768   437     56   437   57     0    0   313   48   471M
>> 31G
>> 13:28:44   877   534     60   534   60     0    0   393   53   470M
>> 31G
>> 13:28:45   957   630     65   630   65     0    0   450   57   470M
>> 31G
>> 13:28:46   819   479     58   479   58     0    0   357   51   471M
>> 31G
>>
>>
>> On Thu, Mar 2, 2017 at 7:18 PM, Juan Pablo <pablo.localhost at gmail.com>
>> wrote:
>>
>>> hey,
>>> what are you using for zfs? get an arc status and show please
>>>
>>>
>>> 2017-03-02 9:57 GMT-03:00 Arman Khalatyan <arm2arm at gmail.com>:
>>>
>>>> no,
>>>> ZFS itself is not on top of lvm. only ssd was spitted by lvm for
>>>> slog(10G) and cache (the rest)
>>>> but in any-case the ssd does not help much on glusterfs/ovirt  load it
>>>> has almost 100% cache misses....:( (terrible performance compare with nfs)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Mar 2, 2017 at 1:47 PM, FERNANDO FREDIANI <
>>>> fernando.frediani at upx.com> wrote:
>>>>
>>>>> Am I understanding correctly, but you have Gluster on the top of ZFS
>>>>> which is on the top of LVM ? If so, why the usage of LVM was necessary ? I
>>>>> have ZFS with any need of LVM.
>>>>>
>>>>> Fernando
>>>>>
>>>>> On 02/03/2017 06:19, Arman Khalatyan wrote:
>>>>>
>>>>> Hi,
>>>>> I use 3 nodes with zfs and glusterfs.
>>>>> Are there any suggestions to optimize it?
>>>>>
>>>>> host zfs config 4TB-HDD+250GB-SSD:
>>>>> [root at clei22 ~]# zpool status
>>>>>   pool: zclei22
>>>>>  state: ONLINE
>>>>>   scan: scrub repaired 0 in 0h0m with 0 errors on Tue Feb 28 14:16:07
>>>>> 2017
>>>>> config:
>>>>>
>>>>>     NAME                                    STATE     READ WRITE CKSUM
>>>>>     zclei22                                 ONLINE       0     0     0
>>>>>       HGST_HUS724040ALA640_PN2334PBJ4SV6T1  ONLINE       0     0     0
>>>>>     logs
>>>>>       lv_slog                               ONLINE       0     0     0
>>>>>     cache
>>>>>       lv_cache                              ONLINE       0     0     0
>>>>>
>>>>> errors: No known data errors
>>>>>
>>>>> Name:
>>>>> GluReplica
>>>>> Volume ID:
>>>>> ee686dfe-203a-4caa-a691-26353460cc48
>>>>> Volume Type:
>>>>> Replicate (Arbiter)
>>>>> Replica Count:
>>>>> 2 + 1
>>>>> Number of Bricks:
>>>>> 3
>>>>> Transport Types:
>>>>> TCP, RDMA
>>>>> Maximum no of snapshots:
>>>>> 256
>>>>> Capacity:
>>>>> 3.51 TiB total, 190.56 GiB used, 3.33 TiB free
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing listUsers at ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170303/3d57be61/attachment.html>


More information about the Users mailing list