1:27 p.m.
Hello,
I have 3 node HCI cluster (Rocky Linux 8,
4.5.7-0.master.20240415165511.git7238a3766d.el8). I had 2 SSD in each
node, each as separate brick. Recently I added third SSD and expanded
volume to 3 x 3 topology. Despite this, free space on volume was not
changed.
gluster volume info vms
Volume Name: vms
Type: Distributed-Replicate
Volume ID: 52032ec6-99d4-4210-8fb8-ffbd7a1e0bf7
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x 3 = 9
Transport-type: tcp
Bricks:
Brick1: 10.0.4.11:/gluster_bricks/vms/vms
Brick2: 10.0.4.13:/gluster_bricks/vms/vms
Brick3: 10.0.4.12:/gluster_bricks/vms/vms
Brick4: 10.0.4.11:/gluster_bricks/vms2/vms2
Brick5: 10.0.4.13:/gluster_bricks/vms2/vms2
Brick6: 10.0.4.12:/gluster_bricks/vms2/vms2
Brick7: 10.0.4.11:/gluster_bricks/vms3/vms3
Brick8: 10.0.4.12:/gluster_bricks/vms3/vms3
Brick9: 10.0.4.13:/gluster_bricks/vms3/vms3
Options Reconfigured:
cluster.shd-max-threads: 1
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
storage.owner-uid: 36
storage.owner-gid: 36
features.shard: on
performance.low-prio-threads: 32
performance.strict-o-direct: on
network.remote-dio: off
network.ping-timeout: 30
user.cifs: off
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
cluster.eager-lock: enable
performance.stat-prefetch: off
cluster.granular-entry-heal: enable
storage.health-check-interval: 0
df -h on all nodes looks the same
...
10.0.4.11:/engine 100G 23G 78G 23%
/rhev/data-center/mnt/glusterSD/10.0.4.11:_engine
10.0.4.11:/vms 1.7T 773G 952G 45%
/rhev/data-center/mnt/glusterSD/10.0.4.11:_vms
...
/dev/mapper/gluster_vg_sdb-gluster_lv_engine 100G 22G 79G 22%
/gluster_bricks/engine
/dev/mapper/gluster_vg_sdb-gluster_lv_vms 794G 476G 319G 60%
/gluster_bricks/vms
/dev/mapper/gluster_vg_sdd-gluster_lv_vms2 930G 553G 378G 60%
/gluster_bricks/vms2
/dev/mapper/gluster_vg_vms3-gluster_lv_vms3 932G 6.6G 925G 1%
/gluster_bricks/vms3
...
size of mounted vms volume is reported as 1.7T which is old value (sum
of two bricks - 794G + 930G). Correct size should be sum of all bricks -
around 2.6T.
what step I am missing?
Cheers,
Jiri
12:18 p.m.
New subject: hci, glusterfs, glusterfs mount shows old size after expanding volume (adding brick)
Hi,
I found solution.
Problem is in wrong detection of shared brick fs. In my case some of
bricks on some of hosts were detected as residing on brick on a shared
fs. If you have all bricks on its own fs you should see
shared-brick-count as 0 or 1 (on brick on host you are checking)
grep shared-brick-count /var/lib/glusterd/vols/*/* | grep -v rpmsave |
grep -Ev "shared-brick-count (0|1)"
if you see count >1 check brick-fsid on bricks
grep brick-fsid /var/lib/glusterd/vols/<volume>/bricks/*
if you see duplicate fsid (on brick on host you are checking) you can
edit that file with brick configuration and delete line with brick-fsid.
Then restart gluster daemon
systemctl restart glusterd
and do some change on volume to repopulate shared-brick-count
gluster volume set <volume> min-free-disk 10%
gluster volume reset <volume> min-free-disk
do it on all hosts with shared-brick-count >1
all is mentioned in
https://github.com/gluster/glusterfs/issues/3642#issuecomment-1188788173
Cheers,
Jiri
On 11/8/24 11:27, Jiří Sléžka via Users wrote:
Hello,
I have 3 node HCI cluster (Rocky Linux 8,
4.5.7-0.master.20240415165511.git7238a3766d.el8). I had 2 SSD in each
node, each as separate brick. Recently I added third SSD and expanded
volume to 3 x 3 topology. Despite this, free space on volume was not
changed.
gluster volume info vms
Volume Name: vms
Type: Distributed-Replicate
Volume ID: 52032ec6-99d4-4210-8fb8-ffbd7a1e0bf7
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x 3 = 9
Transport-type: tcp
Bricks:
Brick1: 10.0.4.11:/gluster_bricks/vms/vms
Brick2: 10.0.4.13:/gluster_bricks/vms/vms
Brick3: 10.0.4.12:/gluster_bricks/vms/vms
Brick4: 10.0.4.11:/gluster_bricks/vms2/vms2
Brick5: 10.0.4.13:/gluster_bricks/vms2/vms2
Brick6: 10.0.4.12:/gluster_bricks/vms2/vms2
Brick7: 10.0.4.11:/gluster_bricks/vms3/vms3
Brick8: 10.0.4.12:/gluster_bricks/vms3/vms3
Brick9: 10.0.4.13:/gluster_bricks/vms3/vms3
Options Reconfigured:
cluster.shd-max-threads: 1
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
storage.owner-uid: 36
storage.owner-gid: 36
features.shard: on
performance.low-prio-threads: 32
performance.strict-o-direct: on
network.remote-dio: off
network.ping-timeout: 30
user.cifs: off
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
cluster.eager-lock: enable
performance.stat-prefetch: off
cluster.granular-entry-heal: enable
storage.health-check-interval: 0
df -h on all nodes looks the same
...
10.0.4.11:/engine 100G 23G 78G 23% /
rhev/data-center/mnt/glusterSD/10.0.4.11:_engine
10.0.4.11:/vms 1.7T 773G 952G 45% /
rhev/data-center/mnt/glusterSD/10.0.4.11:_vms
...
/dev/mapper/gluster_vg_sdb-gluster_lv_engine 100G 22G 79G 22% /
gluster_bricks/engine
/dev/mapper/gluster_vg_sdb-gluster_lv_vms 794G 476G 319G 60% /
gluster_bricks/vms
/dev/mapper/gluster_vg_sdd-gluster_lv_vms2 930G 553G 378G 60% /
gluster_bricks/vms2
/dev/mapper/gluster_vg_vms3-gluster_lv_vms3 932G 6.6G 925G 1% /
gluster_bricks/vms3
...
size of mounted vms volume is reported as 1.7T which is old value (sum
of two bricks - 794G + 930G). Correct size should be sum of all bricks -
around 2.6T.
what step I am missing?
Cheers,
Jiri
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JOVHJZUUEI3...
9
days inactive
13
days old
1 comments
1 participants
participants (1)
-
Jiří Sléžka