hci, glusterfs, glusterfs mount shows old size after expanding volume (adding brick)
Hello, I have 3 node HCI cluster (Rocky Linux 8, 4.5.7-0.master.20240415165511.git7238a3766d.el8). I had 2 SSD in each node, each as separate brick. Recently I added third SSD and expanded volume to 3 x 3 topology. Despite this, free space on volume was not changed. gluster volume info vms Volume Name: vms Type: Distributed-Replicate Volume ID: 52032ec6-99d4-4210-8fb8-ffbd7a1e0bf7 Status: Started Snapshot Count: 0 Number of Bricks: 3 x 3 = 9 Transport-type: tcp Bricks: Brick1: 10.0.4.11:/gluster_bricks/vms/vms Brick2: 10.0.4.13:/gluster_bricks/vms/vms Brick3: 10.0.4.12:/gluster_bricks/vms/vms Brick4: 10.0.4.11:/gluster_bricks/vms2/vms2 Brick5: 10.0.4.13:/gluster_bricks/vms2/vms2 Brick6: 10.0.4.12:/gluster_bricks/vms2/vms2 Brick7: 10.0.4.11:/gluster_bricks/vms3/vms3 Brick8: 10.0.4.12:/gluster_bricks/vms3/vms3 Brick9: 10.0.4.13:/gluster_bricks/vms3/vms3 Options Reconfigured: cluster.shd-max-threads: 1 performance.client-io-threads: off nfs.disable: on storage.fips-mode-rchecksum: on transport.address-family: inet storage.owner-uid: 36 storage.owner-gid: 36 features.shard: on performance.low-prio-threads: 32 performance.strict-o-direct: on network.remote-dio: off network.ping-timeout: 30 user.cifs: off performance.quick-read: off performance.read-ahead: off performance.io-cache: off cluster.eager-lock: enable performance.stat-prefetch: off cluster.granular-entry-heal: enable storage.health-check-interval: 0 df -h on all nodes looks the same ... 10.0.4.11:/engine 100G 23G 78G 23% /rhev/data-center/mnt/glusterSD/10.0.4.11:_engine 10.0.4.11:/vms 1.7T 773G 952G 45% /rhev/data-center/mnt/glusterSD/10.0.4.11:_vms ... /dev/mapper/gluster_vg_sdb-gluster_lv_engine 100G 22G 79G 22% /gluster_bricks/engine /dev/mapper/gluster_vg_sdb-gluster_lv_vms 794G 476G 319G 60% /gluster_bricks/vms /dev/mapper/gluster_vg_sdd-gluster_lv_vms2 930G 553G 378G 60% /gluster_bricks/vms2 /dev/mapper/gluster_vg_vms3-gluster_lv_vms3 932G 6.6G 925G 1% /gluster_bricks/vms3 ... size of mounted vms volume is reported as 1.7T which is old value (sum of two bricks - 794G + 930G). Correct size should be sum of all bricks - around 2.6T. what step I am missing? Cheers, Jiri
Hi, I found solution. Problem is in wrong detection of shared brick fs. In my case some of bricks on some of hosts were detected as residing on brick on a shared fs. If you have all bricks on its own fs you should see shared-brick-count as 0 or 1 (on brick on host you are checking) grep shared-brick-count /var/lib/glusterd/vols/*/* | grep -v rpmsave | grep -Ev "shared-brick-count (0|1)" if you see count >1 check brick-fsid on bricks grep brick-fsid /var/lib/glusterd/vols/<volume>/bricks/* if you see duplicate fsid (on brick on host you are checking) you can edit that file with brick configuration and delete line with brick-fsid. Then restart gluster daemon systemctl restart glusterd and do some change on volume to repopulate shared-brick-count gluster volume set <volume> min-free-disk 10% gluster volume reset <volume> min-free-disk do it on all hosts with shared-brick-count >1 all is mentioned in https://github.com/gluster/glusterfs/issues/3642#issuecomment-1188788173 Cheers, Jiri On 11/8/24 11:27, Jiří Sléžka via Users wrote:
Hello,
I have 3 node HCI cluster (Rocky Linux 8, 4.5.7-0.master.20240415165511.git7238a3766d.el8). I had 2 SSD in each node, each as separate brick. Recently I added third SSD and expanded volume to 3 x 3 topology. Despite this, free space on volume was not changed.
gluster volume info vms
Volume Name: vms Type: Distributed-Replicate Volume ID: 52032ec6-99d4-4210-8fb8-ffbd7a1e0bf7 Status: Started Snapshot Count: 0 Number of Bricks: 3 x 3 = 9 Transport-type: tcp Bricks: Brick1: 10.0.4.11:/gluster_bricks/vms/vms Brick2: 10.0.4.13:/gluster_bricks/vms/vms Brick3: 10.0.4.12:/gluster_bricks/vms/vms Brick4: 10.0.4.11:/gluster_bricks/vms2/vms2 Brick5: 10.0.4.13:/gluster_bricks/vms2/vms2 Brick6: 10.0.4.12:/gluster_bricks/vms2/vms2 Brick7: 10.0.4.11:/gluster_bricks/vms3/vms3 Brick8: 10.0.4.12:/gluster_bricks/vms3/vms3 Brick9: 10.0.4.13:/gluster_bricks/vms3/vms3 Options Reconfigured: cluster.shd-max-threads: 1 performance.client-io-threads: off nfs.disable: on storage.fips-mode-rchecksum: on transport.address-family: inet storage.owner-uid: 36 storage.owner-gid: 36 features.shard: on performance.low-prio-threads: 32 performance.strict-o-direct: on network.remote-dio: off network.ping-timeout: 30 user.cifs: off performance.quick-read: off performance.read-ahead: off performance.io-cache: off cluster.eager-lock: enable performance.stat-prefetch: off cluster.granular-entry-heal: enable storage.health-check-interval: 0
df -h on all nodes looks the same
... 10.0.4.11:/engine 100G 23G 78G 23% / rhev/data-center/mnt/glusterSD/10.0.4.11:_engine 10.0.4.11:/vms 1.7T 773G 952G 45% / rhev/data-center/mnt/glusterSD/10.0.4.11:_vms ... /dev/mapper/gluster_vg_sdb-gluster_lv_engine 100G 22G 79G 22% / gluster_bricks/engine /dev/mapper/gluster_vg_sdb-gluster_lv_vms 794G 476G 319G 60% / gluster_bricks/vms /dev/mapper/gluster_vg_sdd-gluster_lv_vms2 930G 553G 378G 60% / gluster_bricks/vms2 /dev/mapper/gluster_vg_vms3-gluster_lv_vms3 932G 6.6G 925G 1% / gluster_bricks/vms3 ...
size of mounted vms volume is reported as 1.7T which is old value (sum of two bricks - 794G + 930G). Correct size should be sum of all bricks - around 2.6T.
what step I am missing?
Cheers,
Jiri
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/JOVHJZUUEI3SEP...
participants (1)
-
Jiří Sléžka