On Mon, May 8, 2017 at 10:46 PM, Ryan Housand < rhousand@empoweredbenefits.com> wrote:

We have three gluster shares (_data, _engine, _export) created by a brick located on three of our VM hosts. See output from "gluster volume info" below:

Volume Name: data Type: Replicate Volume ID: c07fdf43-b838-4e4b-bb26-61dbf406cb57 Status: Started Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: vmhost01-chi.empoweredbenefits.com:/gluster/brick2/data Brick2: vmhost02-chi.empoweredbenefits.com:/gluster/brick2/data Brick3: vmhost03-chi.empoweredbenefits.com:/gluster/brick2/data (arbiter) Options Reconfigured: performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: off cluster.quorum-type: auto cluster.server-quorum-type: server storage.owner-uid: 36 storage.owner-gid: 36 features.shard: on features.shard-block-size: 512MB performance.low-prio-threads: 32 cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 6 network.ping-timeout: 30 user.cifs: off nfs.disable: on performance.strict-o-direct: on

Volume Name: engine Type: Distributed-Replicate Volume ID: 25455f13-75ba-4bc6-926a-d06ee7c5859a Status: Started Number of Bricks: 2 x (2 + 1) = 6 Transport-type: tcp Bricks: Brick1: vmhost01-chi.empoweredbenefits.com:/gluster/brick1/engine Brick2: vmhost02-chi.empoweredbenefits.com:/gluster/brick1/engine Brick3: vmhost03-chi.empoweredbenefits.com:/gluster/brick1/engine (arbiter) Brick4: vmhost04-chi:/mnt/engine Brick5: vmhost05-chi:/mnt/engine Brick6: vmhost06-chi:/mnt/engine (arbiter) Options Reconfigured: performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: off cluster.quorum-type: auto cluster.server-quorum-type: server storage.owner-uid: 36 storage.owner-gid: 36 features.shard: on features.shard-block-size: 512MB performance.low-prio-threads: 32 cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 6 network.ping-timeout: 30 user.cifs: off nfs.disable: on performance.strict-o-direct: on

Volume Name: export Type: Replicate Volume ID: a4c3a49a-fa83-4a62-9523-989c8e016c35 Status: Started Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: vmhost01-chi.empoweredbenefits.com:/gluster/brick3/export Brick2: vmhost02-chi.empoweredbenefits.com:/gluster/brick3/export Brick3: vmhost03-chi.empoweredbenefits.com:/gluster/brick3/export (arbiter) Options Reconfigured: performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: off cluster.quorum-type: auto cluster.server-quorum-type: server storage.owner-uid: 36 storage.owner-gid: 36 features.shard: on features.shard-block-size: 512MB performance.low-prio-threads: 32 cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 6 network.ping-timeout: 30 user.cifs: off nfs.disable: on performance.strict-o-direct: on

Our issue is that we ran out of space on our gluster-engine bricks which caused our Hosted Engine vm to crash. We added additional bricks from new VM Hosts (see vmhost05 to vmhost06 above) but we still are unable to restart our Hosted Engine due to the first three space being depleted. My understanding is that I need to extend the bricks that are 100% full on our engine partition. Is it the best practice to stop the glusterd service or can I use "gloster volume stop engine" to only stop the volume I need to extend? Also, if I need to stop glusterd will my VMs hosted on my ovirt cluster be affected by mount points export and data being off line?

Adding the 3 bricks to engine does not redistribute the data. You need to run rebalance on gluster volume engine for this. There's a bug currently that rebalance causes corruption when performed with ongoing IO on the volume. I think the best way for you to do this, is put hosted-engine to global maintenance, stop the hosted-engine and rebalance the engine gluster volume. What was the original size of the engine gluster volume? (Curious to understand why you ran out of space) The VMs running on data gluster volume should not be affected by this.

Thanks,

Ryan _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users