3:48 p.m.
yesterday I had a disk failure on my stack of 3 Ovirt 4.4.1 node
on each server I have 3 Bricks (engine, data, vmstore)
brick data 4X600Gig raid0. /dev/gluster_vg_sdb/gluster_lv_data mount /gluster_bricks/data
brick engine 2X1TB raid1 /dev/gluster_vg_sdc/gluster_lv_engine mount
/gluster_bricks/engine
brick vmstore 2X1TB raid1 /dev/gluster_vg_sdc/gluster_lv_vmstore mount
/gluster_bricks/vmstore
Everything was configured by the gui (hyperconverge and hosted-engine)
It is the raid0 of the 2nd server who broke.
all VMs were automatically moved to the other two servers, I haven't lost any data.
the host2 is now in maintenance mode.
I am going to buy 4 new SSD disks to replace the 4 disks of the defective raid0.
When I'm going to erase the faulty raid0 and create the new raid with the new disks on
the raid controler, how do I add in ovirt so that they resynchronize with the other bricks
data?
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 172.16.70.91:/gluster_bricks/data/dat
a 49153 0 Y 79168
Brick 172.16.70.92:/gluster_bricks/data/dat
a N/A N/A N N/A
Brick 172.16.70.93:/gluster_bricks/data/dat
a 49152 0 Y 3095
Self-heal Daemon on localhost N/A N/A Y 2528
Self-heal Daemon on 172.16.70.91 N/A N/A Y 225523
Self-heal Daemon on 172.16.70.93 N/A N/A Y 3121
10:33 p.m.
I'm not sure about the GUI (but I think it has the option) , but under command line
you got several options.
1. Use gluster's remove-brick replica 2 (with flag force)and then 'add-brick
replica 3'2. Use the old way 'replace-brick'
If you need guidance, please provide the 'gluster volume info <VOLUME>' .
Best Regards,Strahil Nikolov
On Tue, Jun 22, 2021 at 2:01, Dominique D<dominique.deschenes(a)gcgenicom.com>
wrote: yesterday I had a disk failure on my stack of 3 Ovirt 4.4.1 node
on each server I have 3 Bricks (engine, data, vmstore)
brick data 4X600Gig raid0. /dev/gluster_vg_sdb/gluster_lv_data mount /gluster_bricks/data
brick engine 2X1TB raid1 /dev/gluster_vg_sdc/gluster_lv_engine mount
/gluster_bricks/engine
brick vmstore 2X1TB raid1 /dev/gluster_vg_sdc/gluster_lv_vmstore mount
/gluster_bricks/vmstore
Everything was configured by the gui (hyperconverge and hosted-engine)
It is the raid0 of the 2nd server who broke.
all VMs were automatically moved to the other two servers, I haven't lost any data.
the host2 is now in maintenance mode.
I am going to buy 4 new SSD disks to replace the 4 disks of the defective raid0.
When I'm going to erase the faulty raid0 and create the new raid with the new disks on
the raid controler, how do I add in ovirt so that they resynchronize with the other bricks
data?
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 172.16.70.91:/gluster_bricks/data/dat
a 49153 0 Y 79168
Brick 172.16.70.92:/gluster_bricks/data/dat
a N/A N/A N N/A
Brick 172.16.70.93:/gluster_bricks/data/dat
a 49152 0 Y 3095
Self-heal Daemon on localhost N/A N/A Y 2528
Self-heal Daemon on 172.16.70.91 N/A N/A Y 225523
Self-heal Daemon on 172.16.70.93 N/A N/A Y 3121
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/P4XAPOA35NE...
12:26 a.m.
Adding Prajith,
Will replace host work in this case..? If yes please update your thoughts
here
On Tue, Jun 22, 2021 at 9:09 AM Strahil Nikolov via Users <users(a)ovirt.org>
wrote:
I'm not sure about the GUI (but I think it has the option) , but
under
command line you got several options.
1. Use gluster's remove-brick replica 2 (with flag force)
and then 'add-brick replica 3'
2. Use the old way 'replace-brick'
If you need guidance, please provide the 'gluster volume info <VOLUME>' .
Best Regards,
Strahil Nikolov
On Tue, Jun 22, 2021 at 2:01, Dominique D
<dominique.deschenes(a)gcgenicom.com> wrote:
yesterday I had a disk failure on my stack of 3 Ovirt 4.4.1 node
on each server I have 3 Bricks (engine, data, vmstore)
brick data 4X600Gig raid0. /dev/gluster_vg_sdb/gluster_lv_data mount
/gluster_bricks/data
brick engine 2X1TB raid1 /dev/gluster_vg_sdc/gluster_lv_engine mount
/gluster_bricks/engine
brick vmstore 2X1TB raid1 /dev/gluster_vg_sdc/gluster_lv_vmstore mount
/gluster_bricks/vmstore
Everything was configured by the gui (hyperconverge and hosted-engine)
It is the raid0 of the 2nd server who broke.
all VMs were automatically moved to the other two servers, I haven't lost
any data.
the host2 is now in maintenance mode.
I am going to buy 4 new SSD disks to replace the 4 disks of the defective
raid0.
When I'm going to erase the faulty raid0 and create the new raid with the
new disks on the raid controler, how do I add in ovirt so that they
resynchronize with the other bricks data?
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 172.16.70.91:/gluster_bricks/data/dat
a 49153 0 Y 79168
Brick 172.16.70.92:/gluster_bricks/data/dat
a N/A N/A N N/A
Brick 172.16.70.93:/gluster_bricks/data/dat
a 49152 0 Y 3095
Self-heal Daemon on localhost N/A N/A Y 2528
Self-heal Daemon on 172.16.70.91 N/A N/A Y
225523
Self-heal Daemon on 172.16.70.93 N/A N/A Y 3121
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/P4XAPOA35NE...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KHBYPWP23TI...
1:13 a.m.
Hi @Dominique,
Once you have added the SSD's to your host, you can follow this link[1] to
do a replace host.
Replace host is essentially
1) Preparing your node based on your current healthy nodes in your cluster
2) Post adding the node in the cluster, gluster syncs with the nodes, and
your cluster will be back to healthy.
[1]
https://github.com/gluster/gluster-ansible/blob/master/playbooks/hc-ansib...
Regards,
Prajith,
On Tue, Jun 22, 2021 at 10:56 AM Ritesh Chikatwar <rchikatw(a)redhat.com>
wrote:
Adding Prajith,
Will replace host work in this case..? If yes please update your thoughts
here
On Tue, Jun 22, 2021 at 9:09 AM Strahil Nikolov via Users <users(a)ovirt.org>
wrote:
> I'm not sure about the GUI (but I think it has the option) , but under
> command line you got several options.
>
> 1. Use gluster's remove-brick replica 2 (with flag force)
> and then 'add-brick replica 3'
> 2. Use the old way 'replace-brick'
>
> If you need guidance, please provide the 'gluster volume info <VOLUME>'
.
>
> Best Regards,
> Strahil Nikolov
>
> On Tue, Jun 22, 2021 at 2:01, Dominique D
> <dominique.deschenes(a)gcgenicom.com> wrote:
> yesterday I had a disk failure on my stack of 3 Ovirt 4.4.1 node
>
> on each server I have 3 Bricks (engine, data, vmstore)
>
> brick data 4X600Gig raid0. /dev/gluster_vg_sdb/gluster_lv_data mount
> /gluster_bricks/data
> brick engine 2X1TB raid1 /dev/gluster_vg_sdc/gluster_lv_engine mount
> /gluster_bricks/engine
> brick vmstore 2X1TB raid1 /dev/gluster_vg_sdc/gluster_lv_vmstore mount
> /gluster_bricks/vmstore
>
> Everything was configured by the gui (hyperconverge and hosted-engine)
>
> It is the raid0 of the 2nd server who broke.
>
> all VMs were automatically moved to the other two servers, I haven't lost
> any data.
>
> the host2 is now in maintenance mode.
>
> I am going to buy 4 new SSD disks to replace the 4 disks of the defective
> raid0.
>
> When I'm going to erase the faulty raid0 and create the new raid with the
> new disks on the raid controler, how do I add in ovirt so that they
> resynchronize with the other bricks data?
>
> Status of volume: data
> Gluster process TCP Port RDMA Port Online
> Pid
>
> ------------------------------------------------------------------------------
> Brick 172.16.70.91:/gluster_bricks/data/dat
> a 49153 0 Y
> 79168
> Brick 172.16.70.92:/gluster_bricks/data/dat
> a N/A N/A N N/A
> Brick 172.16.70.93:/gluster_bricks/data/dat
> a 49152 0 Y 3095
> Self-heal Daemon on localhost N/A N/A Y 2528
> Self-heal Daemon on 172.16.70.91 N/A N/A Y
> 225523
> Self-heal Daemon on 172.16.70.93 N/A N/A Y
> 3121
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/P4XAPOA35NE...
>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KHBYPWP23TI...
>
5:56 p.m.
Hi Strahil
here :
[root@ovnode2 ~]# gluster volume info data
Volume Name: data
Type: Replicate
Volume ID: c6535ef6-c5c5-4097-8eb4-90b9254570c5
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.16.70.91:/gluster_bricks/data/data
Brick2: 172.16.70.92:/gluster_bricks/data/data
Brick3: 172.16.70.93:/gluster_bricks/data/data
Options Reconfigured:
cluster.granular-entry-heal: enable
performance.strict-o-direct: on
network.ping-timeout: 30
storage.owner-gid: 36
storage.owner-uid: 36
server.event-threads: 4
client.event-threads: 4
cluster.choose-local: off
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: off
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: on
Dominique Deschênes
Ingénieur chargé de projets, Responsable TI
816, boulevard Guimond, Longueuil J4G 1T5
450 670-8383 x105 450 670-2259
----- Message reçu -----
De: Strahil Nikolov via Users (users(a)ovirt.org)
Date: 21/06/21 23:38
À: dominique.deschenes(a)gcgenicom.com, users(a)ovirt.org
Objet: [ovirt-users] Re: Disk (brick) failure on my stack
I'm not sure about the GUI (but I think it has the option) , but under command line
you got several options.
1. Use gluster's remove-brick replica 2 (with flag force)
and then 'add-brick replica 3'
2. Use the old way 'replace-brick'
If you need guidance, please provide the 'gluster volume info <VOLUME>' .
Best Regards,
Strahil Nikolov
On Tue, Jun 22, 2021 at 2:01, Dominique D
<dominique.deschenes(a)gcgenicom.com> wrote:
yesterday I had a disk failure on my stack of 3 Ovirt 4.4.1 node
on each server I have 3 Bricks (engine, data, vmstore)
brick data 4X600Gig raid0. /dev/gluster_vg_sdb/gluster_lv_data mount /gluster_bricks/data
brick engine 2X1TB raid1 /dev/gluster_vg_sdc/gluster_lv_engine mount
/gluster_bricks/engine
brick vmstore 2X1TB raid1 /dev/gluster_vg_sdc/gluster_lv_vmstore mount
/gluster_bricks/vmstore
Everything was configured by the gui (hyperconverge and hosted-engine)
It is the raid0 of the 2nd server who broke.
all VMs were automatically moved to the other two servers, I haven't lost any data.
the host2 is now in maintenance mode.
I am going to buy 4 new SSD disks to replace the 4 disks of the defective raid0.
When I'm going to erase the faulty raid0 and create the new raid with the new disks on
the raid controler, how do I add in ovirt so that they resynchronize with the other bricks
data?
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 172.16.70.91:/gluster_bricks/data/dat
a 49153 0 Y 79168
Brick 172.16.70.92:/gluster_bricks/data/dat
a N/A N/A N N/A
Brick 172.16.70.93:/gluster_bricks/data/dat
a 49152 0 Y 3095
Self-heal Daemon on localhost N/A N/A Y 2528
Self-heal Daemon on 172.16.70.91 N/A N/A Y 225523
Self-heal Daemon on 172.16.70.93 N/A N/A Y 3121
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/P4XAPOA35NE...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KHBYPWP23TI...
10:49 p.m.
Are you using HW raid ? The stripe size and the ammount of physical disks is very
important for the storage alignment.
Also do you use VDO ?
LVM allignment:
pvcreate --dataalignment alignment_value disk
For example if you use 12 disks in raid6 with 128KiB stripe unit size, alignment_value
will be '1280k' ( 10 disks without the parity multiplied by 128KiB).
vgcreate --physicalextentsize extentsize VOLGROUP physical_volume
With the same example, extentsize should be again '1280k'.
If you will be relying on gluster snapshots, you should create thinpool with 1-2 MiB chunk
size:
lvcreate --thinpool VOLGROUP/thin_pool --size 800g --chunksize 1280k --poolmetadatasize
16G --zero n
This example is again for raid6 with 12 disks and 128KiB stripe size. Note that the
poolmetadata cannot be changed and it's always smart to set it to the max (16G)
For the LV ontop the thinpool you won't need any alignment:
lvcreate --thin --name LV_name --virtualsize LV_size VOLGROUP/thin_pool
FS alignment:Creating your XFS will need alignment again and also inode size of 512.
mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10 /dev/vg/thinlv
Note: su= HW stripe size, sw= data disks in the raid
I would mount that Filesystem with the following options(SELINUX enabled):
context=system_u:object_r:glusterd_brick_t:s0,noatime,inode64
If you disabled SELINUX , then you won't need the 'context=' option.
Once you have your brick restored you can remove the old and readd the new one:
gluster volume remove-brick <VOLUME> replica 2 <serverX:/PATH/to/failed/brick>
force
Note: ensure that you are removing the brick that has really failed. Full payh to brick
can be taken from gluster volume info. The previous command will turn the volume into
'replica 2' which is sensitive to splitbrain. Don't do any maintenance till
you add and heal the 3rd brick.
Next readd the fixed brick:
gluster volume add-brick <VOLUME> replica 3 <serverX:/PATH/to/new/brick>
You can force a full heal:
gluster volume heal <VOLUME> full
About the disk alignment, you can read:
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1...
I know that there is an ansible role that can do that for you, but I have never used it
directly.
If you use VDO, I can try to find what I'm using (which might not be the best).
Best Regards,Strahil Nikolov
On Wed, Jun 23, 2021 at 1:58, Dominique
Deschênes<dominique.deschenes(a)gcgenicom.com> wrote:
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/RRC5X7DBCDK...
587
days inactive
590
days old
5 comments
5 participants
participants (5)
-
Dominique D -
Dominique Deschênes
-
Prajith Kesava Prasad -
Ritesh Chikatwar -
Strahil Nikolov