10:15 a.m.
Hello to all,
I ve a problem with gluster
[root@kvm10 ~]# gluster volume heal data info summary
Brick kvm10:/gluster_bricks/data
Status: Connected
Total Number of entries: 868
Number of entries in heal pending: 868
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick kvm320.durchhalten.intern:/gluster_bricks/data
Status: Connected
Total Number of entries: 1
Number of entries in heal pending: 1
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick kvm360.durchhalten.intern:/gluster_bricks/data
Status: Connected
Total Number of entries: 867
Number of entries in heal pending: 867
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick kvm380.durchhalten.intern:/gluster_bricks/data
Status: Connected
Total Number of entries: 868
Number of entries in heal pending: 868
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@kvm10 ~]# gluster volume heal data info split-brain
Brick kvm10:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
Brick kvm320.durchhalten.intern:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
Brick kvm360.durchhalten.intern:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
Brick kvm380.durchhalten.intern:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
As I understand there is no split-brain but 868 files ar in state heal pending.
I ve restarted every node.
I ve also tried:
[root@kvm10 ~]# gluster volume heal data full
Launching heal operation to perform full self heal on volume data has been successful
Use heal info commands to check status.
but even after a week there is no really change ( I started with 912 Number of entries in
heal pending)
can somebody tell what exactly is the problem and how can I solve it.
thank you very much
11:02 a.m.
I would try running a full heal first and give it some time to see if it
clears up. I.e. gluster volume heal <vol> full
If that doesn't work, you could try stat on every file to trigger healing
doing something like this: find /fuse-mountpoint -iname '*' -exec stat {} \;
On Mon, Jan 20, 2020 at 12:16 PM Stefan Wolf <shb256(a)gmail.com> wrote:
Hello to all,
I ve a problem with gluster
[root@kvm10 ~]# gluster volume heal data info summary
Brick kvm10:/gluster_bricks/data
Status: Connected
Total Number of entries: 868
Number of entries in heal pending: 868
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick kvm320.durchhalten.intern:/gluster_bricks/data
Status: Connected
Total Number of entries: 1
Number of entries in heal pending: 1
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick kvm360.durchhalten.intern:/gluster_bricks/data
Status: Connected
Total Number of entries: 867
Number of entries in heal pending: 867
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick kvm380.durchhalten.intern:/gluster_bricks/data
Status: Connected
Total Number of entries: 868
Number of entries in heal pending: 868
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@kvm10 ~]# gluster volume heal data info split-brain
Brick kvm10:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
Brick kvm320.durchhalten.intern:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
Brick kvm360.durchhalten.intern:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
Brick kvm380.durchhalten.intern:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
As I understand there is no split-brain but 868 files ar in state heal
pending.
I ve restarted every node.
I ve also tried:
[root@kvm10 ~]# gluster volume heal data full
Launching heal operation to perform full self heal on volume data has been
successful
Use heal info commands to check status.
but even after a week there is no really change ( I started with 912
Number of entries in heal pending)
can somebody tell what exactly is the problem and how can I solve it.
thank you very much
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PN63LC3OBQO...
12:15 p.m.
New subject: Gluster: a lof of Number of ntries in heal pending
yes, I ve already tried a full heal a week a go.
how do i perform a manual heal?
I only have this gfid:
<gfid:3704121a-14ef-43ef-9e31-fa547a0157d1>
<gfid:00a90cef-d45d-48d0-a905-44d7cdac9503>
<gfid:06f59673-7ce2-41d5-b04e-f392d93c1044>
<gfid:8a6278ba-b03b-4b9f-95a3-d2173c8bdb9f>
<gfid:d01e825e-8993-4a51-b232-1014892e9615>
<gfid:4c37710c-6481-4846-ab97-93121fda143f>
<gfid:36729813-9c5f-4199-87db-51ac77b2160e>
<gfid:71594835-6a87-435b-9612-b5f58abd9e78>
<gfid:bda55aa0-9aaa-4482-9448-c3178d5a90d6>
<gfid:c2b47c5c-89b6-49ac-bf10-1733dd8f0902>
Status: Connected
Number of entries: 868
I ve tried to heal it with:
[root@kvm10 ~]# gluster volume heal data split-brain latest-mtime
gfid:c2b47c5c-89b6-49ac-bf10-1733dd8f0902
Healing gfid:c2b47c5c-89b6-49ac-bf10-1733dd8f0902 failed: File not in split-brain.
Volume heal failed.
(the last entry)
And if i understood it correct ther is no split-brain
[root@kvm10 ~]# gluster volume heal data info split-brain
Brick kvm10:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
Brick kvm320.durchhalten.intern:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
Brick kvm360.durchhalten.intern:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
Brick kvm380.durchhalten.intern:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
10:55 p.m.
On January 20, 2020 8:15:03 PM GMT+02:00, Stefan Wolf <shb256(a)gmail.com> wrote:
yes, I ve already tried a full heal a week a go.
how do i perform a manual heal?
I only have this gfid:
<gfid:3704121a-14ef-43ef-9e31-fa547a0157d1>
<gfid:00a90cef-d45d-48d0-a905-44d7cdac9503>
<gfid:06f59673-7ce2-41d5-b04e-f392d93c1044>
<gfid:8a6278ba-b03b-4b9f-95a3-d2173c8bdb9f>
<gfid:d01e825e-8993-4a51-b232-1014892e9615>
<gfid:4c37710c-6481-4846-ab97-93121fda143f>
<gfid:36729813-9c5f-4199-87db-51ac77b2160e>
<gfid:71594835-6a87-435b-9612-b5f58abd9e78>
<gfid:bda55aa0-9aaa-4482-9448-c3178d5a90d6>
<gfid:c2b47c5c-89b6-49ac-bf10-1733dd8f0902>
Status: Connected
Number of entries: 868
I ve tried to heal it with:
[root@kvm10 ~]# gluster volume heal data split-brain latest-mtime
gfid:c2b47c5c-89b6-49ac-bf10-1733dd8f0902
Healing gfid:c2b47c5c-89b6-49ac-bf10-1733dd8f0902 failed: File not in
split-brain.
Volume heal failed.
(the last entry)
And if i understood it correct ther is no split-brain
[root@kvm10 ~]# gluster volume heal data info split-brain
Brick kvm10:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
Brick kvm320.durchhalten.intern:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
Brick kvm360.durchhalten.intern:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
Brick kvm380.durchhalten.intern:/gluster_bricks/data
Status: Connected
Number of entries in split-brain: 0
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AQMSKCYGKKK...
Hi Stefan,
Dis you recently reboot/stopped gluster on kvm320.durchhalten.intern ?
Also, what kind of volume is that - replica 4 ?
If indeed the kvm320 was rebooted, my guess is that you can rsync the files from a good
brick to this one and then run a full heal , so gluster will get notified. Don't rsync
the whole brick, just the folders that contain the 'needs-healing' files.
Of course, it's better to address it in the gluster user's forum, as many dev are
watching it and might help with the resolution (if it is a bug) .
Best Regards,
Strahil Nikolov
11:11 p.m.
New subject: Gluster: a lof of Number of ntries in heal pending
Hi Strahil,
yes it is a replica 4 set
I ve tried to stop and stop every gluster server,
and Ive rebooted every server.
or should I remove the brick and add it again?
bye
stefan
10:36 a.m.
On January 21, 2020 7:11:19 AM GMT+02:00, Stefan Wolf <shb256(a)gmail.com> wrote:
Hi Strahil,
yes it is a replica 4 set
I ve tried to stop and stop every gluster server,
and Ive rebooted every server.
or should I remove the brick and add it again?
bye
stefan
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/LLZ7NIGVYB3...
I hope you plan to add another brick or arbiter, as you are now prone to split-brain and
other issues.
Better first try with:
find /rhev/mnt/<path to where brick is mounted> -exec stat {} \;
If it doesn't heal, you can use the gluster's reset-brick or to remove the brick,
cleanup the brick contents and add it again.
If you really seek the reason your heal is failing, yiu need to check the gluster's
healing daemon logs.
Best Regards,
Strahil Nikolov
12:04 p.m.
New subject: Gluster: a lof of Number of ntries in heal pending
Hello
I hope you plan to add another brick or arbiter, as you are now prone
to split-brain and
other issues.
Yes I will add an other one, but I think this is not a problem. I ve set
cluster.server-quorum-ratio to 51% to avid the split brain problem. of course I know I
just have failure tolarance of one.
I ve solved the probelem with removeing the brick
gluster volume remove-brick data replica 3 kvm320.durchhalten.intern:/gluster_bricks/data/
force
Remove-brick force will not migrate files from the removed bricks, so they will no longer
be available on the volume.
Do you want to continue? (y/n) y
volume remove-brick commit force: success
[root@kvm380 ~]# gluster volume heal data info summary
Brick kvm10:/gluster_bricks/data
Status: Connected
Total Number of entries: 1
Number of entries in heal pending: 1
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick kvm360.durchhalten.intern:/gluster_bricks/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick kvm380.durchhalten.intern:/gluster_bricks/data
Status: Connected
Total Number of entries: 1
Number of entries in heal pending: 1
Number of entries in split-brain: 0
Number of entries possibly healing: 0
after this heal pending was ok for me.
than I removed all files - from this node - and added it back again.
now everything is find
Best Regards
Stefan
1563
days inactive
1564
days old
6 comments
3 participants
participants (3)
-
Jayme -
Stefan Wolf -
Strahil Nikolov