4:06 p.m.
The switches above our environment had some VPC issues and the port channels went offline.
The ports that had issues belonged to 2 of the gfs nodes in our environment. We have 3
storage nodes total with the 3rd being the arbiter. I wound up rebooting the first 2 nodes
and everything came back happy. After a few hours I noticed that the storage was up but
complaining about being out of sync and needing healing. Within the hour I noticed a VM
had paused itself due to storage issues. This is a small environment, for now, with only
30 VM's. I am new to Ovirt so this is uncharted territory for me. I am tailing some
logs and things look sort of normal and google is sending me down a wormhole.
If I run "gluster volume heal cps-vms-gfs info" this number seems to be changing
pretty regularly. Logs are showing lots of entries like this:
[2018-11-08 21:55:05.996675] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk] 0-cps-vms-gfs-client-1: Server and Client
lk-version numbers are not same, reopening the fds
[2018-11-08 21:55:05.997693] I [MSGID: 108002] [afr-common.c:5312:afr_notify]
0-cps-vms-gfs-replicate-0: Client-quorum is met
[2018-11-08 21:55:05.997717] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-cps-vms-gfs-client-1: Server lk
version = 1
I guess I am curious what else should I be looking for? Is this just taking forever to
heal? Is there something else I can run or I should do to verify things are actually
getting better? I ran an actual heal command and it cleared everything for a few seconds
and then the entries started to populate again when I did the info command.
[root@cps-vms-gfs01 glusterfs]# gluster volume status
Status of volume: cps-vms-gfs
Gluster process
TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.8.255.1:/gluster/cps-vms-gfs01/brick
49152 0 Y 4054
Brick 10.8.255.2:/gluster/cps-vms-gfs02/brick
49152 0 Y 4144
Brick 10.8.255.3:/gluster/cps-vms-gfs03/brick
49152 0 Y 4294
Self-heal Daemon on localhost N/A N/A Y 4279
Self-heal Daemon on cps-vms-gfs02.cisco.com N/A N/A Y 5185
Self-heal Daemon on 10.196.152.145 N/A N/A Y 50948
Task Status of Volume cps-vms-gfs
------------------------------------------------------------------------------
There are no active volume tasks
I am running ovirt 4.2.5 and gluster 3.12.11.
Thanks!
10:58 p.m.
On Fri, Nov 9, 2018 at 3:42 AM Dev Ops <sipandbite(a)hotmail.com> wrote:
The switches above our environment had some VPC issues and the port channels went
offline. The ports that had issues belonged to 2 of the gfs nodes in our environment. We
have 3 storage nodes total with the 3rd being the arbiter. I wound up rebooting the first
2 nodes and everything came back happy. After a few hours I noticed that the storage was
up but complaining about being out of sync and needing healing. Within the hour I noticed
a VM had paused itself due to storage issues. This is a small environment, for now, with
only 30 VM's. I am new to Ovirt so this is uncharted territory for me. I am tailing
some logs and things look sort of normal and google is sending me down a wormhole.
If I run "gluster volume heal cps-vms-gfs info" this number seems to be
changing pretty regularly. Logs are showing lots of entries like this:
[2018-11-08 21:55:05.996675] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk] 0-cps-vms-gfs-client-1: Server and Client
lk-version numbers are not same, reopening the fds
[2018-11-08 21:55:05.997693] I [MSGID: 108002] [afr-common.c:5312:afr_notify]
0-cps-vms-gfs-replicate-0: Client-quorum is met
[2018-11-08 21:55:05.997717] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-cps-vms-gfs-client-1: Server lk
version = 1
I guess I am curious what else should I be looking for? Is this just taking forever to
heal? Is there something else I can run or I should do to verify things are actually
getting better? I ran an actual heal command and it cleared everything for a few seconds
and then the entries started to populate again when I did the info command.
[root@cps-vms-gfs01 glusterfs]# gluster volume status
Status of volume: cps-vms-gfs
Gluster process
TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.8.255.1:/gluster/cps-vms-gfs01/brick
49152 0 Y 4054
Brick 10.8.255.2:/gluster/cps-vms-gfs02/brick
49152 0 Y 4144
Brick 10.8.255.3:/gluster/cps-vms-gfs03/brick
49152 0 Y 4294
Self-heal Daemon on localhost N/A N/A Y 4279
Self-heal Daemon on cps-vms-gfs02.cisco.com N/A N/A Y 5185
Self-heal Daemon on 10.196.152.145 N/A N/A Y 50948
Task Status of Volume cps-vms-gfs
------------------------------------------------------------------------------
There are no active volume tasks
I am running ovirt 4.2.5 and gluster 3.12.11.
Can you provide output of gluster volume heal cps-vms-gfs info, and
the logs from /var/log/glusterfs/glfsheal-cps-vms-gfs.log and the
brick logs from /var/log/glusterfs/bricks for this volume.
>
> Thanks!
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MDZXUZQSWQU...
2:35 a.m.
Just a quick note the volume in question is actually called bgl-vms-gfs. The original
message is still valid.
[root@bgl-vms-gfs03 bricks]# gluster volume heal bgl-vms-gfs info
Brick 10.8.255.1:/gluster/bgl-vms-gfs01/brick
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.989
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.988
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.423
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.612
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.614
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.611
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.236
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.48
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.52
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.423
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.424
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.611
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.612
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.799
/.shard/18954415-3210-4d93-8591-0b3e1e5b3a16.498
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.1175
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.1551
/c71bb8b0-c669-4bf6-8348-14aafd4a805f/images/9dc54d22-7cb3-4e07-adbb-70f0ec5b7e6b/5f8515f7-3fae-4af6-adc4-d38426a9aa72
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.611
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.424
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.50
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.51
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.424
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.236
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.425
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.428
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.427
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.614
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.1363
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.238
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.428
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.612
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.423
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.614
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.987
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.429
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.429
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.241
/c71bb8b0-c669-4bf6-8348-14aafd4a805f/dom_md/ids
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.987
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.987
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.241
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.429
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.424
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.987
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.987
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.238
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.428
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.238
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.611
/.shard/18954415-3210-4d93-8591-0b3e1e5b3a16.504
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.238
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.428
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.612
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.614
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.241
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.429
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.236
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.989
/.shard/18954415-3210-4d93-8591-0b3e1e5b3a16.909
/__DIRECT_IO_TEST__
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.240
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.429
/.shard/18954415-3210-4d93-8591-0b3e1e5b3a16.497
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.238
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.428
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.241
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.991
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.236
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.990
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.48
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.52
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.424
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.425
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.799
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.1175
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.1551
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.1363
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.612
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.614
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.236
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.611
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.989
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.988
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.990
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.991
Status: Connected
Number of entries: 86
Brick 10.8.255.2:/gluster/bgl-vms-gfs02/brick
Status: Connected
Number of entries: 0
Brick 10.8.255.3:/gluster/bgl-vms-gfs03/brick
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.236
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.48
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.52
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.423
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.424
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.611
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.612
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.799
/.shard/18954415-3210-4d93-8591-0b3e1e5b3a16.498
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.1175
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.1551
/c71bb8b0-c669-4bf6-8348-14aafd4a805f/images/9dc54d22-7cb3-4e07-adbb-70f0ec5b7e6b/5f8515f7-3fae-4af6-adc4-d38426a9aa72
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.611
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.424
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.50
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.51
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.424
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.236
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.425
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.428
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.427
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.614
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.1363
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.238
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.428
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.612
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.423
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.614
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.987
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.429
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.429
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.241
/c71bb8b0-c669-4bf6-8348-14aafd4a805f/dom_md/ids
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.987
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.987
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.241
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.429
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.424
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.987
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.987
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.238
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.428
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.238
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.611
/.shard/18954415-3210-4d93-8591-0b3e1e5b3a16.504
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.238
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.428
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.612
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.614
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.241
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.429
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.236
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.989
/.shard/18954415-3210-4d93-8591-0b3e1e5b3a16.909
/__DIRECT_IO_TEST__
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.240
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.429
/.shard/18954415-3210-4d93-8591-0b3e1e5b3a16.497
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.238
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.428
/.shard/5bb5bc8b-abfb-4ab8-9f12-cbc020b3d50f.241
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.989
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.988
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.991
/.shard/bd0bf192-e0e1-4b72-85cb-fa3497c555be.423
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.236
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.612
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.611
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.614
/.shard/cca2d4d0-7254-49c5-9db0-c9aaeb34c479.990
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.48
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.52
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.236
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.424
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.425
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.799
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.1175
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.1551
/.shard/6792d5d0-1bd2-41cf-a48e-dbe015d3e9fd.1363
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.612
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.611
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.614
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.989
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.988
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.990
/.shard/dfe31381-6b91-4eb1-9050-0332182e424a.991
Status: Connected
Number of entries: 86
What is the preferred method for sharing log files on this site? I can upload the entire
logs if necessary. Here is a snippet of what the glfsheal file looks like:
[2018-11-07 20:31:07.110982] I [MSGID: 114046]
[client-handshake.c:1231:client_setvolume_cbk] 0-bgl-vms-gfs-client-0: Connected to
bgl-vms-gfs-client-0, attached to remo
te volume '/gluster/bgl-vms-gfs01/brick'.
[2018-11-07 20:31:07.110998] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk] 0-bgl-vms-gfs-client-0: Server and Client
lk-version numbers are not same,
reopening the fds
[2018-11-07 20:31:07.111059] I [MSGID: 108005]
[afr-common.c:5015:__afr_handle_child_up_event] 0-bgl-vms-gfs-replicate-0: Subvolume
'bgl-vms-gfs-client-0' came back up;
going online.
[2018-11-07 20:31:07.112395] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-bgl-vms-gfs-client-0: Server lk
version = 1
[2018-11-07 20:31:07.112523] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-bgl-vms-gfs-client-2:
changing port to 49152 (from 0)
[2018-11-07 20:31:07.114476] I [MSGID: 114057]
[client-handshake.c:1478:select_server_supported_programs] 0-bgl-vms-gfs-client-1: Using
Program GlusterFS 3.3, Num (12984
37), Version (330)
[2018-11-07 20:31:07.114630] I [MSGID: 114057]
[client-handshake.c:1478:select_server_supported_programs] 0-bgl-vms-gfs-client-2: Using
Program GlusterFS 3.3, Num (12984
37), Version (330)
[2018-11-07 20:31:07.115147] I [MSGID: 114046]
[client-handshake.c:1231:client_setvolume_cbk] 0-bgl-vms-gfs-client-2: Connected to
bgl-vms-gfs-client-2, attached to remo
te volume '/gluster/bgl-vms-gfs03/brick'.
[2018-11-07 20:31:07.115161] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk] 0-bgl-vms-gfs-client-2: Server and Client
lk-version numbers are not same,
reopening the fds
[2018-11-07 20:31:07.158057] I [MSGID: 114046]
[client-handshake.c:1231:client_setvolume_cbk] 0-bgl-vms-gfs-client-1: Connected to
bgl-vms-gfs-client-1, attached to remo
te volume '/gluster/bgl-vms-gfs02/brick'.
[2018-11-07 20:31:07.158078] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk] 0-bgl-vms-gfs-client-1: Server and Client
lk-version numbers are not same,
reopening the fds
[2018-11-07 20:31:07.158086] I [MSGID: 108002] [afr-common.c:5312:afr_notify]
0-bgl-vms-gfs-replicate-0: Client-quorum is met
[2018-11-07 20:31:07.161105] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-bgl-vms-gfs-client-2: Server lk
version = 1
[2018-11-07 20:31:07.161131] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-bgl-vms-gfs-client-1: Server lk
version = 1
[2018-11-07 20:31:07.163656] I [MSGID: 104041] [glfs-resolve.c:971:__glfs_active_subvol]
0-bgl-vms-gfs: switched to graph 62676c2d-766d-732d-6766-7330332d3538 (0)
[2018-11-07 20:41:09.175940] W [MSGID: 101174] [graph.c:363:_log_if_unknown_option]
0-bgl-vms-gfs-readdir-ahead: option 'parallel-readdir' is not recognized
[2018-11-07 20:41:09.175998] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New
graph 62676c2d-766d-732d-6766-7330332d3538 (0) coming up
[2018-11-07 20:41:09.176019] I [MSGID: 114020] [client.c:2360:notify]
0-bgl-vms-gfs-client-0: parent translators are ready, attempting connect on transport
[2018-11-07 20:41:09.178552] I [MSGID: 114020] [client.c:2360:notify]
0-bgl-vms-gfs-client-1: parent translators are ready, attempting connect on transport
[2018-11-07 20:41:09.178902] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-bgl-vms-gfs-client-0:
changing port to 49152 (from 0)
[2018-11-07 20:41:09.180514] I [MSGID: 114020] [client.c:2360:notify]
0-bgl-vms-gfs-client-2: parent translators are ready, attempting connect on transport
[2018-11-07 20:41:09.182777] I [MSGID: 114057]
[client-handshake.c:1478:select_server_supported_programs] 0-bgl-vms-gfs-client-0: Using
Program GlusterFS 3.3, Num (12984
37), Version (330)
[2018-11-07 20:41:09.182851] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-bgl-vms-gfs-client-1:
changing port to 49152 (from 0)
Final graph:
+------------------------------------------------------------------------------+
1: volume bgl-vms-gfs-client-0
2: type protocol/client
3: option opversion 31202
4: option clnt-lk-version 1
5: option volfile-checksum 0
6: option volfile-key bgl-vms-gfs
7: option client-version 3.12.11
8: option process-uuid
bgl-vms-gfs03-58334-2018/11/07-20:41:09:168612-bgl-vms-gfs-client-0-0-0
9: option fops-version 1298437
10: option ping-timeout 42
11: option remote-host 10.8.255.1
12: option remote-subvolume /gluster/bgl-vms-gfs01/brick
13: option transport-type socket
14: option transport.address-family inet
15: option username 215d1f1c-087d-40fd-bdfb-6c7774045252
16: option password 2372bb91-cdd8-48ce-bef5-80532b0fe0cc
17: option filter-O_DIRECT enable
18: option transport.tcp-user-timeout 0
19: option transport.socket.keepalive-time 20
20: option transport.socket.keepalive-interval 2
21: option transport.socket.keepalive-count 9
22: option send-gids true
23: end-volume
24:
25: volume bgl-vms-gfs-client-1
26: type protocol/client
27: option ping-timeout 42
28: option remote-host 10.8.255.2
29: option remote-subvolume /gluster/bgl-vms-gfs02/brick
30: option transport-type socket
31: option transport.address-family inet
32: option username 215d1f1c-087d-40fd-bdfb-6c7774045252
33: option password 2372bb91-cdd8-48ce-bef5-80532b0fe0cc
34: option filter-O_DIRECT enable
35: option transport.tcp-user-timeout 0
36: option transport.socket.keepalive-time 20
37: option transport.socket.keepalive-interval 2
38: option transport.socket.keepalive-count 9
39: option send-gids true
40: end-volume
41:
42: volume bgl-vms-gfs-client-2
43: type protocol/client
44: option ping-timeout 42
45: option remote-host 10.8.255.3
46: option remote-subvolume /gluster/bgl-vms-gfs03/brick
47: option transport-type socket
48: option transport.address-family inet
49: option username 215d1f1c-087d-40fd-bdfb-6c7774045252
50: option password 2372bb91-cdd8-48ce-bef5-80532b0fe0cc
51: option filter-O_DIRECT enable
52: option transport.tcp-user-timeout 0
53: option transport.socket.keepalive-time 20
54: option transport.socket.keepalive-interval 2
55: option transport.socket.keepalive-count 9
56: option send-gids true
57: end-volume
58:
59: volume bgl-vms-gfs-replicate-0
60: type cluster/replicate
61: option background-self-heal-count 0
62: option afr-pending-xattr
bgl-vms-gfs-client-0,bgl-vms-gfs-client-1,bgl-vms-gfs-client-2
63: option arbiter-count 1
64: option data-self-heal-algorithm full
65: option eager-lock enable
66: option quorum-type auto
67: option shd-max-threads 8
68: option shd-wait-qlength 10000
69: option locking-scheme granular
70: option use-compound-fops off
71: subvolumes bgl-vms-gfs-client-0 bgl-vms-gfs-client-1 bgl-vms-gfs-client-2
72: end-volume
73:
74: volume bgl-vms-gfs-dht
75: type cluster/distribute
76: option lock-migration off
77: subvolumes bgl-vms-gfs-replicate-0
78: end-volume
79:
80: volume bgl-vms-gfs-shard
81: type features/shard
82: subvolumes bgl-vms-gfs-dht
83: end-volume
84:
85: volume bgl-vms-gfs-write-behind
86: type performance/write-behind
87: subvolumes bgl-vms-gfs-shard
88: end-volume
89:
90: volume bgl-vms-gfs-readdir-ahead
91: type performance/readdir-ahead
92: option parallel-readdir off
93: option rda-request-size 131072
94: option rda-cache-limit 10MB
95: subvolumes bgl-vms-gfs-write-behind
96: end-volume
97:
98: volume bgl-vms-gfs-open-behind
99: type performance/open-behind
100: subvolumes bgl-vms-gfs-readdir-ahead
101: end-volume
102:
103: volume bgl-vms-gfs-md-cache
104: type performance/md-cache
105: subvolumes bgl-vms-gfs-open-behind
106: end-volume
107:
108: volume bgl-vms-gfs
109: type debug/io-stats
110: option log-level INFO
111: option latency-measurement off
112: option count-fop-hits off
113: subvolumes bgl-vms-gfs-md-cache
114: end-volume
115:
116: volume meta-autoload
117: type meta
118: subvolumes bgl-vms-gfs
119: end-volume
120:
Here is a snippet from the brick log file:
[2018-11-07 20:31:07.114742] I [addr.c:55:compare_addr_and_update]
0-/gluster/bgl-vms-gfs03/brick: allowed = "10.8.*", received addr =
"10.8.255.3"
[2018-11-07 20:31:07.114766] I [login.c:111:gf_auth] 0-auth/login: allowed user names:
215d1f1c-087d-40fd-bdfb-6c7774045252
[2018-11-07 20:31:07.114776] I [MSGID: 115029] [server-handshake.c:793:server_setvolume]
0-bgl-vms-gfs-server: accepted client from bgl-vms-gfs03-58190-2018/11/07-20:31:
07:90406-bgl-vms-gfs-client-2-0-0 (version: 3.12.11)
[2018-11-07 20:31:08.996695] I [MSGID: 115036] [server.c:527:server_rpc_notify]
0-bgl-vms-gfs-server: disconnecting connection from bgl-vms-gfs03-58190-2018/11/07-20:31:
07:90406-bgl-vms-gfs-client-2-0-0
[2018-11-07 20:31:08.996821] I [MSGID: 101055] [client_t.c:443:gf_client_unref]
0-bgl-vms-gfs-server: Shutting down connection bgl-vms-gfs03-58190-2018/11/07-20:31:07:90
406-bgl-vms-gfs-client-2-0-0
[2018-11-07 20:34:31.413885] I [addr.c:55:compare_addr_and_update]
0-/gluster/bgl-vms-gfs03/brick: allowed = "10.8.*", received addr =
"10.8.255.1"
[2018-11-07 20:34:31.413920] I [login.c:111:gf_auth] 0-auth/login: allowed user names:
215d1f1c-087d-40fd-bdfb-6c7774045252
[2018-11-07 20:34:31.413930] I [MSGID: 115029] [server-handshake.c:793:server_setvolume]
0-bgl-vms-gfs-server: accepted client from bgl-vms-gfs01-30272-2018/11/07-20:35:
04:88283-bgl-vms-gfs-client-2-0-0 (version: 3.12.11)
[2018-11-07 20:34:31.417331] I [MSGID: 115036] [server.c:527:server_rpc_notify]
0-bgl-vms-gfs-server: disconnecting connection from bgl-vms-gfs01-30272-2018/11/07-20:35:
04:88283-bgl-vms-gfs-client-2-0-0
[2018-11-07 20:34:31.417504] I [MSGID: 101055] [client_t.c:443:gf_client_unref]
0-bgl-vms-gfs-server: Shutting down connection bgl-vms-gfs01-30272-2018/11/07-20:35:04:88
283-bgl-vms-gfs-client-2-0-0
[2018-11-07 20:39:33.594278] I [addr.c:55:compare_addr_and_update]
0-/gluster/bgl-vms-gfs03/brick: allowed = "10.8.*", received addr =
"10.8.255.1"
[2018-11-07 20:39:33.594313] I [login.c:111:gf_auth] 0-auth/login: allowed user names:
215d1f1c-087d-40fd-bdfb-6c7774045252
[2018-11-07 20:39:33.594323] I [MSGID: 115029] [server-handshake.c:793:server_setvolume]
0-bgl-vms-gfs-server: accepted client from bgl-vms-gfs01-32373-2018/11/07-20:40:
06:265383-bgl-vms-gfs-client-2-0-0 (version: 3.12.11)
[2018-11-07 20:39:33.597535] I [MSGID: 115036] [server.c:527:server_rpc_notify]
0-bgl-vms-gfs-server: disconnecting connection from bgl-vms-gfs01-32373-2018/11/07-20:40:
06:265383-bgl-vms-gfs-client-2-0-0
[2018-11-07 20:39:33.597713] I [MSGID: 101055] [client_t.c:443:gf_client_unref]
0-bgl-vms-gfs-server: Shutting down connection bgl-vms-gfs01-32373-2018/11/07-20:40:06:26
5383-bgl-vms-gfs-client-2-0-0
[2018-11-07 20:41:09.189578] I [addr.c:55:compare_addr_and_update]
0-/gluster/bgl-vms-gfs03/brick: allowed = "10.8.*", received addr =
"10.8.255.3"
[2018-11-07 20:41:09.189601] I [login.c:111:gf_auth] 0-auth/login: allowed user names:
215d1f1c-087d-40fd-bdfb-6c7774045252
[2018-11-07 20:41:09.189611] I [MSGID: 115029] [server-handshake.c:793:server_setvolume]
0-bgl-vms-gfs-server: accepted client from bgl-vms-gfs03-58334-2018/11/07-20:41:
09:168612-bgl-vms-gfs-client-2-0-0 (version: 3.12.11)
[2018-11-07 20:41:11.234496] I [MSGID: 115036] [server.c:527:server_rpc_notify]
0-bgl-vms-gfs-server: disconnecting connection from bgl-vms-gfs03-58334-2018/11/07-20:41:
09:168612-bgl-vms-gfs-client-2-0-0
[2018-11-07 20:41:11.234614] I [MSGID: 101055] [client_t.c:443:gf_client_unref]
0-bgl-vms-gfs-server: Shutting down connection bgl-vms-gfs03-58334-2018/11/07-20:41:09:16
8612-bgl-vms-gfs-client-2-0-0
1:38 a.m.
Any help would be appreciated. I have since rebooted the 3rd gluster node which is the
arbiter. This doesn't seem to want to heal.
gluster volume heal bgl-vms-gfs info |grep Number
Number of entries: 68
Number of entries: 0
Number of entries: 68
1990
days inactive
1994
days old
3 comments
2 participants
participants (2)
-
Dev Ops -
Sahina Bose