Hello to all,
I ve done a normal host updadet at the webfrontend.
After the reboot the host is NonResponsive.
It is a HCI setup with glusterfs
But mount shows on this host that only engine is mountet.
The data volume is not mounted.
[root@kvm320 ~]# mount|grep _engine
kvm380.durchhalten.intern:/engine on /rhev/data-center/mnt/glusterSD/kvm380.durchhalten.intern:_engine type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@kvm320 ~]# mount|grep _data
[root@kvm320 ~]# #
on every other host it looks like this
[root@kvm10 ~]# mount|grep _engine
kvm380.durchhalten.intern:/engine on /rhev/data-center/mnt/glusterSD/kvm380.durchhalten.intern:_engine type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@kvm10 ~]# mount|grep _data
kvm380.durchhalten.intern:/data on /rhev/data-center/mnt/glusterSD/kvm380.durchhalten.intern:_data type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@kvm10 ~]#
here so more informaton
[root@kvm320 ~]# systemctl status glusterd -l
● glusterd.service - GlusterFS, a clustered file-system server
Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
Active: active (running) since Fr 2020-03-13 14:19:03 CET; 1h 49min ago
Docs: man:glusterd(8)
Process: 9263 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 9264 (glusterd)
Tasks: 114
CGroup: /system.slice/glusterd.service
├─3839 /usr/sbin/glusterfsd -s kvm320.durchhalten.intern --volfile-id data.kvm320.durchhalten.intern.gluster_bricks-data -p /var/run/gluster/vols/data/kvm320.durchhalten.intern-gluster_bricks-data.pid -S /var/run/gluster/1fd58e7c80335308.socket --brick-name /gluster_bricks/data -l /var/log/glusterfs/bricks/gluster_bricks-data.log --xlator-option *-posix.glusterd-uuid=ce474774-436a-41d3-bfdd-ab153ac77830 --process-name brick --brick-port 49152 --xlator-option data-server.listen-port=49152
├─3896 /usr/sbin/glusterfsd -s kvm320.durchhalten.intern --volfile-id engine.kvm320.durchhalten.intern.gluster_bricks-engine -p /var/run/gluster/vols/engine/kvm320.durchhalten.intern-gluster_bricks-engine.pid -S /var/run/gluster/5d4bcd552e3a3806.socket --brick-name /gluster_bricks/engine -l /var/log/glusterfs/bricks/gluster_bricks-engine.log --xlator-option *-posix.glusterd-uuid=ce474774-436a-41d3-bfdd-ab153ac77830 --process-name brick --brick-port 49153 --xlator-option engine-server.listen-port=49153
├─4032 /usr/sbin/glusterfsd -s kvm320.durchhalten.intern --volfile-id home.kvm320.durchhalten.intern.gluster_bricks-home -p /var/run/gluster/vols/home/kvm320.durchhalten.intern-gluster_bricks-home.pid -S /var/run/gluster/050dbbce51bc7cb8.socket --brick-name /gluster_bricks/home -l /var/log/glusterfs/bricks/gluster_bricks-home.log --xlator-option *-posix.glusterd-uuid=ce474774-436a-41d3-bfdd-ab153ac77830 --process-name brick --brick-port 49154 --xlator-option home-server.listen-port=49154
├─9264 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
└─9381 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/6b30e1d260d31419.socket --xlator-option *replicate*.node-uuid=ce474774-436a-41d3-bfdd-ab153ac77830 --process-name glustershd --client-pid=-6
Mär 13 14:19:02 kvm320.durchhalten.intern systemd[1]: Starting GlusterFS, a clustered file-system server...
Mär 13 14:19:03 kvm320.durchhalten.intern systemd[1]: Started GlusterFS, a clustered file-system server.
Mär 13 14:19:12 kvm320.durchhalten.intern glusterd[9264]: [2020-03-13 13:19:12.869310] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action] 0-management: Server quorum regained for volume data. Starting local bricks.
Mär 13 14:19:13 kvm320.durchhalten.intern glusterd[9264]: [2020-03-13 13:19:13.115484] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action] 0-management: Server quorum regained for volume engine. Starting local bricks.
everything seems to be connected
[root@kvm380 ~]# gluster peer status
Number of Peers: 3
Hostname: kvm360.durchhalten.intern
Uuid: 2f6cf198-afe5-4e61-b018-e7e4c10793b4
State: Peer in Cluster (Connected)
Hostname: kvm320.durchhalten.intern
Uuid: ce474774-436a-41d3-bfdd-ab153ac77830
State: Peer in Cluster (Connected)
Other names:
192.168.200.231
Hostname: kvm10
Uuid: 33cd77a6-3cda-4e21-bd45-a907044f410b
State: Peer in Cluster (Connected)
Other names:
kvm10
and this is the state of the brick from engine and data
[root@kvm320 ~]# gluster volume status engine
Status of volume: engine
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick kvm10:/gluster_bricks/engine 49153 0 Y 2996
Brick kvm320.durchhalten.intern:/gluster_br
icks/engine 49153 0 Y 3896
Brick kvm360.durchhalten.intern:/gluster_br
icks/engine 49153 0 Y 5066
Self-heal Daemon on localhost N/A N/A Y 9381
Self-heal Daemon on kvm10 N/A N/A Y 2325
Self-heal Daemon on kvm380.durchhalten.inte
rn N/A N/A Y 1070
Self-heal Daemon on kvm360.durchhalten.inte
rn N/A N/A Y 20403
Task Status of Volume engine
------------------------------------------------------------------------------
There are no active volume tasks
[root@kvm320 ~]# gluster volume status data
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick kvm10:/gluster_bricks/data 49152 0 Y 2966
Brick kvm360.durchhalten.intern:/gluster_br
icks/data 49152 0 Y 5058
Brick kvm320.durchhalten.intern:/gluster_br
icks/data 49152 0 Y 3839
Self-heal Daemon on localhost N/A N/A Y 9381
Self-heal Daemon on kvm10 N/A N/A Y 2325
Self-heal Daemon on kvm380.durchhalten.inte
rn N/A N/A Y 1070
Self-heal Daemon on kvm360.durchhalten.inte
rn N/A N/A Y 20403
Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks
and this happens, when I restart glusterd on the host
[2020-03-13 15:11:46.470990] W [glusterd-locks.c:807:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/6.7/xlator/mgmt/glusterd.so(+0x23a0a) [0x7fa6e1cd0a0a] -->/usr/lib64/glusterfs/6.7/xlator/mgmt/glusterd.so(+0x2e320) [0x7fa6e1cdb320] -->/usr/lib64/glusterfs/6.7/xlator/mgmt/glusterd.so(+0xed3b3) [0x7fa6e1d9a3b3] ) 0-management: Lock for vol data not held
[2020-03-13 15:11:46.471039] W [glusterd-locks.c:807:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/6.7/xlator/mgmt/glusterd.so(+0x23a0a) [0x7fa6e1cd0a0a] -->/usr/lib64/glusterfs/6.7/xlator/mgmt/glusterd.so(+0x2e320) [0x7fa6e1cdb320] -->/usr/lib64/glusterfs/6.7/xlator/mgmt/glusterd.so(+0xed3b3) [0x7fa6e1d9a3b3] ) 0-management: Lock for vol engine not held
[2020-03-13 15:11:46.471092] W [glusterd-locks.c:807:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/6.7/xlator/mgmt/glusterd.so(+0x23a0a) [0x7fa6e1cd0a0a] -->/usr/lib64/glusterfs/6.7/xlator/mgmt/glusterd.so(+0x2e320) [0x7fa6e1cdb320] -->/usr/lib64/glusterfs/6.7/xlator/mgmt/glusterd.so(+0xed3b3) [0x7fa6e1d9a3b3] ) 0-management: Lock for vol home not held
[2020-03-13 15:11:46.471105] W [MSGID: 106117] [glusterd-handler.c:6466:__glusterd_peer_rpc_notify] 0-management: Lock not released for home
[2020-03-13 15:11:46.871933] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 40000
[2020-03-13 15:11:46.952276] I [MSGID: 106490] [glusterd-handler.c:2611:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: ce474774-436a-41d3-bfdd-ab153ac77830
[2020-03-13 15:11:57.353876] I [MSGID: 106493] [glusterd-handler.c:3883:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to kvm320.durchhalten.intern (0), ret: 0, op_ret: 0
[2020-03-13 15:11:57.366239] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: nfs already stopped
[2020-03-13 15:11:57.366273] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: nfs service is stopped
[2020-03-13 15:11:57.366295] I [MSGID: 106599] [glusterd-nfs-svc.c:81:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed
[2020-03-13 15:11:57.368510] I [MSGID: 106568] [glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 31828
XXXXXXXXXXXX here the disconnects from gluster --> but the peer is still connected as shown above
[2020-03-13 15:11:57.370534] I [MSGID: 106006] [glusterd-svc-mgmt.c:356:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd.
[2020-03-13 15:11:58.368834] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: glustershd service is stopped
[2020-03-13 15:11:58.368972] I [MSGID: 106567] [glusterd-svc-mgmt.c:220:glusterd_svc_start] 0-management: Starting glustershd service
[2020-03-13 15:11:59.372807] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: quotad already stopped
[2020-03-13 15:11:59.372883] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: quotad service is stopped
[2020-03-13 15:11:59.373068] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already stopped
[2020-03-13 15:11:59.373094] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: bitd service is stopped
[2020-03-13 15:11:59.373269] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: scrub already stopped
[2020-03-13 15:11:59.373292] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: scrub service is stopped
[2020-03-13 15:11:59.373787] I [MSGID: 106492] [glusterd-handler.c:2796:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: ce474774-436a-41d3-bfdd-ab153ac77830
[2020-03-13 15:11:59.375418] I [MSGID: 106502] [glusterd-handler.c:2837:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-03-13 15:11:59.375651] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: ce474774-436a-41d3-bfdd-ab153ac77830
[2020-03-13 15:11:59.432044] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: ce474774-436a-41d3-bfdd-ab153ac77830, host: kvm320.durchhalten.intern, port: 0
[2020-03-13 15:11:59.465527] I [MSGID: 106492] [glusterd-handler.c:2796:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: ce474774-436a-41d3-bfdd-ab153ac77830
[2020-03-13 15:11:59.466839] I [MSGID: 106502] [glusterd-handler.c:2837:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-03-13 15:11:59.566501] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: ce474774-436a-41d3-bfdd-ab153ac77830
does someone have an advice for me?
thx bye stefan
