3:55 a.m.
Hi everyone - I really would appreciate it if someone could look over this please.
At 12:12:26 both bricks (engine & isos) are added but at 12:29:50 unable to open pid
file.
This was following a reinstallation via the Compute - Hosts window in oVirt Manager.
[2021-02-26 12:12:24.285936] W [MSGID: 106061]
[glusterd-handler.c:3315:glusterd_transport_inet_options_build] 0-glusterd: Failed to get
tcp-user-timeout
[2021-02-26 12:12:24.287513] I [MSGID: 101190]
[event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2021-02-26 12:12:26.040796] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec]
0-management: Received mount request for volume
engine.bdtovirtprod03-strg.domain.com.gluster_bricks-engine-engine
[2021-02-26 12:12:26.042570] I [MSGID: 106142] [glusterd-pmap.c:290:pmap_registry_bind]
0-pmap: adding brick /gluster_bricks/engine/engine on port 49152
[2021-02-26 12:12:26.042737] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec]
0-management: Received mount request for volume
isos.bdtovirtprod03-strg.domain.com.gluster_bricks-isos-isos
[2021-02-26 12:12:26.043978] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec]
0-management: Received mount request for volume shd/engine
[2021-02-26 12:12:26.044199] I [MSGID: 106142] [glusterd-pmap.c:290:pmap_registry_bind]
0-pmap: adding brick /gluster_bricks/isos/isos on port 49153
[2021-02-26 12:12:26.044283] I [MSGID: 106496] [glusterd-handshake.c:935:__server_getspec]
0-management: Received mount request for volume shd/isos
[2021-02-26 12:12:26.045114] I [MSGID: 106163]
[glusterd-handshake.c:1433:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the
op-version 70200
[2021-02-26 12:12:26.046995] I [MSGID: 106163]
[glusterd-handshake.c:1433:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the
op-version 70200
[2021-02-26 12:13:24.990790] I [MSGID: 106487]
[glusterd-handler.c:1339:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list
req
[2021-02-26 12:15:07.088792] I [MSGID: 106487]
[glusterd-handler.c:1339:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list
req
[2021-02-26 12:15:08.254477] I [MSGID: 106505]
[glusterd-replace-brick.c:70:__glusterd_handle_replace_brick] 0-management: Received
replace brick req
[2021-02-26 12:15:08.254549] I [MSGID: 106587]
[glusterd-replace-brick.c:146:__glusterd_handle_replace_brick] 0-management: Received
reset-brick start request.
[2021-02-26 12:18:08.254818] I [glusterd-locks.c:729:gd_mgmt_v3_unlock_timer_cbk]
0-management: unlock timer is cancelled for volume_type isos_vol
[2021-02-26 12:20:07.618217] I [MSGID: 106487]
[glusterd-handler.c:1339:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list
req
[2021-02-26 12:22:24.298000] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(Peer mgmt), op(--(2)), xid = 0x5, unique = 5, sent = 2021-02-26 12:12:24.294207,
timeout = 600 for 10.237.8.30:24007
[2021-02-26 12:22:24.298060] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(Peer mgmt), op(--(2)), xid = 0x5, unique = 4, sent = 2021-02-26 12:12:24.292298,
timeout = 600 for 10.237.8.31:24007
[2021-02-26 12:23:57.348599] W [glusterfsd.c:1596:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x814a) [0x7fac7879814a]
-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xfd) [0x55d2b9ebfc1d]
-->/usr/sbin/glusterd(cleanup_and_exit+0x58) [0x55d2b9ebfa68] ) 0-: received signum
(15), shutting down
[2021-02-26 12:29:44.658217] I [MSGID: 100030] [glusterfsd.c:2867:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 7.9 (args:
/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2021-02-26 12:29:44.659077] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current
running process is 4198
[2021-02-26 12:29:44.670056] I [MSGID: 106478] [glusterd.c:1426:init] 0-management:
Maximum allowed open file descriptors set to 65536
[2021-02-26 12:29:44.670093] I [MSGID: 106479] [glusterd.c:1482:init] 0-management: Using
/var/lib/glusterd as working directory
[2021-02-26 12:29:44.670098] I [MSGID: 106479] [glusterd.c:1488:init] 0-management: Using
/var/run/gluster as pid file working directory
[2021-02-26 12:29:44.674080] I [socket.c:1015:__socket_server_bind] 0-socket.management:
process started listening on port (24007)
[2021-02-26 12:29:44.677700] W [MSGID: 103071] [rdma.c:4472:__gf_rdma_ctx_create]
0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2021-02-26 12:29:44.677717] W [MSGID: 103055] [rdma.c:4782:init] 0-rdma.management:
Failed to initialize IB Device
[2021-02-26 12:29:44.677724] W [rpc-transport.c:366:rpc_transport_load] 0-rpc-transport:
'rdma' initialization failed
[2021-02-26 12:29:44.677792] W [rpcsvc.c:1981:rpcsvc_create_listener] 0-rpc-service:
cannot create listener, initing the transport failed
[2021-02-26 12:29:44.677798] E [MSGID: 106244] [glusterd.c:1781:init] 0-management:
creation of 1 listeners failed, continuing with succeeded transport
[2021-02-26 12:29:44.679532] I [socket.c:958:__socket_server_bind] 0-socket.management:
closing (AF_UNIX) reuse check socket 12
[2021-02-26 12:29:44.680102] I [MSGID: 106059] [glusterd.c:1865:init] 0-management:
max-port override: 60999
[2021-02-26 12:29:45.975798] I [MSGID: 106513]
[glusterd-store.c:2257:glusterd_restore_op_version] 0-glusterd: retrieved op-version:
70200
[2021-02-26 12:29:45.995205] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key:
tier-enabled
[2021-02-26 12:29:45.995735] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2021-02-26 12:29:45.995771] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2021-02-26 12:29:45.995792] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2021-02-26 12:29:46.003176] I [MSGID: 106544] [glusterd.c:152:glusterd_uuid_init]
0-management: retrieved UUID: 769e5b75-d500-4897-8c8e-a1c0afe5bd58
[2021-02-26 12:29:46.086953] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key:
tier-enabled
[2021-02-26 12:29:46.087071] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2021-02-26 12:29:46.087078] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2021-02-26 12:29:46.087084] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2021-02-26 12:29:46.091410] I [MSGID: 106498]
[glusterd-handler.c:3519:glusterd_friend_add_from_peerinfo] 0-management: connect returned
0
[2021-02-26 12:29:46.091635] I [MSGID: 106498]
[glusterd-handler.c:3519:glusterd_friend_add_from_peerinfo] 0-management: connect returned
0
[2021-02-26 12:29:46.091668] W [MSGID: 106061]
[glusterd-handler.c:3315:glusterd_transport_inet_options_build] 0-glusterd: Failed to get
tcp-user-timeout
[2021-02-26 12:29:46.091685] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600
[2021-02-26 12:29:46.093819] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600
[2021-02-26 12:29:46.093814] W [MSGID: 106061]
[glusterd-handler.c:3315:glusterd_transport_inet_options_build] 0-glusterd: Failed to get
tcp-user-timeout
[2021-02-26 12:29:46.097011] I [MSGID: 101190]
[event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2021-02-26 12:29:46.097399] I [MSGID: 106495]
[glusterd-handler.c:2978:__glusterd_handle_getwd] 0-glusterd: Received getwd req
[2021-02-26 12:29:46.179627] I [MSGID: 106495]
[glusterd-handler.c:2978:__glusterd_handle_getwd] 0-glusterd: Received getwd req
[2021-02-26 12:29:50.225765] I [MSGID: 106004]
[glusterd-handler.c:6204:__glusterd_peer_rpc_notify] 0-management: Peer
<bdtovirtprod01-strg> (<67b5345f-dd3c-4781-8ee3-1f68e37a1e7f>), in state
<Peer in Cluster>, has disconnected from glusterd.
[2021-02-26 12:29:50.226371] C [MSGID: 106002]
[glusterd-server-quorum.c:355:glusterd_do_volume_quorum_action] 0-management: Server
quorum lost for volume engine. Stopping local bricks.
[2021-02-26 12:29:50.227042] E [MSGID: 106028]
[glusterd-utils.c:8665:glusterd_brick_signal] 0-glusterd: Unable to open pidfile:
/var/run/gluster/vols/engine/bdtovirtprod03-strg.domain.com-gluster_bricks-engine-engine.pid
[No such file or directory]
[2021-02-26 12:29:50.227103] C [MSGID: 106002]
[glusterd-server-quorum.c:355:glusterd_do_volume_quorum_action] 0-management: Server
quorum lost for volume isos. Stopping local bricks.
[2021-02-26 12:29:50.227322] E [MSGID: 106028]
[glusterd-utils.c:8665:glusterd_brick_signal] 0-glusterd: Unable to open pidfile:
/var/run/gluster/vols/isos/bdtovirtprod03-strg.domain.com-gluster_bricks-isos-isos.pid [No
such file or directory]
[2021-02-26 12:29:57.606348] I [MSGID: 106163]
[glusterd-handshake.c:1433:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the
op-version 70200
[2021-02-26 12:30:02.364287] I [MSGID: 106163]
[glusterd-handshake.c:1433:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the
op-version 70200
[2021-02-26 12:30:03.956673] I [MSGID: 106487]
[glusterd-handler.c:1339:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list
req
[2021-02-26 12:31:00.325661] I [MSGID: 106487]
[glusterd-handler.c:1339:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list
req
[2021-02-26 12:33:19.901681] I [MSGID: 106488]
[glusterd-handler.c:1400:__glusterd_handle_cli_get_volume] 0-management: Received get vol
req
[2021-02-26 12:33:51.556143] I [MSGID: 106487]
[glusterd-handler.c:1339:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list
req
[2021-02-26 12:34:05.896966] I [MSGID: 106499]
[glusterd-handler.c:4264:__glusterd_handle_status_volume] 0-management: Received status
volume req for volume isos
[2021-02-26 12:34:05.897209] W [glusterd-locks.c:579:glusterd_mgmt_v3_lock]
(-->/usr/lib64/glusterfs/7.9/xlator/mgmt/glusterd.so(+0xdfd24) [0x7f588c535d24]
-->/usr/lib64/glusterfs/7.9/xlator/mgmt/glusterd.so(+0xdf82b) [0x7f588c53582b]
-->/usr/lib64/glusterfs/7.9/xlator/mgmt/glusterd.so(+0xe51d2) [0x7f588c53b1d2] )
0-management: Lock for isos held by 769e5b75-d500-4897-8c8e-a1c0afe5bd58
[2021-02-26 12:34:05.897234] E [MSGID: 106118] [glusterd-syncop.c:1883:gd_sync_task_begin]
0-management: Unable to acquire lock for isos
The message "I [MSGID: 106487]
[glusterd-handler.c:1339:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list
req" repeated 2 times between [2021-02-26 12:33:51.556143] and [2021-02-26
12:35:09.074298]
[2021-02-26 12:36:31.857113] I [glusterd-locks.c:729:gd_mgmt_v3_unlock_timer_cbk]
0-management: unlock timer is cancelled for volume_type isos_vol
[2021-02-26 12:39:56.312206] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(Peer mgmt), op(--(2)), xid = 0x5, unique = 4, sent = 2021-02-26 12:29:56.307354,
timeout = 600 for 10.237.8.30:24007
[2021-02-26 12:40:01.498272] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(Peer mgmt), op(--(2)), xid = 0x5, unique = 7, sent = 2021-02-26 12:30:01.493213,
timeout = 600 for 10.237.8.31:24007
[2021-02-26 12:40:09.087003] I [MSGID: 106487]
[glusterd-handler.c:1339:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list
req
[2021-02-26 12:41:01.499083] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(3)), xid = 0x6, unique = 12, sent = 2021-02-26 12:30:58.209581,
timeout = 600 for 10.237.8.31:24007
[2021-02-26 12:41:01.499141] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Staging failed on bdtovirtprod02-strg.domain.com. Please check log file for
details.
[2021-02-26 12:41:06.313130] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(3)), xid = 0x6, unique = 11, sent = 2021-02-26 12:30:58.209559,
timeout = 600 for 10.237.8.30:24007
[2021-02-26 12:41:06.313173] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Staging failed on bdtovirtprod01-strg. Please check log file for details.
[2021-02-26 12:41:06.313379] I [socket.c:3892:socket_submit_outgoing_msg]
0-socket.management: not connected (priv->connected = -1)
[2021-02-26 12:41:06.313399] E [rpcsvc.c:1573:rpcsvc_submit_generic] 0-rpc-service: failed
to submit message (XID: 0x2, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to
rpc-transport (socket.management)
[2021-02-26 12:41:06.313452] E [MSGID: 106430]
[glusterd-utils.c:553:glusterd_submit_reply] 0-glusterd: Reply submission failed
[2021-02-26 12:41:11.499217] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(3)), xid = 0x7, unique = 16, sent = 2021-02-26 12:31:09.951183,
timeout = 600 for 10.237.8.31:24007
[2021-02-26 12:41:16.313265] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(3)), xid = 0x7, unique = 15, sent = 2021-02-26 12:31:09.951163,
timeout = 600 for 10.237.8.30:24007
[2021-02-26 12:41:16.313505] I [socket.c:3892:socket_submit_outgoing_msg]
0-socket.management: not connected (priv->connected = -1)
[2021-02-26 12:41:16.313527] E [rpcsvc.c:1573:rpcsvc_submit_generic] 0-rpc-service: failed
to submit message (XID: 0x2, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to
rpc-transport (socket.management)
[2021-02-26 12:41:11.499265] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Staging failed on bdtovirtprod02-strg.domain.com. Please check log file for
details.
[2021-02-26 12:41:16.313308] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Staging failed on bdtovirtprod01-strg. Please check log file for details.
[2021-02-26 12:41:16.313546] E [MSGID: 106430]
[glusterd-utils.c:553:glusterd_submit_reply] 0-glusterd: Reply submission failed
[2021-02-26 12:42:42.423584] W [glusterfsd.c:1596:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x814a) [0x7f589239e14a]
-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xfd) [0x564c16673c1d]
-->/usr/sbin/glusterd(cleanup_and_exit+0x58) [0x564c16673a68] ) 0-: received signum
(15), shutting down
[2021-02-26 13:04:03.568312] I [MSGID: 100030] [glusterfsd.c:2867:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 7.9 (args:
/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2021-02-26 13:04:03.569859] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current
running process is 34532
[2021-02-26 13:04:03.573369] I [MSGID: 106478] [glusterd.c:1426:init] 0-management:
Maximum allowed open file descriptors set to 65536
[2021-02-26 13:04:03.573489] I [MSGID: 106479] [glusterd.c:1482:init] 0-management: Using
/var/lib/glusterd as working directory
[2021-02-26 13:04:03.573508] I [MSGID: 106479] [glusterd.c:1488:init] 0-management: Using
/var/run/gluster as pid file working directory
[2021-02-26 13:04:03.579026] I [socket.c:1015:__socket_server_bind] 0-socket.management:
process started listening on port (24007)
[2021-02-26 13:04:03.580716] W [MSGID: 103071] [rdma.c:4472:__gf_rdma_ctx_create]
0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2021-02-26 13:04:03.580736] W [MSGID: 103055] [rdma.c:4782:init] 0-rdma.management:
Failed to initialize IB Device
[2021-02-26 13:04:03.580743] W [rpc-transport.c:366:rpc_transport_load] 0-rpc-transport:
'rdma' initialization failed
[2021-02-26 13:04:03.580826] W [rpcsvc.c:1981:rpcsvc_create_listener] 0-rpc-service:
cannot create listener, initing the transport failed
[2021-02-26 13:04:03.580833] E [MSGID: 106244] [glusterd.c:1781:init] 0-management:
creation of 1 listeners failed, continuing with succeeded transport
[2021-02-26 13:04:03.581928] I [socket.c:958:__socket_server_bind] 0-socket.management:
closing (AF_UNIX) reuse check socket 12
[2021-02-26 13:04:03.582266] I [MSGID: 106059] [glusterd.c:1865:init] 0-management:
max-port override: 60999
[2021-02-26 13:04:05.129916] I [MSGID: 106513]
[glusterd-store.c:2257:glusterd_restore_op_version] 0-glusterd: retrieved op-version:
70200
[2021-02-26 13:04:05.130323] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key:
tier-enabled
[2021-02-26 13:04:05.130687] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2021-02-26 13:04:05.130714] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2021-02-26 13:04:05.130734] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2021-02-26 13:04:05.131161] I [MSGID: 106544] [glusterd.c:152:glusterd_uuid_init]
0-management: retrieved UUID: 769e5b75-d500-4897-8c8e-a1c0afe5bd58
[2021-02-26 13:04:05.200090] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key:
tier-enabled
[2021-02-26 13:04:05.200190] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2021-02-26 13:04:05.200197] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2021-02-26 13:04:05.200202] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2021-02-26 13:04:05.201636] I [MSGID: 106498]
[glusterd-handler.c:3519:glusterd_friend_add_from_peerinfo] 0-management: connect returned
0
[2021-02-26 13:04:05.201736] I [MSGID: 106498]
[glusterd-handler.c:3519:glusterd_friend_add_from_peerinfo] 0-management: connect returned
0
[2021-02-26 13:04:05.201793] W [MSGID: 106061]
[glusterd-handler.c:3315:glusterd_transport_inet_options_build] 0-glusterd: Failed to get
tcp-user-timeout
The logs for engine and isos bricks have not been changed since this happened.
Can anyone point me in the right direction.
I have tried creating new bricks and selecting replace bricks but it's as if it is no
longer part of the gluster cluster.
Peer status shows connected and gluster starts - although it appears no longer configured
as part of the original 3 nodes.
Any help ASAP would be greatly appreciated.
Regards
Shimme