2:28 p.m.
Hi ,
I am in state where my system does not recover from a major failure. I have
pinpoint the probleme to be that the hosted engine storage domain is not
able to mount
I have a glusterfs containing the storage domain. but when it attempt to
mount glusterfs to /rhev/data-center/mnt/glusterSD/ovhost1:_engine i get
+------------------------------------------------------------------------------+
[2019-07-12 19:19:44.063608] I [rpc-clnt.c:1986:rpc_clnt_reconfig]
0-engine-client-2: changing port to 49153 (from 0)
[2019-07-12 19:19:55.033725] I [fuse-bridge.c:4205:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel
7.22
[2019-07-12 19:19:55.033748] I [fuse-bridge.c:4835:fuse_graph_sync] 0-fuse:
switched to graph 0
[2019-07-12 19:19:55.033895] I [MSGID: 108006]
[afr-common.c:5372:afr_local_init] 0-engine-replicate-0: no subvolumes up
[2019-07-12 19:19:55.033938] E [fuse-bridge.c:4271:fuse_first_lookup]
0-fuse: first lookup on root failed (Transport endpoint is not connected)
[2019-07-12 19:19:55.034041] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk]
0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport
endpoint is not connected)
[2019-07-12 19:19:55.034060] E [fuse-bridge.c:900:fuse_getattr_resume]
0-glusterfs-fuse: 2: GETATTR 1 (00000000-0000-0000-0000-000000000001)
resolution failed
[2019-07-12 19:19:55.034095] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk]
0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport
endpoint is not connected)
[2019-07-12 19:19:55.034102] E [fuse-bridge.c:900:fuse_getattr_resume]
0-glusterfs-fuse: 3: GETATTR 1 (00000000-0000-0000-0000-000000000001)
resolution failed
[2019-07-12 19:19:55.035596] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk]
0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport
endpoint is not connected)
[2019-07-12 19:19:55.035611] E [fuse-bridge.c:900:fuse_getattr_resume]
0-glusterfs-fuse: 4: GETATTR 1 (00000000-0000-0000-0000-000000000001)
resolution failed
[2019-07-12 19:19:55.047957] I [fuse-bridge.c:5093:fuse_thread_proc]
0-fuse: initating unmount of /rhev/data-center/mnt/glusterSD/ovhost1:_engine
The message "I [MSGID: 108006] [afr-common.c:5372:afr_local_init]
0-engine-replicate-0: no subvolumes up" repeated 3 times between
[2019-07-12 19:19:55.033895] and [2019-07-12 19:19:55.035588]
[2019-07-12 19:19:55.048138] W [glusterfsd.c:1375:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7e25) [0x7f51cecb3e25]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x5632143bd4b5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x5632143bd32b] ) 0-:
received signum (15), shutting down
[2019-07-12 19:19:55.048150] I [fuse-bridge.c:5852:fini] 0-fuse: Unmounting
'/rhev/data-center/mnt/glusterSD/ovhost1:_engine'.
[2019-07-12 19:19:55.048155] I [fuse-bridge.c:5857:fini] 0-fuse: Closing
fuse connection to '/rhev/data-center/mnt/glusterSD/ovhost1:_engine'.
[2019-07-12 19:19:56.029923] I [MSGID: 100030] [glusterfsd.c:2511:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.11
(args: /usr/sbin/glusterfs --volfile-server=ovhost1
--volfile-server=ovhost2 --volfile-server=ovhost3 --volfile-id=/engine
/rhev/data-center/mnt/glusterSD/ovhost1:_engine)
[2019-07-12 19:19:56.032209] W [MSGID: 101002]
[options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is
deprecated, preferred is 'transport.address-family', continuing with
correction
[2019-07-12 19:19:56.037510] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2019-07-12 19:19:56.039618] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 2
[2019-07-12 19:19:56.039691] W [MSGID: 101174]
[graph.c:363:_log_if_unknown_option] 0-engine-readdir-ahead: option
'parallel-readdir' is not recognized
[2019-07-12 19:19:56.039739] I [MSGID: 114020] [client.c:2360:notify]
0-engine-client-0: parent translators are ready, attempting connect on
transport
[2019-07-12 19:19:56.043324] I [MSGID: 114020] [client.c:2360:notify]
0-engine-client-1: parent translators are ready, attempting connect on
transport
[2019-07-12 19:19:56.043481] I [rpc-clnt.c:1986:rpc_clnt_reconfig]
0-engine-client-0: changing port to 49153 (from 0)
[2019-07-12 19:19:56.048539] I [MSGID: 114020] [client.c:2360:notify]
0-engine-client-2: parent translators are ready, attempting connect on
transport
[2019-07-12 19:19:56.048952] I [rpc-clnt.c:1986:rpc_clnt_reconfig]
0-engine-client-1: changing port to 49153 (from 0)
Final graph:
without this mount point the ha-agent is not starting.
the volume seem to be okey
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ovhost1:/gluster_bricks/data/data 49152 0 Y
7505
Brick ovhost2:/gluster_bricks/data/data 49152 0 Y
3640
Brick ovhost3:/gluster_bricks/data/data 49152 0 Y
6329
Self-heal Daemon on localhost N/A N/A Y
7712
Self-heal Daemon on ovhost2 N/A N/A Y
4925
Self-heal Daemon on ovhost3 N/A N/A Y
6501
Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: engine
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ovhost1:/gluster_bricks/engine/engine 49153 0 Y
7514
Brick ovhost2:/gluster_bricks/engine/engine 49153 0 Y
3662
Brick ovhost3:/gluster_bricks/engine/engine 49153 0 Y
6339
Self-heal Daemon on localhost N/A N/A Y
7712
Self-heal Daemon on ovhost2 N/A N/A Y
4925
Self-heal Daemon on ovhost3 N/A N/A Y
6501
Task Status of Volume engine
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: iso
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ovhost1:/gluster_bricks/iso/iso 49154 0 Y
7523
Brick ovhost2:/gluster_bricks/iso/iso 49154 0 Y
3715
Brick ovhost3:/gluster_bricks/iso/iso 49154 0 Y
6349
Self-heal Daemon on localhost N/A N/A Y
7712
Self-heal Daemon on ovhost2 N/A N/A Y
4925
Self-heal Daemon on ovhost3 N/A N/A Y
6501
Task Status of Volume iso
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: vmstore
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ovhost1:/gluster_bricks/vmstore/vmsto
re 49155 0 Y
7532
Brick ovhost2:/gluster_bricks/vmstore/vmsto
re 49155 0 Y
3739
Brick ovhost3:/gluster_bricks/vmstore/vmsto
re 49155 0 Y
6359
Self-heal Daemon on localhost N/A N/A Y
7712
Self-heal Daemon on ovhost2 N/A N/A Y
4925
Self-heal Daemon on ovhost3 N/A N/A Y
6501
Task Status of Volume vmstore
------------------------------------------------------------------------------
There are no active volume tasks
I am not sure what to look for at this point
Any help would be really appreciated.
Thanks
Carl