Hi ,
I am in state where my system does not recover from a major failure. I have pinpoint the probleme to be that the hosted engine storage domain is not able to mount
I have a glusterfs containing the storage domain. but when it attempt to mount glusterfs to /rhev/data-center/mnt/glusterSD/ovhost1:_engine i get
+------------------------------------------------------------------------------+
[2019-07-12 19:19:44.063608] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-engine-client-2: changing port to 49153 (from 0)
[2019-07-12 19:19:55.033725] I [fuse-bridge.c:4205:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2019-07-12 19:19:55.033748] I [fuse-bridge.c:4835:fuse_graph_sync] 0-fuse: switched to graph 0
[2019-07-12 19:19:55.033895] I [MSGID: 108006] [afr-common.c:5372:afr_local_init] 0-engine-replicate-0: no subvolumes up
[2019-07-12 19:19:55.033938] E [fuse-bridge.c:4271:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
[2019-07-12 19:19:55.034041] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)
[2019-07-12 19:19:55.034060] E [fuse-bridge.c:900:fuse_getattr_resume] 0-glusterfs-fuse: 2: GETATTR 1 (00000000-0000-0000-0000-000000000001) resolution failed
[2019-07-12 19:19:55.034095] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)
[2019-07-12 19:19:55.034102] E [fuse-bridge.c:900:fuse_getattr_resume] 0-glusterfs-fuse: 3: GETATTR 1 (00000000-0000-0000-0000-000000000001) resolution failed
[2019-07-12 19:19:55.035596] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)
[2019-07-12 19:19:55.035611] E [fuse-bridge.c:900:fuse_getattr_resume] 0-glusterfs-fuse: 4: GETATTR 1 (00000000-0000-0000-0000-000000000001) resolution failed
[2019-07-12 19:19:55.047957] I [fuse-bridge.c:5093:fuse_thread_proc] 0-fuse: initating unmount of /rhev/data-center/mnt/glusterSD/ovhost1:_engine
The message "I [MSGID: 108006] [afr-common.c:5372:afr_local_init] 0-engine-replicate-0: no subvolumes up" repeated 3 times between [2019-07-12 19:19:55.033895] and [2019-07-12 19:19:55.035588]
[2019-07-12 19:19:55.048138] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f51cecb3e25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x5632143bd4b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x5632143bd32b] ) 0-: received signum (15), shutting down
[2019-07-12 19:19:55.048150] I [fuse-bridge.c:5852:fini] 0-fuse: Unmounting '/rhev/data-center/mnt/glusterSD/ovhost1:_engine'.
[2019-07-12 19:19:55.048155] I [fuse-bridge.c:5857:fini] 0-fuse: Closing fuse connection to '/rhev/data-center/mnt/glusterSD/ovhost1:_engine'.
[2019-07-12 19:19:56.029923] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.11 (args: /usr/sbin/glusterfs --volfile-server=ovhost1 --volfile-server=ovhost2 --volfile-server=ovhost3 --volfile-id=/engine /rhev/data-center/mnt/glusterSD/ovhost1:_engine)
[2019-07-12 19:19:56.032209] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2019-07-12 19:19:56.037510] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2019-07-12 19:19:56.039618] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2019-07-12 19:19:56.039691] W [MSGID: 101174] [graph.c:363:_log_if_unknown_option] 0-engine-readdir-ahead: option 'parallel-readdir' is not recognized
[2019-07-12 19:19:56.039739] I [MSGID: 114020] [client.c:2360:notify] 0-engine-client-0: parent translators are ready, attempting connect on transport
[2019-07-12 19:19:56.043324] I [MSGID: 114020] [client.c:2360:notify] 0-engine-client-1: parent translators are ready, attempting connect on transport
[2019-07-12 19:19:56.043481] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-engine-client-0: changing port to 49153 (from 0)
[2019-07-12 19:19:56.048539] I [MSGID: 114020] [client.c:2360:notify] 0-engine-client-2: parent translators are ready, attempting connect on transport
[2019-07-12 19:19:56.048952] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-engine-client-1: changing port to 49153 (from 0)
Final graph:
without this mount point the ha-agent is not starting.
the volume seem to be okey
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ovhost1:/gluster_bricks/data/data 49152 0 Y 7505
Brick ovhost2:/gluster_bricks/data/data 49152 0 Y 3640
Brick ovhost3:/gluster_bricks/data/data 49152 0 Y 6329
Self-heal Daemon on localhost N/A N/A Y 7712
Self-heal Daemon on ovhost2 N/A N/A Y 4925
Self-heal Daemon on ovhost3 N/A N/A Y 6501
Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: engine
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ovhost1:/gluster_bricks/engine/engine 49153 0 Y 7514
Brick ovhost2:/gluster_bricks/engine/engine 49153 0 Y 3662
Brick ovhost3:/gluster_bricks/engine/engine 49153 0 Y 6339
Self-heal Daemon on localhost N/A N/A Y 7712
Self-heal Daemon on ovhost2 N/A N/A Y 4925
Self-heal Daemon on ovhost3 N/A N/A Y 6501
Task Status of Volume engine
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: iso
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ovhost1:/gluster_bricks/iso/iso 49154 0 Y 7523
Brick ovhost2:/gluster_bricks/iso/iso 49154 0 Y 3715
Brick ovhost3:/gluster_bricks/iso/iso 49154 0 Y 6349
Self-heal Daemon on localhost N/A N/A Y 7712
Self-heal Daemon on ovhost2 N/A N/A Y 4925
Self-heal Daemon on ovhost3 N/A N/A Y 6501
Task Status of Volume iso
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: vmstore
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ovhost1:/gluster_bricks/vmstore/vmsto
re 49155 0 Y 7532
Brick ovhost2:/gluster_bricks/vmstore/vmsto
re 49155 0 Y 3739
Brick ovhost3:/gluster_bricks/vmstore/vmsto
re 49155 0 Y 6359
Self-heal Daemon on localhost N/A N/A Y 7712
Self-heal Daemon on ovhost2 N/A N/A Y 4925
Self-heal Daemon on ovhost3 N/A N/A Y 6501
Task Status of Volume vmstore
------------------------------------------------------------------------------
There are no active volume tasks
I am not sure what to look for at this point
Any help would be really appreciated.
Thanks
Carl