Hi,

I know people are busy but do we have any progress on this, I removed the OVS cluster and recreated it with bridge networking.

But I'm still stuck on the "failed to fetch volume file" and am unable to activate the hosts to continue testing...

Anything else I can try to debug? The storage network is ip based without dns.

Regards, Jorick

On 1/25/19 3:26 PM, Sahina Bose wrote:

On Fri, Jan 25, 2019 at 7:36 PM Jorick Astrego <jorick@netbulae.eu> wrote:

The mount failure is probably related to glusterfs. There are glusterfs logs on the host that 

can give more info on this error. 

Oh duh, sorry forgot to check :-&

"failed to fetch volume file"

[2019-01-25 14:02:45.067440] I [glusterfsd-mgmt.c:2424:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: *.*.*.*

Are you able to access this host, and the gluster ports open?

Anything in the glusterd.log of the gluster server?

Adding Sanju to help

From the host:

Telnet *.*.*.14 24007
Trying *.*.*.14...
Connected to *.*.*.14.
Escape character is '^]'.

glusterd.log on *.*.*.14:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination        
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere           
ACCEPT     all  --  anywhere             anywhere           
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:54321
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:54322
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:sunrpc
ACCEPT     udp  --  anywhere             anywhere             udp dpt:sunrpc
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     udp  --  anywhere             anywhere             udp dpt:snmp
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:websm
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:24007
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:webcache
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38465
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38466
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38467
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:nfs
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38469
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:5666
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:39543
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:55863
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:38468
ACCEPT     udp  --  anywhere             anywhere             udp dpt:963
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:965
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ctdb
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:netbios-ssn
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:microsoft-ds
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:24009:24108
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:49152:49251
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:49217:49316
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:zabbix-agent /* Zabbix agent */
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        
REJECT     all  --  anywhere             anywhere             PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        

And in the glusterd.log has nothing else but:

[2019-01-25 11:02:32.617694] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd6
[2019-01-25 11:02:32.623892] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd9
[2019-01-25 11:03:31.847006] I [MSGID: 106488] [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2019-01-25 11:04:33.023685] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume hdd2
[2019-01-25 11:04:33.030549] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume sdd8
[2019-01-25 11:04:33.037024] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd3
[2019-01-25 11:04:33.043442] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd4
[2019-01-25 11:04:33.049494] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd5
[2019-01-25 11:04:33.056049] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd6
[2019-01-25 11:04:33.062081] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd9
[2019-01-25 11:05:47.710098] I [MSGID: 106488] [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2019-01-25 11:06:33.481200] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume hdd2
[2019-01-25 11:06:33.488137] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume sdd8
[2019-01-25 11:06:33.494492] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd3
[2019-01-25 11:06:33.501053] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd4
[2019-01-25 11:06:33.507302] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd5
[2019-01-25 11:06:33.513299] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd6
[2019-01-25 11:06:33.519269] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd9
[2019-01-25 11:08:33.940033] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume hdd2
[2019-01-25 11:08:33.947011] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume sdd8
[2019-01-25 11:08:33.953445] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd3
[2019-01-25 11:08:33.959618] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd4
[2019-01-25 11:08:33.965659] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd5
[2019-01-25 11:08:33.971609] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd6
[2019-01-25 11:08:33.977763] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd9
[2019-01-25 11:10:58.656508] I [MSGID: 106488] [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2019-01-25 11:11:05.746810] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd4
[2019-01-25 11:11:08.153573] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd5
[2019-01-25 11:11:11.108019] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd6
[2019-01-25 11:11:13.456055] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume hdd2
[2019-01-25 11:11:13.584535] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume hdd2
[2019-01-25 11:11:16.046244] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd3
[2019-01-25 11:11:16.172035] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd3
[2019-01-25 11:11:18.370339] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume sdd8
[2019-01-25 11:11:18.504515] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume sdd8
[2019-01-25 11:11:20.710031] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd9
[2019-01-25 11:11:05.886182] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd4
[2019-01-25 11:11:08.284806] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd5
[2019-01-25 11:11:11.244245] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd6
[2019-01-25 11:11:20.835972] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd9
[2019-01-25 11:13:33.238060] I [MSGID: 106488] [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2019-01-25 11:15:36.929536] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume hdd2
[2019-01-25 11:15:36.937647] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume sdd8
[2019-01-25 11:15:36.943936] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd3
[2019-01-25 11:15:36.950206] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd4
[2019-01-25 11:15:36.956460] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd5
[2019-01-25 11:15:36.962459] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd6
[2019-01-25 11:15:36.968594] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd9
[2019-01-25 11:21:14.123760] I [MSGID: 106488] [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2019-01-25 11:22:38.575269] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume hdd2
[2019-01-25 11:22:38.582011] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume sdd8
[2019-01-25 11:22:38.588583] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume ssd3