6:27 a.m.
Hi,
tonight - without any apparent reason - the /var/log/gluster directory filled up disk
space of one node.
I shut down services, cleaned logs, rebooted but services won't start any more.
glusterd log says:
[2015-03-12 09:08:14.919478] I [MSGID: 100030] [glusterfsd.c:2018:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.6.2 (args:
/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2015-03-12 09:08:14.935111] I [glusterd.c:1214:init] 0-management: Maximum allowed open
file descriptors set to 65536
[2015-03-12 09:08:14.935142] I [glusterd.c:1259:init] 0-management: Using
/var/lib/glusterd as working directory
[2015-03-12 09:08:14.953202] W [rdma.c:4221:__gf_rdma_ctx_create] 0-rpc-transport/rdma:
rdma_cm event channel creation failed (No such device)
[2015-03-12 09:08:14.953221] E [rdma.c:4519:init] 0-rdma.management: Failed to initialize
IB Device
[2015-03-12 09:08:14.953229] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport:
'rdma' initialization failed
[2015-03-12 09:08:14.953280] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service:
cannot create listener, initing the transport failed
[2015-03-12 09:08:14.956004] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd:
geo-replication module not installed in the system
[2015-03-12 09:08:14.958341] I [glusterd-store.c:2063:glusterd_restore_op_version]
0-management: Detected new install. Setting op-version to maximum : 30600
[2015-03-12 09:08:15.166709] E [xlator.c:425:xlator_init] 0-management: Initialization of
volume 'management' failed, review your volfile again
[2015-03-12 09:08:15.166729] E [graph.c:322:glusterfs_graph_init] 0-management:
initializing translator failed
[2015-03-12 09:08:15.166737] E [graph.c:525:glusterfs_graph_activate] 0-graph: init
failed
[2015-03-12 09:08:15.166987] W [glusterfsd.c:1194:cleanup_and_exit] (--> 0-: received
signum (0), shutting down
Can you please help?
Thank you
Alessandro
1:57 a.m.
Did you upgrade glusterfs on the node?
Looks like there's some problem with your volume file?
[Adding gluster-users for further help]
On 03/12/2015 03:57 PM, RASTELLI Alessandro wrote:
Hi,
tonight - without any apparent reason - the /var/log/gluster directory filled up disk
space of one node.
I shut down services, cleaned logs, rebooted but services won't start any more.
glusterd log says:
[2015-03-12 09:08:14.919478] I [MSGID: 100030] [glusterfsd.c:2018:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.6.2 (args:
/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2015-03-12 09:08:14.935111] I [glusterd.c:1214:init] 0-management: Maximum allowed open
file descriptors set to 65536
[2015-03-12 09:08:14.935142] I [glusterd.c:1259:init] 0-management: Using
/var/lib/glusterd as working directory
[2015-03-12 09:08:14.953202] W [rdma.c:4221:__gf_rdma_ctx_create] 0-rpc-transport/rdma:
rdma_cm event channel creation failed (No such device)
[2015-03-12 09:08:14.953221] E [rdma.c:4519:init] 0-rdma.management: Failed to initialize
IB Device
[2015-03-12 09:08:14.953229] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport:
'rdma' initialization failed
[2015-03-12 09:08:14.953280] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service:
cannot create listener, initing the transport failed
[2015-03-12 09:08:14.956004] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd:
geo-replication module not installed in the system
[2015-03-12 09:08:14.958341] I [glusterd-store.c:2063:glusterd_restore_op_version]
0-management: Detected new install. Setting op-version to maximum : 30600
[2015-03-12 09:08:15.166709] E [xlator.c:425:xlator_init] 0-management: Initialization of
volume 'management' failed, review your volfile again
[2015-03-12 09:08:15.166729] E [graph.c:322:glusterfs_graph_init] 0-management:
initializing translator failed
[2015-03-12 09:08:15.166737] E [graph.c:525:glusterfs_graph_activate] 0-graph: init
failed
[2015-03-12 09:08:15.166987] W [glusterfsd.c:1194:cleanup_and_exit] (--> 0-: received
signum (0), shutting down
Can you please help?
Thank you
Alessandro
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
2:49 a.m.
New subject: [Gluster-users] Gluster services won't start any more
> [glusterd-store.c:2063:glusterd_restore_op_version]
0-management: Detected
> new install. Setting op-version to maximum : 30600
The above message indicates that /var/lib/glusterd/glusterd.info file, carrying the
identify (UUID) of the node and the operating version of the glusterd binary, was empty.
This _shouldn't_ happen. We need to check for messages in glusterd log around the
time
/var/ filesystem was full to understand why this happened.
> [2015-03-12 09:08:15.166709] E [xlator.c:425:xlator_init]
0-management:
> Initialization of volume 'management' failed, review your volfile again
> [2015-03-12 09:08:15.166729] E [graph.c:322:glusterfs_graph_init]
> 0-management: initializing translator failed
> [2015-03-12 09:08:15.166737] E [graph.c:525:glusterfs_graph_activate]
> 0-graph: init failed
As part of the 'init' process, glusterd resolves identities of daemons
that need to be spawned as part of hosting volumes. The resolution would fail
if the identity of this node changes between a stop and start of glusterd service.
Glusterd wouldn't start until the point this inconsistency is resolved.
> [2015-03-12 09:08:15.166987] W
[glusterfsd.c:1194:cleanup_and_exit] (-->
> 0-: received signum (0), shutting down
>
> Can you please help?
To get out of this situation, we need to reconstruct the configuration files
that are 'out of date' with respect to the cluster. This could be tedious but
possible if other nodes didn't have their /var filesystem getting filled.
Each glusterd maintains its copy of volume and peer configuration under
/var/lib/glusterd.
* /var/lib/glusterd/peers - Holds one file for every peer, excluding 'self'.
This implies that with the help of remaining nodes in the cluster, we can determine
this
node's identity. This means we can reconstruct /var/lib/glusterd/glusterd.info on
this
node.
For other files under /var/lib/glusterd that are empty, we could use the fact that
each node has a copy of the configuration and it can be used to reconstruct.
Hope that helps,
kp
4:30 a.m.
New subject: [Gluster-users] Gluster services won't start any more
Hi,
I've reconstructed the glusterd.info file and the peer files, that solved the issue.
Next time it happens, I'll collect the glusterd log that filled up the filesystem for
further analysis.
Thank you
A.
-----Original Message-----
From: Krishnan Parthasarathi [mailto:kparthas@redhat.com]
Sent: venerdì 13 marzo 2015 07:49
To: RASTELLI Alessandro
Cc: users(a)ovirt.org; gluster-users; BREGA Martino; Sahina Bose
Subject: Re: [Gluster-users] [ovirt-users] Gluster services won't start any more
> [glusterd-store.c:2063:glusterd_restore_op_version]
0-management:
> Detected new install. Setting op-version to maximum : 30600
The above message indicates that /var/lib/glusterd/glusterd.info file, carrying the
identify (UUID) of the node and the operating version of the glusterd binary, was empty.
This _shouldn't_ happen. We need to check for messages in glusterd log around the time
/var/ filesystem was full to understand why this happened.
> [2015-03-12 09:08:15.166709] E [xlator.c:425:xlator_init]
0-management:
> Initialization of volume 'management' failed, review your volfile
> again
> [2015-03-12 09:08:15.166729] E [graph.c:322:glusterfs_graph_init]
> 0-management: initializing translator failed
> [2015-03-12 09:08:15.166737] E
> [graph.c:525:glusterfs_graph_activate]
> 0-graph: init failed
As part of the 'init' process, glusterd resolves identities of daemons that need
to be spawned as part of hosting volumes. The resolution would fail if the identity of
this node changes between a stop and start of glusterd service.
Glusterd wouldn't start until the point this inconsistency is resolved.
> [2015-03-12 09:08:15.166987] W
[glusterfsd.c:1194:cleanup_and_exit]
> (-->
> 0-: received signum (0), shutting down
>
> Can you please help?
To get out of this situation, we need to reconstruct the configuration files that are
'out of date' with respect to the cluster. This could be tedious but possible if
other nodes didn't have their /var filesystem getting filled.
Each glusterd maintains its copy of volume and peer configuration under
/var/lib/glusterd.
* /var/lib/glusterd/peers - Holds one file for every peer, excluding 'self'.
This implies that with the help of remaining nodes in the cluster, we can determine
this
node's identity. This means we can reconstruct /var/lib/glusterd/glusterd.info on
this
node.
For other files under /var/lib/glusterd that are empty, we could use the fact that each
node has a copy of the configuration and it can be used to reconstruct.
Hope that helps,
kp
3576
days inactive
3577
days old
3 comments
3 participants
participants (3)
-
Krishnan Parthasarathi -
RASTELLI Alessandro -
Sahina Bose