[ovirt-users] Persisting glusterfs configs on an oVirt node

[Simon Barrett]

Thanks for the replies.

I cannot get glusterd to start on boot and I lose all gluster config every reboot. 

The following shows what I did on the node to start glusterd, create a volume etc, followed by the state of the node after a reboot. 


[root at ovirt_node]# service glusterd status
glusterd is stopped

[root at ovirt_node]# chkconfig --list glusterd
glusterd        0:off   1:off   2:off   3:off   4:off   5:off   6:off

[root at ovirt_node]# service glusterd start
Starting glusterd:[  OK  ]

gluster> volume create vmstore 10.22.8.46:/data/glusterfs/vmstore
volume create: vmstore: success: please start the volume to access data

gluster> vol start vmstore
volume start: vmstore: success

gluster> vol info
Volume Name: vmstore
Type: Distribute
Volume ID: 5bd01043-1352-4014-88ca-e632e264d088
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 10.22.8.46:/data/glusterfs/vmstore

[root at ovirt_node]# ls -1 /var/lib/glusterd/vols/vmstore/
bricks  
node_state.info  
trusted-vmstore-fuse.vol
cksum  
rbstate         
vmstore.10.22.8.46.data-glusterfs-vmstore.vol
info    
run              
vmstore-fuse.vol

[root at ovirt_node]# grep gluster /etc/rwtab.d/*
/etc/rwtab.d/ovirt:files   /var/lib/glusterd

[root at ovirt_node]# chkconfig glusterd on
[root at ovirt_node]# chkconfig --list glusterd
glusterd        0:off   1:off   2:on    3:on    4:on    5:on    6:off


####################################
 I then reboot the node and see the following:
####################################

[root at ovirt_node]# service glusterd status
glusterd is stopped

[root at ovirt_node]# chkconfig --list glusterd
glusterd        0:off   1:off   2:off   3:off   4:off   5:off   6:off

[root at ovirt_node]# ls -l /var/lib/glusterd/vols/
total 0

No more gluster volume configuration files.

I've taken a look through http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Persisting_changes but I'm unsure what needs to be done to persist this configuration.

To get glusterd to start on boot, do I need to manually persist /etc/rc* files?

I see "files /var/lib/glusterd" mentioned in /etc/rwtab.d/ovirt. Is this a list of the files/dirs that should be persisted automatically? If so, is it recursive and should it include everything in /var/lib/glusterd/vols?


TIA for any help with this.

Simon



-----Original Message-----
From: Ryan Barry [mailto:rbarry at redhat.com] 
Sent: 27 May 2014 14:01
To: Fabian Deutsch; Doron Fediuck; Simon Barrett
Cc: users at ovirt.org
Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node

On 05/26/2014 04:14 AM, Fabian Deutsch wrote:
> Am Sonntag, den 25.05.2014, 08:18 -0400 schrieb Doron Fediuck:
>>
>> ----- Original Message -----
>>> From: "Simon Barrett" <Simon.Barrett at tradingscreen.com>
>>> To: users at ovirt.org
>>> Sent: Friday, May 23, 2014 11:29:39 AM
>>> Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
>>>
>>>
>>>
>>> I am working through the setup of oVirt node for a 3.4.1 deployment.
>>>
>>>
>>>
>>> I setup some glusterfs volumes/bricks on oVirt Node Hypervisor 
>>> release 3.0.4
>>> (1.0.201401291204.el6) and created a storage domain. All was working 
>>> OK until I rebooted the node and found that the glusterfs 
>>> configuration had not been retained.
>>>
>>>
>>>
>>> Is there something I should be doing to persist any glusterfs 
>>> configuration so it survives a node reboot?
>>>
>>>
>>>
>>> Many thanks,
>>>
>>>
>>>
>>> Simon
>>>
>>
>> Hi Simon,
>> it actually sounds like a bug to me, as node are supposed to support 
>> gluster.
>>
>> Ryan / Fabian- thoughts?
>
> Hey,
>
> I vaguely remember that we were seeing a bug like this some time ago.
> We fixed /var/lib/glusterd to be writable (using tmpfs), but it can 
> actually be that we need to persist those contents.
>
> But Simon, can you give details which configuration files are missing 
> and why glusterd is not starting?
Is glusterd starting? I'm getting the impression that it's starting, but that it has no configuration. As far as I know, Gluster keeps most of the configuration on the brick itself, but it finds brick information in /var/lib/glusterd.

The last patch simply opened the firewall, and it's entirely possible that we need to persist this. It may be a good idea to just persist the entire directory from the get-go, unless we want to try to have a thread watching /var/lib/glusterd for relevant files, but then we're stuck trying to keep up with what's happening with gluster itself...

Can we
>
> Thanks
> fabian
>
>> Either way I suggest you take a look in the below link- 
>> http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Per
>> sisting_changes
>>
>> Let s know how it works.
>>
>> Doron
>
>