Persisting glusterfs configs on an oVirt node
--_000_D86C48DF8800164BBE50B87623F7AC954836B051ln2wio001devtra_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I am working through the setup of oVirt node for a 3.4.1 deployment. I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.= 4 (1.0.201401291204.el6) and created a storage domain. All was working OK u= ntil I rebooted the node and found that the glusterfs configuration had not= been retained. Is there something I should be doing to persist any glusterfs configuration= so it survives a node reboot? Many thanks, Simon --_000_D86C48DF8800164BBE50B87623F7AC954836B051ln2wio001devtra_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-fareast-language:EN-US;} a:link, span.MsoHyperlink {mso-style-priority:99; color:#0563C1; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} span.EmailStyle17 {mso-style-type:personal-compose; font-family:"Calibri","sans-serif"; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-family:"Calibri","sans-serif"; mso-fareast-language:EN-US;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3D"EN-GB" link=3D"#0563C1" vlink=3D"#954F72"> <div class=3D"WordSection1"> <p class=3D"MsoNormal">I am working through the setup of oVirt node for a 3= .4.1 deployment.<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">I setup some glusterfs volumes/bricks on oVirt Node = Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domai= n. All was working OK until I rebooted the node and found that the glusterf= s configuration had not been retained.<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Is there something I should be doing to persist any = glusterfs configuration so it survives a node reboot?<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Many thanks,<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Simon<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> </body> </html> --_000_D86C48DF8800164BBE50B87623F7AC954836B051ln2wio001devtra_--
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: users@ovirt.org Sent: Friday, May 23, 2014 11:29:39 AM Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
I am working through the setup of oVirt node for a 3.4.1 deployment.
I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domain. All was working OK until I rebooted the node and found that the glusterfs configuration had not been retained.
Is there something I should be doing to persist any glusterfs configuration so it survives a node reboot?
Many thanks,
Simon
Hi Simon, it actually sounds like a bug to me, as node are supposed to support gluster. Ryan / Fabian- thoughts? Either way I suggest you take a look in the below link- http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Persisting... Let s know how it works. Doron
Am Sonntag, den 25.05.2014, 08:18 -0400 schrieb Doron Fediuck:
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: users@ovirt.org Sent: Friday, May 23, 2014 11:29:39 AM Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
I am working through the setup of oVirt node for a 3.4.1 deployment.
I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domain. All was working OK until I rebooted the node and found that the glusterfs configuration had not been retained.
Is there something I should be doing to persist any glusterfs configuration so it survives a node reboot?
Many thanks,
Simon
Hi Simon, it actually sounds like a bug to me, as node are supposed to support gluster.
Ryan / Fabian- thoughts?
Hey, I vaguely remember that we were seeing a bug like this some time ago. We fixed /var/lib/glusterd to be writable (using tmpfs), but it can actually be that we need to persist those contents. But Simon, can you give details which configuration files are missing and why glusterd is not starting? Thanks fabian
Either way I suggest you take a look in the below link- http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Persisting...
Let s know how it works.
Doron
Am Sonntag, den 25.05.2014, 08:18 -0400 schrieb Doron Fediuck:
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: users@ovirt.org Sent: Friday, May 23, 2014 11:29:39 AM Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
I am working through the setup of oVirt node for a 3.4.1 deployment.
I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domain. All was working OK until I rebooted the node and found that the glusterfs configuration had not been retained.
Is there something I should be doing to persist any glusterfs configuration so it survives a node reboot?
Many thanks,
Simon
Hi Simon, it actually sounds like a bug to me, as node are supposed to support gluster.
Ryan / Fabian- thoughts?
Hey,
I vaguely remember that we were seeing a bug like this some time ago. We fixed /var/lib/glusterd to be writable (using tmpfs), but it can actually be that we need to persist those contents.
But Simon, can you give details which configuration files are missing and why glusterd is not starting? Is glusterd starting? I'm getting the impression that it's starting, but
On 05/26/2014 04:14 AM, Fabian Deutsch wrote: that it has no configuration. As far as I know, Gluster keeps most of the configuration on the brick itself, but it finds brick information in /var/lib/glusterd. The last patch simply opened the firewall, and it's entirely possible that we need to persist this. It may be a good idea to just persist the entire directory from the get-go, unless we want to try to have a thread watching /var/lib/glusterd for relevant files, but then we're stuck trying to keep up with what's happening with gluster itself... Can we
Thanks fabian
Either way I suggest you take a look in the below link- http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Persisting...
Let s know how it works.
Doron
Thanks for the replies. I cannot get glusterd to start on boot and I lose all gluster config every reboot. The following shows what I did on the node to start glusterd, create a volume etc, followed by the state of the node after a reboot. [root@ovirt_node]# service glusterd status glusterd is stopped [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off [root@ovirt_node]# service glusterd start Starting glusterd:[ OK ] gluster> volume create vmstore 10.22.8.46:/data/glusterfs/vmstore volume create: vmstore: success: please start the volume to access data gluster> vol start vmstore volume start: vmstore: success gluster> vol info Volume Name: vmstore Type: Distribute Volume ID: 5bd01043-1352-4014-88ca-e632e264d088 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.22.8.46:/data/glusterfs/vmstore [root@ovirt_node]# ls -1 /var/lib/glusterd/vols/vmstore/ bricks node_state.info trusted-vmstore-fuse.vol cksum rbstate vmstore.10.22.8.46.data-glusterfs-vmstore.vol info run vmstore-fuse.vol [root@ovirt_node]# grep gluster /etc/rwtab.d/* /etc/rwtab.d/ovirt:files /var/lib/glusterd [root@ovirt_node]# chkconfig glusterd on [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:on 3:on 4:on 5:on 6:off #################################### I then reboot the node and see the following: #################################### [root@ovirt_node]# service glusterd status glusterd is stopped [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off [root@ovirt_node]# ls -l /var/lib/glusterd/vols/ total 0 No more gluster volume configuration files. I've taken a look through http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Persisting... but I'm unsure what needs to be done to persist this configuration. To get glusterd to start on boot, do I need to manually persist /etc/rc* files? I see "files /var/lib/glusterd" mentioned in /etc/rwtab.d/ovirt. Is this a list of the files/dirs that should be persisted automatically? If so, is it recursive and should it include everything in /var/lib/glusterd/vols? TIA for any help with this. Simon -----Original Message----- From: Ryan Barry [mailto:rbarry@redhat.com] Sent: 27 May 2014 14:01 To: Fabian Deutsch; Doron Fediuck; Simon Barrett Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node On 05/26/2014 04:14 AM, Fabian Deutsch wrote:
Am Sonntag, den 25.05.2014, 08:18 -0400 schrieb Doron Fediuck:
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: users@ovirt.org Sent: Friday, May 23, 2014 11:29:39 AM Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
I am working through the setup of oVirt node for a 3.4.1 deployment.
I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domain. All was working OK until I rebooted the node and found that the glusterfs configuration had not been retained.
Is there something I should be doing to persist any glusterfs configuration so it survives a node reboot?
Many thanks,
Simon
Hi Simon, it actually sounds like a bug to me, as node are supposed to support gluster.
Ryan / Fabian- thoughts?
Hey,
I vaguely remember that we were seeing a bug like this some time ago. We fixed /var/lib/glusterd to be writable (using tmpfs), but it can actually be that we need to persist those contents.
But Simon, can you give details which configuration files are missing and why glusterd is not starting? Is glusterd starting? I'm getting the impression that it's starting, but that it has no configuration. As far as I know, Gluster keeps most of the configuration on the brick itself, but it finds brick information in /var/lib/glusterd.
The last patch simply opened the firewall, and it's entirely possible that we need to persist this. It may be a good idea to just persist the entire directory from the get-go, unless we want to try to have a thread watching /var/lib/glusterd for relevant files, but then we're stuck trying to keep up with what's happening with gluster itself... Can we
Thanks fabian
Either way I suggest you take a look in the below link- http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Per sisting_changes
Let s know how it works.
Doron
I did a "persist /var/lib/glusterd" and things are looking better. The gluster config is now still in place after a reboot. As a workaround to getting glusterd running on boot, I added "service glusterd start" to /etc/rc.local and ran persist /etc/rc.local. It appears to be working but feels like a bit of a hack. Does anyone have any other suggestions as to the correct way to do this? Thanks, Simon -----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Simon Barrett Sent: 28 May 2014 14:12 To: Ryan Barry; Fabian Deutsch; Doron Fediuck Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node Thanks for the replies. I cannot get glusterd to start on boot and I lose all gluster config every reboot. The following shows what I did on the node to start glusterd, create a volume etc, followed by the state of the node after a reboot. [root@ovirt_node]# service glusterd status glusterd is stopped [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off [root@ovirt_node]# service glusterd start Starting glusterd:[ OK ] gluster> volume create vmstore 10.22.8.46:/data/glusterfs/vmstore volume create: vmstore: success: please start the volume to access data gluster> vol start vmstore volume start: vmstore: success gluster> vol info Volume Name: vmstore Type: Distribute Volume ID: 5bd01043-1352-4014-88ca-e632e264d088 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.22.8.46:/data/glusterfs/vmstore [root@ovirt_node]# ls -1 /var/lib/glusterd/vols/vmstore/ bricks node_state.info trusted-vmstore-fuse.vol cksum rbstate vmstore.10.22.8.46.data-glusterfs-vmstore.vol info run vmstore-fuse.vol [root@ovirt_node]# grep gluster /etc/rwtab.d/* /etc/rwtab.d/ovirt:files /var/lib/glusterd [root@ovirt_node]# chkconfig glusterd on [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:on 3:on 4:on 5:on 6:off #################################### I then reboot the node and see the following: #################################### [root@ovirt_node]# service glusterd status glusterd is stopped [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off [root@ovirt_node]# ls -l /var/lib/glusterd/vols/ total 0 No more gluster volume configuration files. I've taken a look through http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Persisting... but I'm unsure what needs to be done to persist this configuration. To get glusterd to start on boot, do I need to manually persist /etc/rc* files? I see "files /var/lib/glusterd" mentioned in /etc/rwtab.d/ovirt. Is this a list of the files/dirs that should be persisted automatically? If so, is it recursive and should it include everything in /var/lib/glusterd/vols? TIA for any help with this. Simon -----Original Message----- From: Ryan Barry [mailto:rbarry@redhat.com] Sent: 27 May 2014 14:01 To: Fabian Deutsch; Doron Fediuck; Simon Barrett Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node On 05/26/2014 04:14 AM, Fabian Deutsch wrote:
Am Sonntag, den 25.05.2014, 08:18 -0400 schrieb Doron Fediuck:
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: users@ovirt.org Sent: Friday, May 23, 2014 11:29:39 AM Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
I am working through the setup of oVirt node for a 3.4.1 deployment.
I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domain. All was working OK until I rebooted the node and found that the glusterfs configuration had not been retained.
Is there something I should be doing to persist any glusterfs configuration so it survives a node reboot?
Many thanks,
Simon
Hi Simon, it actually sounds like a bug to me, as node are supposed to support gluster.
Ryan / Fabian- thoughts?
Hey,
I vaguely remember that we were seeing a bug like this some time ago. We fixed /var/lib/glusterd to be writable (using tmpfs), but it can actually be that we need to persist those contents.
But Simon, can you give details which configuration files are missing and why glusterd is not starting? Is glusterd starting? I'm getting the impression that it's starting, but that it has no configuration. As far as I know, Gluster keeps most of the configuration on the brick itself, but it finds brick information in /var/lib/glusterd.
The last patch simply opened the firewall, and it's entirely possible that we need to persist this. It may be a good idea to just persist the entire directory from the get-go, unless we want to try to have a thread watching /var/lib/glusterd for relevant files, but then we're stuck trying to keep up with what's happening with gluster itself... Can we
Thanks fabian
Either way I suggest you take a look in the below link- http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Per sisting_changes
Let s know how it works.
Doron
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Am Mittwoch, den 28.05.2014, 14:14 +0000 schrieb Simon Barrett:
I did a "persist /var/lib/glusterd" and things are looking better. The gluster config is now still in place after a reboot.
As a workaround to getting glusterd running on boot, I added "service glusterd start" to /etc/rc.local and ran persist /etc/rc.local. It appears to be working but feels like a bit of a hack.
Does anyone have any other suggestions as to the correct way to do this?
Hey Simon, I was also investigating both the steps you did. And was also about to recommend them :) They are more a workaround. We basically need some patches to change the defaults on Node, to let gluster work out of the box. This would include persisting the correct paths and enabling glusterd if enabled. - fabian
Thanks,
Simon
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Simon Barrett Sent: 28 May 2014 14:12 To: Ryan Barry; Fabian Deutsch; Doron Fediuck Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Thanks for the replies.
I cannot get glusterd to start on boot and I lose all gluster config every reboot.
The following shows what I did on the node to start glusterd, create a volume etc, followed by the state of the node after a reboot.
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# service glusterd start Starting glusterd:[ OK ]
gluster> volume create vmstore 10.22.8.46:/data/glusterfs/vmstore volume create: vmstore: success: please start the volume to access data
gluster> vol start vmstore volume start: vmstore: success
gluster> vol info Volume Name: vmstore Type: Distribute Volume ID: 5bd01043-1352-4014-88ca-e632e264d088 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.22.8.46:/data/glusterfs/vmstore
[root@ovirt_node]# ls -1 /var/lib/glusterd/vols/vmstore/ bricks node_state.info trusted-vmstore-fuse.vol cksum rbstate vmstore.10.22.8.46.data-glusterfs-vmstore.vol info run vmstore-fuse.vol
[root@ovirt_node]# grep gluster /etc/rwtab.d/* /etc/rwtab.d/ovirt:files /var/lib/glusterd
[root@ovirt_node]# chkconfig glusterd on [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
#################################### I then reboot the node and see the following: ####################################
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# ls -l /var/lib/glusterd/vols/ total 0
No more gluster volume configuration files.
I've taken a look through http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Persisting... but I'm unsure what needs to be done to persist this configuration.
To get glusterd to start on boot, do I need to manually persist /etc/rc* files?
I see "files /var/lib/glusterd" mentioned in /etc/rwtab.d/ovirt. Is this a list of the files/dirs that should be persisted automatically? If so, is it recursive and should it include everything in /var/lib/glusterd/vols?
TIA for any help with this.
Simon
-----Original Message----- From: Ryan Barry [mailto:rbarry@redhat.com] Sent: 27 May 2014 14:01 To: Fabian Deutsch; Doron Fediuck; Simon Barrett Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
On 05/26/2014 04:14 AM, Fabian Deutsch wrote:
Am Sonntag, den 25.05.2014, 08:18 -0400 schrieb Doron Fediuck:
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: users@ovirt.org Sent: Friday, May 23, 2014 11:29:39 AM Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
I am working through the setup of oVirt node for a 3.4.1 deployment.
I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domain. All was working OK until I rebooted the node and found that the glusterfs configuration had not been retained.
Is there something I should be doing to persist any glusterfs configuration so it survives a node reboot?
Many thanks,
Simon
Hi Simon, it actually sounds like a bug to me, as node are supposed to support gluster.
Ryan / Fabian- thoughts?
Hey,
I vaguely remember that we were seeing a bug like this some time ago. We fixed /var/lib/glusterd to be writable (using tmpfs), but it can actually be that we need to persist those contents.
But Simon, can you give details which configuration files are missing and why glusterd is not starting? Is glusterd starting? I'm getting the impression that it's starting, but that it has no configuration. As far as I know, Gluster keeps most of the configuration on the brick itself, but it finds brick information in /var/lib/glusterd.
The last patch simply opened the firewall, and it's entirely possible that we need to persist this. It may be a good idea to just persist the entire directory from the get-go, unless we want to try to have a thread watching /var/lib/glusterd for relevant files, but then we're stuck trying to keep up with what's happening with gluster itself...
Can we
Thanks fabian
Either way I suggest you take a look in the below link- http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Per sisting_changes
Let s know how it works.
Doron
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I just wasn't sure if I was missing something in the configuration to enable this. I'll stick with the workarounds I have for now and see how it goes. Thanks again. Simon -----Original Message----- From: Fabian Deutsch [mailto:fdeutsch@redhat.com] Sent: 28 May 2014 15:20 To: Simon Barrett Cc: Ryan Barry; Doron Fediuck; users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node Am Mittwoch, den 28.05.2014, 14:14 +0000 schrieb Simon Barrett:
I did a "persist /var/lib/glusterd" and things are looking better. The gluster config is now still in place after a reboot.
As a workaround to getting glusterd running on boot, I added "service glusterd start" to /etc/rc.local and ran persist /etc/rc.local. It appears to be working but feels like a bit of a hack.
Does anyone have any other suggestions as to the correct way to do this?
Hey Simon, I was also investigating both the steps you did. And was also about to recommend them :) They are more a workaround. We basically need some patches to change the defaults on Node, to let gluster work out of the box. This would include persisting the correct paths and enabling glusterd if enabled. - fabian
Thanks,
Simon
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Simon Barrett Sent: 28 May 2014 14:12 To: Ryan Barry; Fabian Deutsch; Doron Fediuck Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Thanks for the replies.
I cannot get glusterd to start on boot and I lose all gluster config every reboot.
The following shows what I did on the node to start glusterd, create a volume etc, followed by the state of the node after a reboot.
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# service glusterd start Starting glusterd:[ OK ]
gluster> volume create vmstore 10.22.8.46:/data/glusterfs/vmstore volume create: vmstore: success: please start the volume to access data
gluster> vol start vmstore volume start: vmstore: success
gluster> vol info Volume Name: vmstore Type: Distribute Volume ID: 5bd01043-1352-4014-88ca-e632e264d088 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.22.8.46:/data/glusterfs/vmstore
[root@ovirt_node]# ls -1 /var/lib/glusterd/vols/vmstore/ bricks node_state.info trusted-vmstore-fuse.vol cksum rbstate vmstore.10.22.8.46.data-glusterfs-vmstore.vol info run vmstore-fuse.vol
[root@ovirt_node]# grep gluster /etc/rwtab.d/* /etc/rwtab.d/ovirt:files /var/lib/glusterd
[root@ovirt_node]# chkconfig glusterd on [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
#################################### I then reboot the node and see the following: ####################################
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# ls -l /var/lib/glusterd/vols/ total 0
No more gluster volume configuration files.
I've taken a look through http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Persisting... but I'm unsure what needs to be done to persist this configuration.
To get glusterd to start on boot, do I need to manually persist /etc/rc* files?
I see "files /var/lib/glusterd" mentioned in /etc/rwtab.d/ovirt. Is this a list of the files/dirs that should be persisted automatically? If so, is it recursive and should it include everything in /var/lib/glusterd/vols?
TIA for any help with this.
Simon
-----Original Message----- From: Ryan Barry [mailto:rbarry@redhat.com] Sent: 27 May 2014 14:01 To: Fabian Deutsch; Doron Fediuck; Simon Barrett Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
On 05/26/2014 04:14 AM, Fabian Deutsch wrote:
Am Sonntag, den 25.05.2014, 08:18 -0400 schrieb Doron Fediuck:
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: users@ovirt.org Sent: Friday, May 23, 2014 11:29:39 AM Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
I am working through the setup of oVirt node for a 3.4.1 deployment.
I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domain. All was working OK until I rebooted the node and found that the glusterfs configuration had not been retained.
Is there something I should be doing to persist any glusterfs configuration so it survives a node reboot?
Many thanks,
Simon
Hi Simon, it actually sounds like a bug to me, as node are supposed to support gluster.
Ryan / Fabian- thoughts?
Hey,
I vaguely remember that we were seeing a bug like this some time ago. We fixed /var/lib/glusterd to be writable (using tmpfs), but it can actually be that we need to persist those contents.
But Simon, can you give details which configuration files are missing and why glusterd is not starting? Is glusterd starting? I'm getting the impression that it's starting, but that it has no configuration. As far as I know, Gluster keeps most of the configuration on the brick itself, but it finds brick information in /var/lib/glusterd.
The last patch simply opened the firewall, and it's entirely possible that we need to persist this. It may be a good idea to just persist the entire directory from the get-go, unless we want to try to have a thread watching /var/lib/glusterd for relevant files, but then we're stuck trying to keep up with what's happening with gluster itself...
Can we
Thanks fabian
Either way I suggest you take a look in the below link- http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_P er sisting_changes
Let s know how it works.
Doron
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Am Mittwoch, den 28.05.2014, 14:22 +0000 schrieb Simon Barrett:
I just wasn't sure if I was missing something in the configuration to enable this.
I'll stick with the workarounds I have for now and see how it goes.
Hey Simon, it would be great if you could let us know about more issues, which will help us to improve the situation even more. Greetings fabian
Thanks again.
Simon
-----Original Message----- From: Fabian Deutsch [mailto:fdeutsch@redhat.com] Sent: 28 May 2014 15:20 To: Simon Barrett Cc: Ryan Barry; Doron Fediuck; users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Am Mittwoch, den 28.05.2014, 14:14 +0000 schrieb Simon Barrett:
I did a "persist /var/lib/glusterd" and things are looking better. The gluster config is now still in place after a reboot.
As a workaround to getting glusterd running on boot, I added "service glusterd start" to /etc/rc.local and ran persist /etc/rc.local. It appears to be working but feels like a bit of a hack.
Does anyone have any other suggestions as to the correct way to do this?
Hey Simon,
I was also investigating both the steps you did. And was also about to recommend them :) They are more a workaround.
We basically need some patches to change the defaults on Node, to let gluster work out of the box.
This would include persisting the correct paths and enabling glusterd if enabled.
- fabian
Thanks,
Simon
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Simon Barrett Sent: 28 May 2014 14:12 To: Ryan Barry; Fabian Deutsch; Doron Fediuck Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Thanks for the replies.
I cannot get glusterd to start on boot and I lose all gluster config every reboot.
The following shows what I did on the node to start glusterd, create a volume etc, followed by the state of the node after a reboot.
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# service glusterd start Starting glusterd:[ OK ]
gluster> volume create vmstore 10.22.8.46:/data/glusterfs/vmstore volume create: vmstore: success: please start the volume to access data
gluster> vol start vmstore volume start: vmstore: success
gluster> vol info Volume Name: vmstore Type: Distribute Volume ID: 5bd01043-1352-4014-88ca-e632e264d088 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.22.8.46:/data/glusterfs/vmstore
[root@ovirt_node]# ls -1 /var/lib/glusterd/vols/vmstore/ bricks node_state.info trusted-vmstore-fuse.vol cksum rbstate vmstore.10.22.8.46.data-glusterfs-vmstore.vol info run vmstore-fuse.vol
[root@ovirt_node]# grep gluster /etc/rwtab.d/* /etc/rwtab.d/ovirt:files /var/lib/glusterd
[root@ovirt_node]# chkconfig glusterd on [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
#################################### I then reboot the node and see the following: ####################################
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# ls -l /var/lib/glusterd/vols/ total 0
No more gluster volume configuration files.
I've taken a look through http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Persisting... but I'm unsure what needs to be done to persist this configuration.
To get glusterd to start on boot, do I need to manually persist /etc/rc* files?
I see "files /var/lib/glusterd" mentioned in /etc/rwtab.d/ovirt. Is this a list of the files/dirs that should be persisted automatically? If so, is it recursive and should it include everything in /var/lib/glusterd/vols?
TIA for any help with this.
Simon
-----Original Message----- From: Ryan Barry [mailto:rbarry@redhat.com] Sent: 27 May 2014 14:01 To: Fabian Deutsch; Doron Fediuck; Simon Barrett Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
On 05/26/2014 04:14 AM, Fabian Deutsch wrote:
Am Sonntag, den 25.05.2014, 08:18 -0400 schrieb Doron Fediuck:
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: users@ovirt.org Sent: Friday, May 23, 2014 11:29:39 AM Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
I am working through the setup of oVirt node for a 3.4.1 deployment.
I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domain. All was working OK until I rebooted the node and found that the glusterfs configuration had not been retained.
Is there something I should be doing to persist any glusterfs configuration so it survives a node reboot?
Many thanks,
Simon
Hi Simon, it actually sounds like a bug to me, as node are supposed to support gluster.
Ryan / Fabian- thoughts?
Hey,
I vaguely remember that we were seeing a bug like this some time ago. We fixed /var/lib/glusterd to be writable (using tmpfs), but it can actually be that we need to persist those contents.
But Simon, can you give details which configuration files are missing and why glusterd is not starting? Is glusterd starting? I'm getting the impression that it's starting, but that it has no configuration. As far as I know, Gluster keeps most of the configuration on the brick itself, but it finds brick information in /var/lib/glusterd.
The last patch simply opened the firewall, and it's entirely possible that we need to persist this. It may be a good idea to just persist the entire directory from the get-go, unless we want to try to have a thread watching /var/lib/glusterd for relevant files, but then we're stuck trying to keep up with what's happening with gluster itself...
Can we
Thanks fabian
Either way I suggest you take a look in the below link- http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_P er sisting_changes
Let s know how it works.
Doron
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Am Mittwoch, den 28.05.2014, 14:22 +0000 schrieb Simon Barrett:
I just wasn't sure if I was missing something in the configuration to enable this.
I'll stick with the workarounds I have for now and see how it goes.
Thanks again.
You are welcome! :)
Simon
-----Original Message----- From: Fabian Deutsch [mailto:fdeutsch@redhat.com] Sent: 28 May 2014 15:20 To: Simon Barrett Cc: Ryan Barry; Doron Fediuck; users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Am Mittwoch, den 28.05.2014, 14:14 +0000 schrieb Simon Barrett:
I did a "persist /var/lib/glusterd" and things are looking better. The gluster config is now still in place after a reboot.
As a workaround to getting glusterd running on boot, I added "service glusterd start" to /etc/rc.local and ran persist /etc/rc.local. It appears to be working but feels like a bit of a hack.
Does anyone have any other suggestions as to the correct way to do this?
Hey Simon,
I was also investigating both the steps you did. And was also about to recommend them :) They are more a workaround.
We basically need some patches to change the defaults on Node, to let gluster work out of the box.
This would include persisting the correct paths and enabling glusterd if enabled.
- fabian
Thanks,
Simon
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Simon Barrett Sent: 28 May 2014 14:12 To: Ryan Barry; Fabian Deutsch; Doron Fediuck Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Thanks for the replies.
I cannot get glusterd to start on boot and I lose all gluster config every reboot.
The following shows what I did on the node to start glusterd, create a volume etc, followed by the state of the node after a reboot.
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# service glusterd start Starting glusterd:[ OK ]
gluster> volume create vmstore 10.22.8.46:/data/glusterfs/vmstore volume create: vmstore: success: please start the volume to access data
gluster> vol start vmstore volume start: vmstore: success
gluster> vol info Volume Name: vmstore Type: Distribute Volume ID: 5bd01043-1352-4014-88ca-e632e264d088 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.22.8.46:/data/glusterfs/vmstore
[root@ovirt_node]# ls -1 /var/lib/glusterd/vols/vmstore/ bricks node_state.info trusted-vmstore-fuse.vol cksum rbstate vmstore.10.22.8.46.data-glusterfs-vmstore.vol info run vmstore-fuse.vol
[root@ovirt_node]# grep gluster /etc/rwtab.d/* /etc/rwtab.d/ovirt:files /var/lib/glusterd
[root@ovirt_node]# chkconfig glusterd on [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
#################################### I then reboot the node and see the following: ####################################
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# ls -l /var/lib/glusterd/vols/ total 0
No more gluster volume configuration files.
I've taken a look through http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Persisting... but I'm unsure what needs to be done to persist this configuration.
To get glusterd to start on boot, do I need to manually persist /etc/rc* files?
I see "files /var/lib/glusterd" mentioned in /etc/rwtab.d/ovirt. Is this a list of the files/dirs that should be persisted automatically? If so, is it recursive and should it include everything in /var/lib/glusterd/vols?
TIA for any help with this.
Simon
-----Original Message----- From: Ryan Barry [mailto:rbarry@redhat.com] Sent: 27 May 2014 14:01 To: Fabian Deutsch; Doron Fediuck; Simon Barrett Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
On 05/26/2014 04:14 AM, Fabian Deutsch wrote:
Am Sonntag, den 25.05.2014, 08:18 -0400 schrieb Doron Fediuck:
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: users@ovirt.org Sent: Friday, May 23, 2014 11:29:39 AM Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
I am working through the setup of oVirt node for a 3.4.1 deployment.
I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domain. All was working OK until I rebooted the node and found that the glusterfs configuration had not been retained.
Is there something I should be doing to persist any glusterfs configuration so it survives a node reboot?
Many thanks,
Simon
Hi Simon, it actually sounds like a bug to me, as node are supposed to support gluster.
Ryan / Fabian- thoughts?
Hey,
I vaguely remember that we were seeing a bug like this some time ago. We fixed /var/lib/glusterd to be writable (using tmpfs), but it can actually be that we need to persist those contents.
But Simon, can you give details which configuration files are missing and why glusterd is not starting? Is glusterd starting? I'm getting the impression that it's starting, but that it has no configuration. As far as I know, Gluster keeps most of the configuration on the brick itself, but it finds brick information in /var/lib/glusterd.
The last patch simply opened the firewall, and it's entirely possible that we need to persist this. It may be a good idea to just persist the entire directory from the get-go, unless we want to try to have a thread watching /var/lib/glusterd for relevant files, but then we're stuck trying to keep up with what's happening with gluster itself...
Can we
Thanks fabian
Either way I suggest you take a look in the below link- http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_P er sisting_changes
Let s know how it works.
Doron
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 05/28/2014 10:23 AM, Fabian Deutsch wrote:
Am Mittwoch, den 28.05.2014, 14:22 +0000 schrieb Simon Barrett:
I just wasn't sure if I was missing something in the configuration to enable this.
I'll stick with the workarounds I have for now and see how it goes.
Thanks again.
You are welcome! :)
Simon
-----Original Message----- From: Fabian Deutsch [mailto:fdeutsch@redhat.com] Sent: 28 May 2014 15:20 To: Simon Barrett Cc: Ryan Barry; Doron Fediuck; users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Am Mittwoch, den 28.05.2014, 14:14 +0000 schrieb Simon Barrett:
I did a "persist /var/lib/glusterd" and things are looking better. The gluster config is now still in place after a reboot.
As a workaround to getting glusterd running on boot, I added "service glusterd start" to /etc/rc.local and ran persist /etc/rc.local. It appears to be working but feels like a bit of a hack.
Does anyone have any other suggestions as to the correct way to do this?
Hey Simon,
I was also investigating both the steps you did. And was also about to recommend them :) They are more a workaround.
We basically need some patches to change the defaults on Node, to let gluster work out of the box.
This would include persisting the correct paths and enabling glusterd if enabled.
- fabian
Thanks,
Simon
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Simon Barrett Sent: 28 May 2014 14:12 To: Ryan Barry; Fabian Deutsch; Doron Fediuck Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Thanks for the replies.
I cannot get glusterd to start on boot and I lose all gluster config every reboot.
The following shows what I did on the node to start glusterd, create a volume etc, followed by the state of the node after a reboot.
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# service glusterd start Starting glusterd:[ OK ]
gluster> volume create vmstore 10.22.8.46:/data/glusterfs/vmstore volume create: vmstore: success: please start the volume to access data
gluster> vol start vmstore volume start: vmstore: success
gluster> vol info Volume Name: vmstore Type: Distribute Volume ID: 5bd01043-1352-4014-88ca-e632e264d088 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.22.8.46:/data/glusterfs/vmstore
[root@ovirt_node]# ls -1 /var/lib/glusterd/vols/vmstore/ bricks node_state.info trusted-vmstore-fuse.vol cksum rbstate vmstore.10.22.8.46.data-glusterfs-vmstore.vol info run vmstore-fuse.vol
[root@ovirt_node]# grep gluster /etc/rwtab.d/* /etc/rwtab.d/ovirt:files /var/lib/glusterd
[root@ovirt_node]# chkconfig glusterd on [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
#################################### I then reboot the node and see the following: ####################################
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# ls -l /var/lib/glusterd/vols/ total 0 I believe that we intentionally do not start glusterd, since glusterfsd is all that's required for the engine to manage volumes, but I could be mis-remembering this, and I don't have any real arguments to starting glusterd at boot unless somebody speaks up against it.
No more gluster volume configuration files.
I've taken a look through http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Persisting... but I'm unsure what needs to be done to persist this configuration.
To get glusterd to start on boot, do I need to manually persist /etc/rc* files?
I see "files /var/lib/glusterd" mentioned in /etc/rwtab.d/ovirt. Is this a list of the files/dirs that should be persisted automatically? If so, is it recursive and should it include everything in /var/lib/glusterd/vols? rwtab is a mechanism from readonly-root, which walks through the filesystem and says "copy these files to /var/lib/stateless/writable/${path} and bind mount them back in their original location. So you can write files there, but they don't survive reboots on Node.
Since Node is booting from the same ramdisk every time (essentially the ISO copied to the hard drive), this mechanism doesn't really work for us, and persistence is a different mechanism entirely.
TIA for any help with this.
Simon
-----Original Message----- From: Ryan Barry [mailto:rbarry@redhat.com] Sent: 27 May 2014 14:01 To: Fabian Deutsch; Doron Fediuck; Simon Barrett Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
On 05/26/2014 04:14 AM, Fabian Deutsch wrote:
Am Sonntag, den 25.05.2014, 08:18 -0400 schrieb Doron Fediuck:
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: users@ovirt.org Sent: Friday, May 23, 2014 11:29:39 AM Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
I am working through the setup of oVirt node for a 3.4.1 deployment.
I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domain. All was working OK until I rebooted the node and found that the glusterfs configuration had not been retained.
Is there something I should be doing to persist any glusterfs configuration so it survives a node reboot?
Many thanks,
Simon
Hi Simon, it actually sounds like a bug to me, as node are supposed to support gluster.
Ryan / Fabian- thoughts?
Hey,
I vaguely remember that we were seeing a bug like this some time ago. We fixed /var/lib/glusterd to be writable (using tmpfs), but it can actually be that we need to persist those contents.
But Simon, can you give details which configuration files are missing and why glusterd is not starting? Is glusterd starting? I'm getting the impression that it's starting, but that it has no configuration. As far as I know, Gluster keeps most of the configuration on the brick itself, but it finds brick information in /var/lib/glusterd.
The last patch simply opened the firewall, and it's entirely possible that we need to persist this. It may be a good idea to just persist the entire directory from the get-go, unless we want to try to have a thread watching /var/lib/glusterd for relevant files, but then we're stuck trying to keep up with what's happening with gluster itself...
Can we
Thanks fabian
Either way I suggest you take a look in the below link- http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_P er sisting_changes
Let s know how it works.
Doron
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: "Fabian Deutsch" <fdeutsch@redhat.com> Cc: "Ryan Barry" <rbarry@redhat.com>, "Doron Fediuck" <dfediuck@redhat.com>, users@ovirt.org Sent: Wednesday, May 28, 2014 5:22:17 PM Subject: RE: [ovirt-users] Persisting glusterfs configs on an oVirt node
I just wasn't sure if I was missing something in the configuration to enable this.
I'll stick with the workarounds I have for now and see how it goes.
Thanks again.
Simon
-----Original Message----- From: Fabian Deutsch [mailto:fdeutsch@redhat.com] Sent: 28 May 2014 15:20 To: Simon Barrett Cc: Ryan Barry; Doron Fediuck; users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Am Mittwoch, den 28.05.2014, 14:14 +0000 schrieb Simon Barrett:
I did a "persist /var/lib/glusterd" and things are looking better. The gluster config is now still in place after a reboot.
As a workaround to getting glusterd running on boot, I added "service glusterd start" to /etc/rc.local and ran persist /etc/rc.local. It appears to be working but feels like a bit of a hack.
Does anyone have any other suggestions as to the correct way to do this?
Hey Simon,
I was also investigating both the steps you did. And was also about to recommend them :) They are more a workaround.
We basically need some patches to change the defaults on Node, to let gluster work out of the box.
This would include persisting the correct paths and enabling glusterd if enabled.
- fabian
Thanks,
Simon
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Simon Barrett Sent: 28 May 2014 14:12 To: Ryan Barry; Fabian Deutsch; Doron Fediuck Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Thanks for the replies.
I cannot get glusterd to start on boot and I lose all gluster config every reboot.
The following shows what I did on the node to start glusterd, create a volume etc, followed by the state of the node after a reboot.
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# service glusterd start Starting glusterd:[ OK ]
gluster> volume create vmstore 10.22.8.46:/data/glusterfs/vmstore volume create: vmstore: success: please start the volume to access data
gluster> vol start vmstore volume start: vmstore: success
gluster> vol info Volume Name: vmstore Type: Distribute Volume ID: 5bd01043-1352-4014-88ca-e632e264d088 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.22.8.46:/data/glusterfs/vmstore
[root@ovirt_node]# ls -1 /var/lib/glusterd/vols/vmstore/ bricks node_state.info trusted-vmstore-fuse.vol cksum rbstate vmstore.10.22.8.46.data-glusterfs-vmstore.vol info run vmstore-fuse.vol
[root@ovirt_node]# grep gluster /etc/rwtab.d/* /etc/rwtab.d/ovirt:files /var/lib/glusterd
[root@ovirt_node]# chkconfig glusterd on [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
#################################### I then reboot the node and see the following: ####################################
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# ls -l /var/lib/glusterd/vols/ total 0
No more gluster volume configuration files.
I've taken a look through http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Persisting... but I'm unsure what needs to be done to persist this configuration.
To get glusterd to start on boot, do I need to manually persist /etc/rc* files?
I see "files /var/lib/glusterd" mentioned in /etc/rwtab.d/ovirt. Is this a list of the files/dirs that should be persisted automatically? If so, is it recursive and should it include everything in /var/lib/glusterd/vols?
TIA for any help with this.
Simon
-----Original Message----- From: Ryan Barry [mailto:rbarry@redhat.com] Sent: 27 May 2014 14:01 To: Fabian Deutsch; Doron Fediuck; Simon Barrett Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Am Sonntag, den 25.05.2014, 08:18 -0400 schrieb Doron Fediuck:
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: users@ovirt.org Sent: Friday, May 23, 2014 11:29:39 AM Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
I am working through the setup of oVirt node for a 3.4.1 deployment.
I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domain. All was working OK until I rebooted the node and found that the glusterfs configuration had not been retained.
Is there something I should be doing to persist any glusterfs configuration so it survives a node reboot?
Many thanks,
Simon
Hi Simon, it actually sounds like a bug to me, as node are supposed to support gluster.
Ryan / Fabian- thoughts?
Hey,
I vaguely remember that we were seeing a bug like this some time ago. We fixed /var/lib/glusterd to be writable (using tmpfs), but it can actually be that we need to persist those contents.
But Simon, can you give details which configuration files are missing and why glusterd is not starting? Is glusterd starting? I'm getting the impression that it's starting, but
On 05/26/2014 04:14 AM, Fabian Deutsch wrote: that it has no configuration. As far as I know, Gluster keeps most of the configuration on the brick itself, but it finds brick information in /var/lib/glusterd.
The last patch simply opened the firewall, and it's entirely possible that we need to persist this. It may be a good idea to just persist the entire directory from the get-go, unless we want to try to have a thread watching /var/lib/glusterd for relevant files, but then we're stuck trying to keep up with what's happening with gluster itself...
Can we
Thanks fabian
Either way I suggest you take a look in the below link- http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_P er sisting_changes
Let s know how it works.
Doron
I suggest you open a bug so we'll be able to track this issue properly rather than using hacks. Thanks, Doron
Will do. Thanks -----Original Message----- From: Doron Fediuck [mailto:dfediuck@redhat.com] Sent: 28 May 2014 16:17 To: Simon Barrett Cc: Fabian Deutsch; Ryan Barry; users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node ----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: "Fabian Deutsch" <fdeutsch@redhat.com> Cc: "Ryan Barry" <rbarry@redhat.com>, "Doron Fediuck" <dfediuck@redhat.com>, users@ovirt.org Sent: Wednesday, May 28, 2014 5:22:17 PM Subject: RE: [ovirt-users] Persisting glusterfs configs on an oVirt node
I just wasn't sure if I was missing something in the configuration to enable this.
I'll stick with the workarounds I have for now and see how it goes.
Thanks again.
Simon
-----Original Message----- From: Fabian Deutsch [mailto:fdeutsch@redhat.com] Sent: 28 May 2014 15:20 To: Simon Barrett Cc: Ryan Barry; Doron Fediuck; users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Am Mittwoch, den 28.05.2014, 14:14 +0000 schrieb Simon Barrett:
I did a "persist /var/lib/glusterd" and things are looking better. The gluster config is now still in place after a reboot.
As a workaround to getting glusterd running on boot, I added "service glusterd start" to /etc/rc.local and ran persist /etc/rc.local. It appears to be working but feels like a bit of a hack.
Does anyone have any other suggestions as to the correct way to do this?
Hey Simon,
I was also investigating both the steps you did. And was also about to recommend them :) They are more a workaround.
We basically need some patches to change the defaults on Node, to let gluster work out of the box.
This would include persisting the correct paths and enabling glusterd if enabled.
- fabian
Thanks,
Simon
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Simon Barrett Sent: 28 May 2014 14:12 To: Ryan Barry; Fabian Deutsch; Doron Fediuck Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
Thanks for the replies.
I cannot get glusterd to start on boot and I lose all gluster config every reboot.
The following shows what I did on the node to start glusterd, create a volume etc, followed by the state of the node after a reboot.
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# service glusterd start Starting glusterd:[ OK ]
gluster> volume create vmstore 10.22.8.46:/data/glusterfs/vmstore volume create: vmstore: success: please start the volume to access data
gluster> vol start vmstore volume start: vmstore: success
gluster> vol info Volume Name: vmstore Type: Distribute Volume ID: 5bd01043-1352-4014-88ca-e632e264d088 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.22.8.46:/data/glusterfs/vmstore
[root@ovirt_node]# ls -1 /var/lib/glusterd/vols/vmstore/ bricks node_state.info trusted-vmstore-fuse.vol cksum rbstate vmstore.10.22.8.46.data-glusterfs-vmstore.vol info run vmstore-fuse.vol
[root@ovirt_node]# grep gluster /etc/rwtab.d/* /etc/rwtab.d/ovirt:files /var/lib/glusterd
[root@ovirt_node]# chkconfig glusterd on [root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
#################################### I then reboot the node and see the following: ####################################
[root@ovirt_node]# service glusterd status glusterd is stopped
[root@ovirt_node]# chkconfig --list glusterd glusterd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@ovirt_node]# ls -l /var/lib/glusterd/vols/ total 0
No more gluster volume configuration files.
I've taken a look through http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F_Pe rsisting_changes but I'm unsure what needs to be done to persist this configuration.
To get glusterd to start on boot, do I need to manually persist /etc/rc* files?
I see "files /var/lib/glusterd" mentioned in /etc/rwtab.d/ovirt. Is this a list of the files/dirs that should be persisted automatically? If so, is it recursive and should it include everything in /var/lib/glusterd/vols?
TIA for any help with this.
Simon
-----Original Message----- From: Ryan Barry [mailto:rbarry@redhat.com] Sent: 27 May 2014 14:01 To: Fabian Deutsch; Doron Fediuck; Simon Barrett Cc: users@ovirt.org Subject: Re: [ovirt-users] Persisting glusterfs configs on an oVirt node
On 05/26/2014 04:14 AM, Fabian Deutsch wrote:
Am Sonntag, den 25.05.2014, 08:18 -0400 schrieb Doron Fediuck:
----- Original Message -----
From: "Simon Barrett" <Simon.Barrett@tradingscreen.com> To: users@ovirt.org Sent: Friday, May 23, 2014 11:29:39 AM Subject: [ovirt-users] Persisting glusterfs configs on an oVirt node
I am working through the setup of oVirt node for a 3.4.1 deployment.
I setup some glusterfs volumes/bricks on oVirt Node Hypervisor release 3.0.4 (1.0.201401291204.el6) and created a storage domain. All was working OK until I rebooted the node and found that the glusterfs configuration had not been retained.
Is there something I should be doing to persist any glusterfs configuration so it survives a node reboot?
Many thanks,
Simon
Hi Simon, it actually sounds like a bug to me, as node are supposed to support gluster.
Ryan / Fabian- thoughts?
Hey,
I vaguely remember that we were seeing a bug like this some time ago. We fixed /var/lib/glusterd to be writable (using tmpfs), but it can actually be that we need to persist those contents.
But Simon, can you give details which configuration files are missing and why glusterd is not starting? Is glusterd starting? I'm getting the impression that it's starting, but that it has no configuration. As far as I know, Gluster keeps most of the configuration on the brick itself, but it finds brick information in /var/lib/glusterd.
The last patch simply opened the firewall, and it's entirely possible that we need to persist this. It may be a good idea to just persist the entire directory from the get-go, unless we want to try to have a thread watching /var/lib/glusterd for relevant files, but then we're stuck trying to keep up with what's happening with gluster itself...
Can we
Thanks fabian
Either way I suggest you take a look in the below link- http://www.ovirt.org/Node_Troubleshooting#Making_changes_last_.2F _P er sisting_changes
Let s know how it works.
Doron
I suggest you open a bug so we'll be able to track this issue properly rather than using hacks. Thanks, Doron
participants (4)
-
Doron Fediuck -
Fabian Deutsch -
Ryan Barry -
Simon Barrett