On 06/25/2012 07:59 PM, Nathan Stratton wrote: > > Steps: > > 1) Installed ovirt-engine, configured Cluster for GlusterFS > 2) Installed 8 ovirt 3.1 nodes > 3) Joined all 8 notes with ovirt-engine (all up and happy) > 4) Manually added all 8 peers for GlusterFS on hosts (all peers happy) > 5) Create Volume errors: > > 2012-06-25 16:44:42,412 WARN > [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] > (ajp--0.0.0.0-8009-4) [6c02c1f5] The message key CreateGlusterVolume is > missing from bundles/ExecutionMessages > 2012-06-25 16:44:42,483 INFO > [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] > (ajp--0.0.0.0-8009-4) [6c02c1f5] Running command: > CreateGlusterVolumeCommand internal: false. Entities affected : ID: > 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups > 2012-06-25 16:44:42,486 INFO > [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] > (ajp--0.0.0.0-8009-4) [6c02c1f5] START, CreateGlusterVolumeVDSCommand(vdsId > = 8324ff12-bf1c-11e1-b235-43d3f71a81d8), log id: 15757d4c > 2012-06-25 16:44:42,593 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] > (ajp--0.0.0.0-8009-4) [6c02c1f5] Failed in CreateGlusterVolumeVDS method > 2012-06-25 16:44:42,594 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] > (ajp--0.0.0.0-8009-4) [6c02c1f5] Error code unexpected and error message > VDSGenericException: VDSErrorException: Failed to CreateGlusterVolumeVDS, > error = Unexpected exception > 2012-06-25 16:44:42,594 ERROR > [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp--0.0.0.0-8009-4) > [6c02c1f5] Command CreateGlusterVolumeVDS execution failed. Exception: > VDSErrorException: VDSGenericException: VDSErrorException: Failed to > CreateGlusterVolumeVDS, error = Unexpected exception > 2012-06-25 16:44:42,595 INFO > [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] > (ajp--0.0.0.0-8009-4) [6c02c1f5] FINISH, CreateGlusterVolumeVDSCommand, log > id: 15757d4c > 2012-06-25 16:44:42,596 ERROR > [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] > (ajp--0.0.0.0-8009-4) [6c02c1f5] Command > org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand throw Vdc Bll > exception. With error message VdcBLLException: > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: > VDSGenericException: VDSErrorException: Failed to CreateGlusterVolumeVDS, > error = Unexpected exception > I am tiring to write up a howto on the process. Since I just got everything working recently and am testing right now. Make sure you have gluster 3.3 installed not gluster 3.2.5 that comes with F17. I installed from http://repos.fedorapeople.org/repos/kkeithle/glusterfs/ .

<>

Make sure vdsm-gluster is installed on all nodes? It wasn't installed on nodes that were added before I added a check box to support gluster on the cluster.

<>

> I now have a happy gluster volume called share. Now I am lost as to how to > use that volume. Do I have to change the Default NFS storage type? How do I > get this bad boy mounted? I spent 2 weeks tiring to get glusterfs working with no luck. However I was able to get gluster nfs working. Here are the steps I needed to do.

1) I added the following option to the volume. Might only need the 1st one but I have all 3 of them set on my working system. a) nfs.nlm off b) nfs.register-with-portmap on c) nfs.addr-namelookup off 2) Mount the volume outside of ovirt. mount -t nfs ipofonenode:/volumename /temp/share/name 3) chown -R 36.36 /temp/share/name. 4) umount /temp/share/name 5) Added a new NFS domain with the NFS Export Path: localhost:/volumename I was then able to active the NFS based Data Center and add working VM's

<>

> Problme I am running into is that I need to add -o mountproto=tcp,vers=3 > because it looks like it is defaulting to udp and gluster NFS does not > support that. Try editing file /etc/nfsmount.conf on each node and add the following. [ NFSMount_Global_Options ] Defaultvers=3 Nfsvers=3

<>

On 06/25/2012 09:26 PM, Nathan Stratton wrote: > On Mon, 25 Jun 2012, Robert Middleswarth wrote: > >> Make sure vdsm-gluster is installed on all nodes? It wasn't installed >> on nodes that were added before I added a check box to support >> gluster on the cluster. > > Yes, it was out of 8 boxes I was looking at the vdsm logs on box 1 and > ovirt-engine tried to start gluster on a different box so I did not > see the vdsm log. Once I did, it was clear that gluster we not happy > about the directory being used for gluster in the past. To fix that I > just ran the following on all 8 of my hosts: > > cd /export/ ;for i in `attr -lq .`; do setfattr -x trusted.$i .; done > > I now have a happy gluster volume called share. Now I am lost as to > how to use that volume. Do I have to change the Default NFS storage > type? How do I get this bad boy mounted? I spent 2 weeks tiring to get glusterfs working with no luck. However I was able to get gluster nfs working. Here are the steps I needed to do. 1) I added the following option to the volume. Might only need the 1st one but I have all 3 of them set on my working system. a) nfs.nlm off b) nfs.register-with-portmap on c) nfs.addr-namelookup off 2) Mount the volume outside of ovirt. mount -t nfs ipofonenode:/volumename /temp/share/name 3) chown -R 36.36 /temp/share/name. 4) umount /temp/share/name 5) Added a new NFS domain with the NFS Export Path: localhost:/volumename

I was then able to active the NFS based Data Center and add working VM's Thanks Robert >> <> > Nathan Stratton > nathan at robotics.net > http://www.robotics.net > _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On 06/25/2012 07:59 PM, Nathan Stratton wrote: > > Steps: > > 1) Installed ovirt-engine, configured Cluster for GlusterFS > 2) Installed 8 ovirt 3.1 nodes > 3) Joined all 8 notes with ovirt-engine (all up and happy) > 4) Manually added all 8 peers for GlusterFS on hosts (all peers happy) I thought ovirt engine takes care of that. did you add these nodes to a gluster cluster from the UI?

> 5) Create Volume errors: > > 2012-06-25 16:44:42,412 WARN > [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] > (ajp--0.0.0.0-8009-4) [6c02c1f5] The message key CreateGlusterVolume is > missing from bundles/ExecutionMessages > 2012-06-25 16:44:42,483 INFO > [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] > (ajp--0.0.0.0-8009-4) [6c02c1f5] Running command: > CreateGlusterVolumeCommand internal: false. Entities affected : ID: > 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups > 2012-06-25 16:44:42,486 INFO > [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] > (ajp--0.0.0.0-8009-4) [6c02c1f5] START, > CreateGlusterVolumeVDSCommand(vdsId = > 8324ff12-bf1c-11e1-b235-43d3f71a81d8), log id: 15757d4c > 2012-06-25 16:44:42,593 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] > (ajp--0.0.0.0-8009-4) [6c02c1f5] Failed in CreateGlusterVolumeVDS method > 2012-06-25 16:44:42,594 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] > (ajp--0.0.0.0-8009-4) [6c02c1f5] Error code unexpected and error message > VDSGenericException: VDSErrorException: Failed to > CreateGlusterVolumeVDS, error = Unexpected exception > 2012-06-25 16:44:42,594 ERROR > [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp--0.0.0.0-8009-4) > [6c02c1f5] Command CreateGlusterVolumeVDS execution failed. Exception: > VDSErrorException: VDSGenericException: VDSErrorException: Failed to > CreateGlusterVolumeVDS, error = Unexpected exception > 2012-06-25 16:44:42,595 INFO > [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] > (ajp--0.0.0.0-8009-4) [6c02c1f5] FINISH, CreateGlusterVolumeVDSCommand, > log id: 15757d4c > 2012-06-25 16:44:42,596 ERROR > [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] > (ajp--0.0.0.0-8009-4) [6c02c1f5] Command > org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand throw Vdc > Bll exception. With error message VdcBLLException: > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: > VDSGenericException: VDSErrorException: Failed to > CreateGlusterVolumeVDS, error = Unexpected exception > > >> <> > Nathan Stratton > nathan at robotics.net > http://www.robotics.net > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On 06/26/2012 12:22 AM, Robert Middleswarth wrote: > On 06/25/2012 11:50 PM, Itamar Heim wrote: >> On 06/25/2012 07:59 PM, Nathan Stratton wrote: >>> >>> Steps: >>> >>> 1) Installed ovirt-engine, configured Cluster for GlusterFS >>> 2) Installed 8 ovirt 3.1 nodes >>> 3) Joined all 8 notes with ovirt-engine (all up and happy) >>> 4) Manually added all 8 peers for GlusterFS on hosts (all peers happy) >> >> I thought ovirt engine takes care of that. >> did you add these nodes to a gluster cluster from the UI? >> > No it doesn't. I have done several installs. I already bugged the issue > about the fact that the iptables ports don't get opened and if you do > anything inside the engine that updates the iptables it removes the ones > I manually add. I have done both a f17 install using beta and Jenkins > and the CentOS builds including the latest build 8 and none of these > setup IP tables or links the nodes. ip tables - no. peer addition - from what i understand, works if you use installVds=false still needs to be fixed to work with installVds=true as well. option A - a hacky way for now might be: 1. use installVds=true 2. add all hosts so they will get configured with vdsm and certificates and regular ip tables 3. change installVds to false 4. remove/add them with the gluster flag not sure worth it for peer detection for now. option B 1. set InstallVds=false and disable ssl 2. yum install the hosts yourself option C 1. use like today and do peer detection manually until resolved. btw, i saw you manged to get it to work using NFS. did you try the gluster native client/mount as well?

> > Thanks > Robert >>> 5) Create Volume errors: >>> >>> 2012-06-25 16:44:42,412 WARN >>> [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] >>> (ajp--0.0.0.0-8009-4) [6c02c1f5] The message key >>> CreateGlusterVolume is >>> missing from bundles/ExecutionMessages >>> 2012-06-25 16:44:42,483 INFO >>> [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] >>> (ajp--0.0.0.0-8009-4) [6c02c1f5] Running command: >>> CreateGlusterVolumeCommand internal: false. Entities affected : ID: >>> 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups >>> 2012-06-25 16:44:42,486 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] >>> >>> (ajp--0.0.0.0-8009-4) [6c02c1f5] START, >>> CreateGlusterVolumeVDSCommand(vdsId = >>> 8324ff12-bf1c-11e1-b235-43d3f71a81d8), log id: 15757d4c >>> 2012-06-25 16:44:42,593 ERROR >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] >>> (ajp--0.0.0.0-8009-4) [6c02c1f5] Failed in CreateGlusterVolumeVDS >>> method >>> 2012-06-25 16:44:42,594 ERROR >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] >>> (ajp--0.0.0.0-8009-4) [6c02c1f5] Error code unexpected and error >>> message >>> VDSGenericException: VDSErrorException: Failed to >>> CreateGlusterVolumeVDS, error = Unexpected exception >>> 2012-06-25 16:44:42,594 ERROR >>> [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp--0.0.0.0-8009-4) >>> [6c02c1f5] Command CreateGlusterVolumeVDS execution failed. Exception: >>> VDSErrorException: VDSGenericException: VDSErrorException: Failed to >>> CreateGlusterVolumeVDS, error = Unexpected exception >>> 2012-06-25 16:44:42,595 INFO >>> [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] >>> >>> (ajp--0.0.0.0-8009-4) [6c02c1f5] FINISH, >>> CreateGlusterVolumeVDSCommand, >>> log id: 15757d4c >>> 2012-06-25 16:44:42,596 ERROR >>> [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] >>> (ajp--0.0.0.0-8009-4) [6c02c1f5] Command >>> org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand throw Vdc >>> Bll exception. With error message VdcBLLException: >>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: >>> VDSGenericException: VDSErrorException: Failed to >>> CreateGlusterVolumeVDS, error = Unexpected exception >>> >>> >>>> <> >>> Nathan Stratton >>> nathan at robotics.net >>> http://www.robotics.net >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users