[ovirt-users] ovirt 3.6 and gluster arbiter volumes?

[arik.mitschang at sbibits.com]

Hi Nir,

> On Wed, Jan 20, 2016 at 1:31 AM,  <arik.mitschang at sbibits.com> wrote:
>>> On 25-12-2015 5:26, Arik Mitschang wrote:
>>>> Hi ovirt-users,
>>>>
>>>> I have been working on a new install of ovirt 3.6 hosted-engine and ran
>>>> into difficulty adding a gluster data storage domain to host my VMs. I
>>>> have 4 servers for gluster (separate from vm hosts) and would like to
>>>> have the quorum enforcement of replica 3 without sacrificing space. I
>>>> created a gluster using
>>>>
>>>>  replica 3 arbiter 1
>>>>
>>>> That looks like this:
>>>>
>>>>  Volume Name: arbtest
>>>>  Type: Distributed-Replicate
>>>>  Volume ID: 01b36368-1f37-435c-9f48-0442e0c34160
>>>>  Status: Stopped
>>>>  Number of Bricks: 2 x 3 = 6
>>>>  Transport-type: tcp
>>>>  Bricks:
>>>>  Brick1: t2-gluster01b:/gluster/00/arbtest
>>>>  Brick2: t2-gluster02b:/gluster/00/arbtest
>>>>  Brick3: t2-gluster03b:/gluster/00/arbtest.arb
>>>>  Brick4: t2-gluster03b:/gluster/00/arbtest
>>>>  Brick5: t2-gluster04b:/gluster/00/arbtest
>>>>  Brick6: t2-gluster01b:/gluster/00/arbtest.arb
>>>>  Options Reconfigured:
>>>>  nfs.disable: true
>>>>  network.ping-timeout: 10
>>>>  storage.owner-uid: 36
>>>>  storage.owner-gid: 36
>>>>  cluster.server-quorum-type: server
>>>>  cluster.quorum-type: auto
>>>>  network.remote-dio: enable
>>>>  cluster.eager-lock: enable
>>>>  performance.stat-prefetch: off
>>>>  performance.io-cache: off
>>>>  performance.read-ahead: off
>>>>  performance.quick-read: off
>>>>  performance.readdir-ahead: on
>>>>
>>>> But adding to gluster I get the following error:
>>>>
>>>>  "Error while executing action AddGlusterFsStorageDomain: Error creating
>>>>  a storage domain's metadata"
> 
> In vdsm log we see:
> 
> StorageDomainMetadataCreationError: Error creating a storage domain's
> metadata: ("create meta file 'outbox' failed: [Errno 5] Input/output
> error",)
> 
> Which does not mean much.
> 
>>>>
>>>>
>>> Anything in engine.log (/var/log/ovirt-engine/engine.log) around that time?
>>> Anything in vdsm.log on your 2 hypervisors around that time?
>>> (Guessing that you'll see an error about replication unsupported by
>>> vdsm, if so, have a look at /etc/vdsmd.conf.rpmnew)
>>
>> Hi Joop,
>>
>> Thanks for your response, and sorry for the long delay in mine. I had a
>> chance to test adding again and catch the logs around that operation. I
>> am attaching the engine logs and vdsm logs of the hypervisor that was
>> responsible for the storage operations.
>>
>> Also, I have the following:
>>
>>  [gluster]
>>  allowed_replica_counts = 1,2,3
>>
>> in /etc/vdsm/vdsm.conf.
>>
>> The volume was successfully mounted and I see the following in it after
>> trying to add:
>>
>>  arik at t2-virt01:~$ sudo mount -t glusterfs t2-gluster01b:arbtest /mnt/
>>  arik at t2-virt01:~$ ls -ltr /mnt/
>>  total 0
>>  -rwxr-xr-x 1 vdsm kvm  0 Jan 20 08:08 __DIRECT_IO_TEST__
>>  drwxr-xr-x 3 vdsm kvm 54 Jan 20 08:08 3d31af0b-18ad-45c4-90f1-18e2f804f34b
>>
>> I hope you can see something interesting in these logs!
> 
> You may find more info in gluster mount log, which should be at:
> 
> /var/log/glusterfs/<server>:<volname><date>.log

I will take a look if I can find something in the gluster logs.

> 
> We (ovirt storage developers) did not try arbiter volumes yet, so this
> is basically
> unsupported :-)

Ah, understood. Any plans to try?

> 
> The recommended setup is replica 3. Can you try to create a small
> replica 3 volume,
> just to check that replica 3 works for you?

The hosted engine for our setup is on a replica 3 volume, and this works
well.

Regards,
-Arik