Hello All,

I recently took a new job in a RedHat shop and I'd like to move all of my homelab systems to RedHat upstream products to better align with what I manage at work. I had a "custom" (aka - hacked together) 3-node Hyperconverged XenServer cluster and would like to get this moved over to Ovirt (I'm currently testing with 4.2.7). Unfortunately, my storage is limited to software RAID with a 128GB SSD for cache. If at all possible, I would prefer to use ZFS (RAIDZ+ZIL+L2ARC) instead of MD RAID + lvmcache, however I'm not able to get this working and I'm not sure why. My ZFS and Gluster configuration is working - at least where I can manually mount all of my gluster volumes from all of my nodes, however, hosted-engine --deploy fails. I understand this isn't an out of the box configuration for Ovirt, however I see no reason why this shouldn't work. I would think this would be no different than using any other Gluster volume for the engine datastore, Am I missing something that would prevent this from working?

[ INFO  ] TASK [Add glusterfs storage domain]
[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Storage Domain target is unsupported]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage Domain target is unsupported]\". HTTP response code is 400."}

Even though it fails, it appears to have mounted and written __DIRECT_IO_TEST__ to my Gluster volume:

[root@vmh1 ~]# mount -t glusterfs localhost:/engine /mnt/engine/
[root@vmh1 ~]# ls /mnt/engine/
__DIRECT_IO_TEST__

If I cancel and try to run the deploy again, I get a different failure:

[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Error creating a storage domain]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Error creating a storage domain]\". HTTP response code is 400."}

Gluster seems ok...

[root@vmh1 /]# gluster volume info engine
 
Volume Name: engine
Type: Replicate
Volume ID: 2e34f8f5-0129-4ba5-983f-1eb5178deadc
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: vmh1-ib:/zpool1/engine
Brick2: vmh2-ib:/zpool1/engine
Brick3: vmh3-ib:/zpool1/engine
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

ZFS looks good too...

[root@vmh1 ~]# ansible ovirthosts -m shell -a 'zpool status' -b
vmh1 | CHANGED | rc=0 >>
  pool: zpool1
 state: ONLINE
  scan: none requested
config:

NAME        STATE     READ WRITE CKSUM
zpool1      ONLINE       0     0     0
  sdc       ONLINE       0     0     0
  sdd       ONLINE       0     0     0
  sde       ONLINE       0     0     0
logs
  sdb2      ONLINE       0     0     0
cache
  sdb1      ONLINE       0     0     0

errors: No known data errors

vmh3 | CHANGED | rc=0 >>
  pool: zpool1
 state: ONLINE
  scan: none requested
config:

NAME        STATE     READ WRITE CKSUM
zpool1      ONLINE       0     0     0
  sdc       ONLINE       0     0     0
  sdd       ONLINE       0     0     0
  sde       ONLINE       0     0     0
logs
  sdb2      ONLINE       0     0     0
cache
  sdb1      ONLINE       0     0     0

errors: No known data errors

vmh2 | CHANGED | rc=0 >>
  pool: zpool1
 state: ONLINE
  scan: none requested
config:

NAME        STATE     READ WRITE CKSUM
zpool1      ONLINE       0     0     0
  sdc       ONLINE       0     0     0
  sdd       ONLINE       0     0     0
  sde       ONLINE       0     0     0
logs
  sdb2      ONLINE       0     0     0
cache
  sdb1      ONLINE       0     0     0

errors: No known data errors

Permissions seem ok

[root@vmh1 ~]# ansible ovirthosts -m shell -a 'ls -n /zpool1' -b
vmh3 | CHANGED | rc=0 >>
total 2
drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data
drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine
drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso

vmh1 | CHANGED | rc=0 >>
total 2
drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data
drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine
drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso

vmh2 | CHANGED | rc=0 >>
total 2
drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data
drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine
drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso