Hyperconverged Ovirt + ZFS
Hello All, I recently took a new job in a RedHat shop and I'd like to move all of my homelab systems to RedHat upstream products to better align with what I manage at work. I had a "custom" (aka - hacked together) 3-node Hyperconverged XenServer cluster and would like to get this moved over to Ovirt (I'm currently testing with 4.2.7). Unfortunately, my storage is limited to software RAID with a 128GB SSD for cache. If at all possible, I would prefer to use ZFS (RAIDZ+ZIL+L2ARC) instead of MD RAID + lvmcache, however I'm not able to get this working and I'm not sure why. My ZFS and Gluster configuration is working - at least where I can manually mount all of my gluster volumes from all of my nodes, however, hosted-engine --deploy fails. I understand this isn't an out of the box configuration for Ovirt, however I see no reason why this shouldn't work. I would think this would be no different than using any other Gluster volume for the engine datastore, Am I missing something that would prevent this from working? [ INFO ] TASK [Add glusterfs storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Storage Domain target is unsupported]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage Domain target is unsupported]\". HTTP response code is 400."} Even though it fails, it appears to have mounted and written __DIRECT_IO_TEST__ to my Gluster volume: [root@vmh1 ~]# mount -t glusterfs localhost:/engine /mnt/engine/ [root@vmh1 ~]# ls /mnt/engine/ __DIRECT_IO_TEST__ If I cancel and try to run the deploy again, I get a different failure: [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Error creating a storage domain]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Error creating a storage domain]\". HTTP response code is 400."} Gluster seems ok... [root@vmh1 /]# gluster volume info engine Volume Name: engine Type: Replicate Volume ID: 2e34f8f5-0129-4ba5-983f-1eb5178deadc Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: vmh1-ib:/zpool1/engine Brick2: vmh2-ib:/zpool1/engine Brick3: vmh3-ib:/zpool1/engine Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off ZFS looks good too... [root@vmh1 ~]# ansible ovirthosts -m shell -a 'zpool status' -b vmh1 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0 errors: No known data errors vmh3 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0 errors: No known data errors vmh2 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0 errors: No known data errors Permissions seem ok [root@vmh1 ~]# ansible ovirthosts -m shell -a 'ls -n /zpool1' -b vmh3 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso vmh1 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso vmh2 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
I did this in the past and didn’t have any trouble with gluster/ZFS, but 4.2.x probably does more validation. I recommend these settings on your zfs volumes, I set mine at the root(v0 here) and let them inherit: required: v0 xattr sa local v0 acltype posixacl local optional but I recommend them: v0 relatime on local v0 compression lz4 local For gluster, I think it checks that it’s been “optimized for virt storage”. Either apply the virt group or set the options you’ll find in /var/lib/glusterd/groups/virt. Note that I don’t recommend the default settings for cluster.shd-max-threads & cluster.shd-wait-qlength. They can swamp your machines during heals unless you have a lot of cores and ram. You get a slightly faster heal, but often have VMs pausing for storage or other odd ball storage related errors. I prefer max-threads = 1 or maybe 2, and wait-qlength=1024 or 2048. These are per volume, so they hit harder than you think they will if you have a lot of volumes running. Also make sure the gluster volumes themselves got set to 36.36 for owner.group, doesn’t matter for the bricks. Can do it with volume settings or mount the volume and set it manually. Hope it helps! -Darrell
On Nov 15, 2018, at 6:28 AM, Thomas Simmons <twsnnva@gmail.com> wrote:
Hello All,
I recently took a new job in a RedHat shop and I'd like to move all of my homelab systems to RedHat upstream products to better align with what I manage at work. I had a "custom" (aka - hacked together) 3-node Hyperconverged XenServer cluster and would like to get this moved over to Ovirt (I'm currently testing with 4.2.7). Unfortunately, my storage is limited to software RAID with a 128GB SSD for cache. If at all possible, I would prefer to use ZFS (RAIDZ+ZIL+L2ARC) instead of MD RAID + lvmcache, however I'm not able to get this working and I'm not sure why. My ZFS and Gluster configuration is working - at least where I can manually mount all of my gluster volumes from all of my nodes, however, hosted-engine --deploy fails. I understand this isn't an out of the box configuration for Ovirt, however I see no reason why this shouldn't work. I would think this would be no different than using any other Gluster volume for the engine datastore, Am I missing something that would prevent this from working?
[ INFO ] TASK [Add glusterfs storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Storage Domain target is unsupported]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage Domain target is unsupported]\". HTTP response code is 400."}
Even though it fails, it appears to have mounted and written __DIRECT_IO_TEST__ to my Gluster volume:
[root@vmh1 ~]# mount -t glusterfs localhost:/engine /mnt/engine/ [root@vmh1 ~]# ls /mnt/engine/ __DIRECT_IO_TEST__
If I cancel and try to run the deploy again, I get a different failure:
[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Error creating a storage domain]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Error creating a storage domain]\". HTTP response code is 400."}
Gluster seems ok...
[root@vmh1 /]# gluster volume info engine
Volume Name: engine Type: Replicate Volume ID: 2e34f8f5-0129-4ba5-983f-1eb5178deadc Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: vmh1-ib:/zpool1/engine Brick2: vmh2-ib:/zpool1/engine Brick3: vmh3-ib:/zpool1/engine Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off
ZFS looks good too...
[root@vmh1 ~]# ansible ovirthosts -m shell -a 'zpool status' -b vmh1 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
vmh3 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
vmh2 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
Permissions seem ok
[root@vmh1 ~]# ansible ovirthosts -m shell -a 'ls -n /zpool1' -b vmh3 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
vmh1 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
vmh2 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/SEZLAUIUSLR4UF...
Why not just use the built in stuff like VDO. What benefits does ZFS bring for the use case? For most vm based workloads ZFS is the opposite of ideal over the lifecycle of a VM. COW filesystems have downsides over time. On Thu, Nov 15, 2018 at 6:09 PM Darrell Budic <budic@onholyground.com> wrote:
I did this in the past and didn’t have any trouble with gluster/ZFS, but 4.2.x probably does more validation.
I recommend these settings on your zfs volumes, I set mine at the root(v0 here) and let them inherit:
required: v0 xattr sa local v0 acltype posixacl local optional but I recommend them: v0 relatime on local v0 compression lz4 local
For gluster, I think it checks that it’s been “optimized for virt storage”. Either apply the virt group or set the options you’ll find in /var/lib/glusterd/groups/virt.
Note that I don’t recommend the default settings for cluster.shd-max-threads & cluster.shd-wait-qlength. They can swamp your machines during heals unless you have a lot of cores and ram. You get a slightly faster heal, but often have VMs pausing for storage or other odd ball storage related errors. I prefer max-threads = 1 or maybe 2, and wait-qlength=1024 or 2048. These are per volume, so they hit harder than you think they will if you have a lot of volumes running.
Also make sure the gluster volumes themselves got set to 36.36 for owner.group, doesn’t matter for the bricks. Can do it with volume settings or mount the volume and set it manually.
Hope it helps!
-Darrell
On Nov 15, 2018, at 6:28 AM, Thomas Simmons <twsnnva@gmail.com> wrote:
Hello All,
I recently took a new job in a RedHat shop and I'd like to move all of my homelab systems to RedHat upstream products to better align with what I manage at work. I had a "custom" (aka - hacked together) 3-node Hyperconverged XenServer cluster and would like to get this moved over to Ovirt (I'm currently testing with 4.2.7). Unfortunately, my storage is limited to software RAID with a 128GB SSD for cache. If at all possible, I would prefer to use ZFS (RAIDZ+ZIL+L2ARC) instead of MD RAID + lvmcache, however I'm not able to get this working and I'm not sure why. My ZFS and Gluster configuration is working - at least where I can manually mount all of my gluster volumes from all of my nodes, however, hosted-engine --deploy fails. I understand this isn't an out of the box configuration for Ovirt, however I see no reason why this shouldn't work. I would think this would be no different than using any other Gluster volume for the engine datastore, Am I missing something that would prevent this from working?
[ INFO ] TASK [Add glusterfs storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Storage Domain target is unsupported]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage Domain target is unsupported]\". HTTP response code is 400."}
Even though it fails, it appears to have mounted and written __DIRECT_IO_TEST__ to my Gluster volume:
[root@vmh1 ~]# mount -t glusterfs localhost:/engine /mnt/engine/ [root@vmh1 ~]# ls /mnt/engine/ __DIRECT_IO_TEST__
If I cancel and try to run the deploy again, I get a different failure:
[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Error creating a storage domain]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Error creating a storage domain]\". HTTP response code is 400."}
Gluster seems ok...
[root@vmh1 /]# gluster volume info engine
Volume Name: engine Type: Replicate Volume ID: 2e34f8f5-0129-4ba5-983f-1eb5178deadc Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: vmh1-ib:/zpool1/engine Brick2: vmh2-ib:/zpool1/engine Brick3: vmh3-ib:/zpool1/engine Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off
ZFS looks good too...
[root@vmh1 ~]# ansible ovirthosts -m shell -a 'zpool status' -b vmh1 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
vmh3 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
vmh2 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
Permissions seem ok
[root@vmh1 ~]# ansible ovirthosts -m shell -a 'ls -n /zpool1' -b vmh3 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
vmh1 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
vmh2 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/SEZLAUIUSLR4UF...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZFZ7P6RFZ7XB3Y...
Well, personally I know ZFS and I don’t know VDO. Going to have to check it out now that I know it exists, sounds interesting to have it at the dm layer. I don’t use it exclusively either, but have been finding it useful for compression. What is your source for the statement that COW filesystems have downsides over time for VM workloads?
On Nov 16, 2018, at 3:11 PM, Donny Davis <donny@fortnebula.com> wrote:
Why not just use the built in stuff like VDO. What benefits does ZFS bring for the use case? For most vm based workloads ZFS is the opposite of ideal over the lifecycle of a VM. COW filesystems have downsides over time.
On Thu, Nov 15, 2018 at 6:09 PM Darrell Budic <budic@onholyground.com <mailto:budic@onholyground.com>> wrote: I did this in the past and didn’t have any trouble with gluster/ZFS, but 4.2.x probably does more validation.
I recommend these settings on your zfs volumes, I set mine at the root(v0 here) and let them inherit:
required: v0 xattr sa local v0 acltype posixacl local optional but I recommend them: v0 relatime on local v0 compression lz4 local
For gluster, I think it checks that it’s been “optimized for virt storage”. Either apply the virt group or set the options you’ll find in /var/lib/glusterd/groups/virt.
Note that I don’t recommend the default settings for cluster.shd-max-threads & cluster.shd-wait-qlength. They can swamp your machines during heals unless you have a lot of cores and ram. You get a slightly faster heal, but often have VMs pausing for storage or other odd ball storage related errors. I prefer max-threads = 1 or maybe 2, and wait-qlength=1024 or 2048. These are per volume, so they hit harder than you think they will if you have a lot of volumes running.
Also make sure the gluster volumes themselves got set to 36.36 for owner.group, doesn’t matter for the bricks. Can do it with volume settings or mount the volume and set it manually.
Hope it helps!
-Darrell
On Nov 15, 2018, at 6:28 AM, Thomas Simmons <twsnnva@gmail.com <mailto:twsnnva@gmail.com>> wrote:
Hello All,
I recently took a new job in a RedHat shop and I'd like to move all of my homelab systems to RedHat upstream products to better align with what I manage at work. I had a "custom" (aka - hacked together) 3-node Hyperconverged XenServer cluster and would like to get this moved over to Ovirt (I'm currently testing with 4.2.7). Unfortunately, my storage is limited to software RAID with a 128GB SSD for cache. If at all possible, I would prefer to use ZFS (RAIDZ+ZIL+L2ARC) instead of MD RAID + lvmcache, however I'm not able to get this working and I'm not sure why. My ZFS and Gluster configuration is working - at least where I can manually mount all of my gluster volumes from all of my nodes, however, hosted-engine --deploy fails. I understand this isn't an out of the box configuration for Ovirt, however I see no reason why this shouldn't work. I would think this would be no different than using any other Gluster volume for the engine datastore, Am I missing something that would prevent this from working?
[ INFO ] TASK [Add glusterfs storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Storage Domain target is unsupported]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage Domain target is unsupported]\". HTTP response code is 400."}
Even though it fails, it appears to have mounted and written __DIRECT_IO_TEST__ to my Gluster volume:
[root@vmh1 ~]# mount -t glusterfs localhost:/engine /mnt/engine/ [root@vmh1 ~]# ls /mnt/engine/ __DIRECT_IO_TEST__
If I cancel and try to run the deploy again, I get a different failure:
[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Error creating a storage domain]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Error creating a storage domain]\". HTTP response code is 400."}
Gluster seems ok...
[root@vmh1 /]# gluster volume info engine
Volume Name: engine Type: Replicate Volume ID: 2e34f8f5-0129-4ba5-983f-1eb5178deadc Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: vmh1-ib:/zpool1/engine Brick2: vmh2-ib:/zpool1/engine Brick3: vmh3-ib:/zpool1/engine Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off
ZFS looks good too...
[root@vmh1 ~]# ansible ovirthosts -m shell -a 'zpool status' -b vmh1 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
vmh3 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
vmh2 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
Permissions seem ok
[root@vmh1 ~]# ansible ovirthosts -m shell -a 'ls -n /zpool1' -b vmh3 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
vmh1 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
vmh2 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/SEZLAUIUSLR4UF... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/SEZLAUIUSLR4UFHGQNS27KQ5Y5TIZDL6/>
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZFZ7P6RFZ7XB3Y... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZFZ7P6RFZ7XB3YMRIIIVOLE4H4DGYTWY/>
Hello Donny, I also am familiar with ZFS and am not familiar with VDO at all. I took a quick look at VDO, but it's something to look into. It sounds like VDO is more of a layer above RAID, whereas I'm looking for something to implement the RAID. Apart from all of the other features provided by ZFS, I was looking for a specific solution - I have 3 "large" spinning disks and 1 smaller ssd in each of my 3 nodes. I would like to "combine" the 3 spinning disks (while providing some redundancy) and use the SSD as a cache device. I can't buy more hardware right now so this need to be done in software. The obvious choices for me would be Linux MD software RAID + LVM cache or ZFS. I personally prefer ZFS due to it's feature-set and simplicity. On Fri, Nov 16, 2018 at 9:22 PM Darrell Budic <budic@onholyground.com> wrote:
Well, personally I know ZFS and I don’t know VDO. Going to have to check it out now that I know it exists, sounds interesting to have it at the dm layer. I don’t use it exclusively either, but have been finding it useful for compression.
What is your source for the statement that COW filesystems have downsides over time for VM workloads?
On Nov 16, 2018, at 3:11 PM, Donny Davis <donny@fortnebula.com> wrote:
Why not just use the built in stuff like VDO. What benefits does ZFS bring for the use case? For most vm based workloads ZFS is the opposite of ideal over the lifecycle of a VM. COW filesystems have downsides over time.
On Thu, Nov 15, 2018 at 6:09 PM Darrell Budic <budic@onholyground.com> wrote:
I did this in the past and didn’t have any trouble with gluster/ZFS, but 4.2.x probably does more validation.
I recommend these settings on your zfs volumes, I set mine at the root(v0 here) and let them inherit:
required: v0 xattr sa local v0 acltype posixacl local optional but I recommend them: v0 relatime on local v0 compression lz4 local
For gluster, I think it checks that it’s been “optimized for virt storage”. Either apply the virt group or set the options you’ll find in /var/lib/glusterd/groups/virt.
Note that I don’t recommend the default settings for cluster.shd-max-threads & cluster.shd-wait-qlength. They can swamp your machines during heals unless you have a lot of cores and ram. You get a slightly faster heal, but often have VMs pausing for storage or other odd ball storage related errors. I prefer max-threads = 1 or maybe 2, and wait-qlength=1024 or 2048. These are per volume, so they hit harder than you think they will if you have a lot of volumes running.
Also make sure the gluster volumes themselves got set to 36.36 for owner.group, doesn’t matter for the bricks. Can do it with volume settings or mount the volume and set it manually.
Hope it helps!
-Darrell
On Nov 15, 2018, at 6:28 AM, Thomas Simmons <twsnnva@gmail.com> wrote:
Hello All,
I recently took a new job in a RedHat shop and I'd like to move all of my homelab systems to RedHat upstream products to better align with what I manage at work. I had a "custom" (aka - hacked together) 3-node Hyperconverged XenServer cluster and would like to get this moved over to Ovirt (I'm currently testing with 4.2.7). Unfortunately, my storage is limited to software RAID with a 128GB SSD for cache. If at all possible, I would prefer to use ZFS (RAIDZ+ZIL+L2ARC) instead of MD RAID + lvmcache, however I'm not able to get this working and I'm not sure why. My ZFS and Gluster configuration is working - at least where I can manually mount all of my gluster volumes from all of my nodes, however, hosted-engine --deploy fails. I understand this isn't an out of the box configuration for Ovirt, however I see no reason why this shouldn't work. I would think this would be no different than using any other Gluster volume for the engine datastore, Am I missing something that would prevent this from working?
[ INFO ] TASK [Add glusterfs storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Storage Domain target is unsupported]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage Domain target is unsupported]\". HTTP response code is 400."}
Even though it fails, it appears to have mounted and written __DIRECT_IO_TEST__ to my Gluster volume:
[root@vmh1 ~]# mount -t glusterfs localhost:/engine /mnt/engine/ [root@vmh1 ~]# ls /mnt/engine/ __DIRECT_IO_TEST__
If I cancel and try to run the deploy again, I get a different failure:
[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Error creating a storage domain]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Error creating a storage domain]\". HTTP response code is 400."}
Gluster seems ok...
[root@vmh1 /]# gluster volume info engine
Volume Name: engine Type: Replicate Volume ID: 2e34f8f5-0129-4ba5-983f-1eb5178deadc Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: vmh1-ib:/zpool1/engine Brick2: vmh2-ib:/zpool1/engine Brick3: vmh3-ib:/zpool1/engine Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off
ZFS looks good too...
[root@vmh1 ~]# ansible ovirthosts -m shell -a 'zpool status' -b vmh1 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
vmh3 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
vmh2 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
Permissions seem ok
[root@vmh1 ~]# ansible ovirthosts -m shell -a 'ls -n /zpool1' -b vmh3 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
vmh1 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
vmh2 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/SEZLAUIUSLR4UF...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZFZ7P6RFZ7XB3Y...
Hello Darrell, I can report back that after making all of the config changes you noted, deployment completed and hosted engine is up and running! Hyperconverged Ovirt 4.2 on ZFS! Thank you sir! On Thu, Nov 15, 2018 at 6:09 PM Darrell Budic <budic@onholyground.com> wrote:
I did this in the past and didn’t have any trouble with gluster/ZFS, but 4.2.x probably does more validation.
I recommend these settings on your zfs volumes, I set mine at the root(v0 here) and let them inherit:
required: v0 xattr sa local v0 acltype posixacl local optional but I recommend them: v0 relatime on local v0 compression lz4 local
For gluster, I think it checks that it’s been “optimized for virt storage”. Either apply the virt group or set the options you’ll find in /var/lib/glusterd/groups/virt.
Note that I don’t recommend the default settings for cluster.shd-max-threads & cluster.shd-wait-qlength. They can swamp your machines during heals unless you have a lot of cores and ram. You get a slightly faster heal, but often have VMs pausing for storage or other odd ball storage related errors. I prefer max-threads = 1 or maybe 2, and wait-qlength=1024 or 2048. These are per volume, so they hit harder than you think they will if you have a lot of volumes running.
Also make sure the gluster volumes themselves got set to 36.36 for owner.group, doesn’t matter for the bricks. Can do it with volume settings or mount the volume and set it manually.
Hope it helps!
-Darrell
On Nov 15, 2018, at 6:28 AM, Thomas Simmons <twsnnva@gmail.com> wrote:
Hello All,
I recently took a new job in a RedHat shop and I'd like to move all of my homelab systems to RedHat upstream products to better align with what I manage at work. I had a "custom" (aka - hacked together) 3-node Hyperconverged XenServer cluster and would like to get this moved over to Ovirt (I'm currently testing with 4.2.7). Unfortunately, my storage is limited to software RAID with a 128GB SSD for cache. If at all possible, I would prefer to use ZFS (RAIDZ+ZIL+L2ARC) instead of MD RAID + lvmcache, however I'm not able to get this working and I'm not sure why. My ZFS and Gluster configuration is working - at least where I can manually mount all of my gluster volumes from all of my nodes, however, hosted-engine --deploy fails. I understand this isn't an out of the box configuration for Ovirt, however I see no reason why this shouldn't work. I would think this would be no different than using any other Gluster volume for the engine datastore, Am I missing something that would prevent this from working?
[ INFO ] TASK [Add glusterfs storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Storage Domain target is unsupported]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage Domain target is unsupported]\". HTTP response code is 400."}
Even though it fails, it appears to have mounted and written __DIRECT_IO_TEST__ to my Gluster volume:
[root@vmh1 ~]# mount -t glusterfs localhost:/engine /mnt/engine/ [root@vmh1 ~]# ls /mnt/engine/ __DIRECT_IO_TEST__
If I cancel and try to run the deploy again, I get a different failure:
[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Error creating a storage domain]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Error creating a storage domain]\". HTTP response code is 400."}
Gluster seems ok...
[root@vmh1 /]# gluster volume info engine
Volume Name: engine Type: Replicate Volume ID: 2e34f8f5-0129-4ba5-983f-1eb5178deadc Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: vmh1-ib:/zpool1/engine Brick2: vmh2-ib:/zpool1/engine Brick3: vmh3-ib:/zpool1/engine Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off
ZFS looks good too...
[root@vmh1 ~]# ansible ovirthosts -m shell -a 'zpool status' -b vmh1 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
vmh3 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
vmh2 | CHANGED | rc=0 >> pool: zpool1 state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM zpool1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 logs sdb2 ONLINE 0 0 0 cache sdb1 ONLINE 0 0 0
errors: No known data errors
Permissions seem ok
[root@vmh1 ~]# ansible ovirthosts -m shell -a 'ls -n /zpool1' -b vmh3 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
vmh1 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
vmh2 | CHANGED | rc=0 >> total 2 drwxr-xr-x. 3 36 36 3 Nov 15 04:56 data drwxr-xr-x. 3 36 36 4 Nov 15 06:31 engine drwxr-xr-x. 3 36 36 3 Nov 15 04:56 iso
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/SEZLAUIUSLR4UF...
participants (3)
-
Darrell Budic -
Donny Davis -
Thomas Simmons