[ovirt-users] Users cannot create disks in portal

Help, banging my head against a wall. Ovirt Engine Software Version:4.4.4.7-1.el8 RHEL 8.3 I have set up my cluster going to a TrueNAS storage. Everything is going great and works well from Admin or any user with SuperUser. I would like to set it up so that students can create their own machines. Great, plowing through the documentation it looks like after creating a user, it works well by giving them at the Data Center object the user roles of: PowerUserRole VmCreator At that level, according to my reading of the documentation, it is also supposed to grant DiskCreator automatically. Well, when I do that it creates the VM but the creation of the disk does not work. Looking in the log I find: Validation of action 'AddImageFromScratch' failed for user will@internal-authz. Reasons: VAR__TYPE__STORAGE__DOMAIN,NON_ADMIN_USER_NOT_AUTHORIZED_TO_PERFORM_ACTION_ON_HE Could anyone please let me know what this is referring to? I have tried many roll combinations, and have even made my own user roll with full privileges to no avail. I looked for issues and the closest I could find is: Bug 1511697 - [RFE] Unable to set permission on all but Hosted-Engine VM and Storage Domain Which showed as resolved in 4.3 Could someone be kind enough to maybe tell me where I am missing it? I'll be the first to admit I can be a little slow at times. Thanks. For added detail: 2021-01-30 13:53:55,592-05 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (default task-26) [c0df3518-23b0-4311-9418-d9e192d9874f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[willtest4=VM_NAME]', sharedLocks=''}' 2021-01-30 13:53:55,632-05 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (default task-26) [] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 9e5b3a76-4000-11eb-82a1-00163e3be3c4 Type: ClusterAction group CREATE_VM with role type USER 2021-01-30 13:53:55,691-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-26) [] EVENT_ID: USER_ADD_VM(34), VM willtest4 was created by will@internal-authz. 2021-01-30 13:53:55,693-05 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (default task-26) [] Lock freed to object 'EngineLock:{exclusiveLocks='[willtest4=VM_NAME]', sharedLocks=''}' 2021-01-30 13:53:55,845-05 INFO [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-26) [13dfcdfc-10af-4840-9433-68d84fd05daf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[willtest4=VM_NAME]', sharedLocks='[361a430e-ef3d-4dee-bef6-256651bee6c0=VM]'}' 2021-01-30 13:53:55,866-05 INFO [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-26) [13dfcdfc-10af-4840-9433-68d84fd05daf] Running command: UpdateVmCommand internal: false. Entities affected : ID: 361a430e-ef3d-4dee-bef6-256651bee6c0 Type: VMAction group EDIT_VM_PROPERTIES with role type USER 2021-01-30 13:53:55,881-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-26) [13dfcdfc-10af-4840-9433-68d84fd05daf] EVENT_ID: USER_UPDATE_VM(35), VM willtest4 configuration was updated by will@internal-authz. 2021-01-30 13:53:55,883-05 INFO [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-26) [13dfcdfc-10af-4840-9433-68d84fd05daf] Lock freed to object 'EngineLock:{exclusiveLocks='[willtest4=VM_NAME]', sharedLocks='[361a430e-ef3d-4dee-bef6-256651bee6c0=VM]'}' 2021-01-30 13:53:56,210-05 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (default task-26) [55ce945c-425e-400a-876b-b65d4a4f2d7d] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 361a430e-ef3d-4dee-bef6-256651bee6c0 Type: VMAction group CONFIGURE_VM_NETWORK with role type USER, ID: 8501221e-bff1-487c-8db5-685422f95022 Type: VnicProfileAction group CONFIGURE_VM_NETWORK with role type USER 2021-01-30 13:53:56,232-05 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (default task-26) [7c8579df] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 361a430e-ef3d-4dee-bef6-256651bee6c0 Type: VM 2021-01-30 13:53:56,237-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-26) [7c8579df] EVENT_ID: NETWORK_ACTIVATE_VM_INTERFACE_SUCCESS(1,012), Network Interface nic1 (VirtIO) was plugged to VM willtest4. (User: will@internal-authz) 2021-01-30 13:53:56,241-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-26) [7c8579df] EVENT_ID: NETWORK_ADD_VM_INTERFACE(932), Interface nic1 (VirtIO) was added to VM willtest4. (User: will@internal-authz) 2021-01-30 13:53:56,430-05 INFO [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (default task-26) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] Lock Acquired to object 'EngineLock:{exclusiveLocks='', sharedLocks='[361a430e-ef3d-4dee-bef6-256651bee6c0=VM]'}' 2021-01-30 13:53:56,455-05 INFO [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (default task-26) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] Running command: AddDiskCommand internal: false. Entities affected : ID: 361a430e-ef3d-4dee-bef6-256651bee6c0 Type: VMAction group CONFIGURE_VM_STORAGE with role type USER, ID: 768bbdab-3a53-4341-8144-3ceb29db23c9 Type: StorageAction group CREATE_DISK with role type USER 2021-01-30 13:53:56,460-05 WARN [org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand] (default task-26) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] Validation of action 'AddImageFromScratch' failed for user will@internal-authz. Reasons: VAR__TYPE__STORAGE__DOMAIN,NON_ADMIN_USER_NOT_AUTHORIZED_TO_PERFORM_ACTION_ON_HE 2021-01-30 13:53:56,462-05 INFO [org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand] (default task-26) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] Lock freed to object 'EngineLock:{exclusiveLocks='', sharedLocks='[361a430e-ef3d-4dee-bef6-256651bee6c0=VM]'}' 2021-01-30 13:53:56,473-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-26) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] EVENT_ID: USER_FAILED_ADD_DISK_TO_VM(79), Add-Disk operation failed on VM willtest4 (User: will@internal-authz). 2021-01-30 13:53:56,475-05 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-26) [] Operation Failed: [] 2021-01-30 13:53:56,694-05 INFO [org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] Getting volume info for image '68480589-8585-41e5-a37c-9213b58fd5f6/00000000-0000-0000-0000-000000000000' 2021-01-30 13:53:56,695-05 ERROR [org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] Failed to get volume info: org.ovirt.engine.core.common.errors.EngineException: EngineException: No host was found to perform the operation (Failed with error RESOURCE_MANAGER_VDS_NOT_FOUND and code 5004) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.utils.VdsCommandsHelper.runVdsCommand(VdsCommandsHelper.java:86) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.utils.VdsCommandsHelper.runVdsCommandWithFailover(VdsCommandsHelper.java:70) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.ImagesHandler.getVolumeInfoFromVdsm(ImagesHandler.java:857) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback.childCommandsExecutionEnded(AddDiskCommandCallback.java:44) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:80) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:181) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:360) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:511) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227) 2021-01-30 13:53:56,695-05 INFO [org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] Command 'AddDisk' id: '82b5a690-8b7f-4568-9b41-0f777a461adb' child commands '[acc2d284-5dc3-49f9-b81a-d2612eb2c999]' executions were completed, status 'FAILED' 2021-01-30 13:53:56,698-05 INFO [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] Exception in invoking callback of command AddDisk (82b5a690-8b7f-4568-9b41-0f777a461adb): EngineException: EngineException: No host was found to perform the operation (Failed with error RESOURCE_MANAGER_VDS_NOT_FOUND and code 5004) 2021-01-30 13:53:56,698-05 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] Error invoking callback method 'onFailed' for 'EXECUTION_FAILED' command '82b5a690-8b7f-4568-9b41-0f777a461adb' 2021-01-30 13:53:56,698-05 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] Exception: org.ovirt.engine.core.common.errors.EngineException: EngineException: No host was found to perform the operation (Failed with error RESOURCE_MANAGER_VDS_NOT_FOUND and code 5004) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.utils.VdsCommandsHelper.runVdsCommand(VdsCommandsHelper.java:86) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.utils.VdsCommandsHelper.runVdsCommandWithFailover(VdsCommandsHelper.java:70) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.ImagesHandler.getVolumeInfoFromVdsm(ImagesHandler.java:857) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback.childCommandsExecutionEnded(AddDiskCommandCallback.java:44) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:80) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:181) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:360) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:511) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227) 2021-01-30 13:53:57,704-05 ERROR [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] Ending command 'org.ovirt.engine.core.bll.storage.disk.AddDiskCommand' with failure. 2021-01-30 13:53:57,707-05 ERROR [org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [b9d4d5c6-9f6a-4fbd-b1af-29b60d5a0e4c] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand' with failure. 2021-01-30 13:53:57,732-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [] EVENT_ID: USER_ADD_DISK_FINISHED_FAILURE(2,022), Add-Disk operation failed to complete.