Whoops, sent before I finished replying- I'll look into first trying the Nightly, to see if that has the same behavior, and then trying the _setupAuth patching et. al. to see if I can narrow down what might be happening here. Brent Saner SENIOR SYSTEMS ENGINEER Follow us on LinkedIn! brent.saner@netfire.com 855-696-3834 Ext. 110 www.netfire.com ________________________________ From: Brent Saner <brent.saner@netfire.com> Sent: Thursday, July 11, 2024 5:38 PM To: users@ovirt.org <users@ovirt.org> Subject: Re: [ovirt-users] Re: engine-setup fails: "Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute" Looking into this, I can confirm that *after* the packages are installed and *before* engine-setup is run, /etc/ovirt-engine/aaa/ does not even exist: # ls /etc/ovirt-engine/aaa ls: cannot access '/etc/ovirt-engine/aaa': No such file or directory Which may be expected, I'm unclear on that. Further details inline below: ________________________________ From: Yedidyah Bar David <didi@redhat.com> Sent: Thursday, July 4, 2024 2:13 AM To: Brent Saner <brent.saner@netfire.com> Cc: users@ovirt.org <users@ovirt.org> Subject: Re: [ovirt-users] Re: engine-setup fails: "Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute" You don't often get email from didi@redhat.com. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Caution: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Thu, Jul 4, 2024 at 9:11 AM Yedidyah Bar David <didi@redhat.com<mailto:didi@redhat.com>> wrote: On Wed, Jun 19, 2024 at 10:38 PM Brent S. <brent.saner@netfire.com<mailto:brent.saner@netfire.com>> wrote: As a quick update to this: # ovirt-aaa-jdbc-tool Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false Jun 19, 2024 7:28:14 PM org.ovirt.engine.extension.aaa.jdbc.binding.cli.Cli main SEVERE: Unexpected Exception invoking Cli: Could not read properties from: /etc/ovirt-engine/aaa/internal.properties Which is, of course, the same message in the log. This is probably expected, since *engine-setup never actually created the file*: Are you sure about this? Presumably it was removed, but I do know the directory is not present before engine-setup is run and the file is not present after. # ls -la /etc/ovirt-engine/aaa total 4 drwxr-xr-x. 2 root root 6 Jun 19 19:27 . drwxr-xr-x. 18 root root 4096 Jun 19 19:27 .. # I guess you checked the above only after engine-setup failed/finished, right? At the time, yes; see above inlines; the directory itself does not even exist before engine-setup. And: 2024-06-19 19:27:10,917+0000 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc plugin.execute:923 execute-output: ['/usr/share/ovirt-engine-extension-aaa-jdbc/dbscripts/schema.sh', '-s', '[REDACTED_REMOTE_DB_HOST]', '-p', '5432', '-u', '[REDACTED_REMOTE_DB_USER]', '-d', '[REDACTED_REMOTE_DB_NAME]', '-e', 'aaa_jdbc', '-l', '/root/ovirt-engine-setup.log', '-c', 'apply'] stderr: 2024-06-19 19:27:10,917+0000 DEBUG otopi.transaction transaction._prepare:61 preparing 'File transaction for '/etc/ovirt-engine/aaa/internal.properties'' 2024-06-19 19:27:10,917+0000 DEBUG otopi.filetransaction filetransaction.prepare:184 file '/etc/ovirt-engine/aaa/internal.properties' missing Indeed (SNIP) If you grep ovirt-engine sources, you'll find internal.properties in: packaging/setup/ovirt_engine_setup/engine/constants.py: AAA_JDBC_CONFIG_DB = os.path.join( OVIRT_ENGINE_SYSCONFDIR, 'aaa', 'internal.properties' ) If you then grep for AAA_JDBC_CONFIG_DB, you see it in: packaging/setup/plugins/ovirt-engine-setup/ovirt-engine/config/aaajdbc.py: def _setupAuth(self): self.environment[otopicons.CoreEnv.MAIN_TRANSACTION].append( filetransaction.FileTransaction( name=oenginecons.FileLocations.AAA_JDBC_CONFIG_DB, ... visibleButUnsafe=True, Forgot to mention: You can check otopi sources:src/otopi/filetransaction.py to see what this means. ... def _setupAdminUser(self): toolArgs = ( oenginecons.FileLocations.AAA_JDBC_TOOL, '--db-config=%s' % oenginecons.FileLocations.AAA_JDBC_CONFIG_DB, ) ... @plugin.event( stage=plugin.Stages.STAGE_MISC, name=AAA_JDBC_SETUP_ADMIN_USER, after=( oengcommcons.Stages.DB_SCHEMA, oengcommcons.Stages.DB_CONNECTION_AVAILABLE, oenginecons.Stages.CONFIG_EXTENSIONS_UPGRADE, ), before=( oenginecons.Stages.CONFIG_AAA_ADMIN_USER_SETUP, ), condition=lambda self: self.environment[ oenginecons.ConfigEnv.ADMIN_USER_AUTHZ_TYPE ] == self.AAA_JDBC_AUTHZ_TYPE, ) def _misc(self): # TODO: if we knew that aaa-jdbc package was upgraded by engine-setup # TODO: we could display summary note that custom profiles have to be # TODO: upgraded manually self._setupSchema() self._setupAuth() self._setupAdminUser() ... This means that: At STAGE_MISC, _misc calls _setupAuth, which creates this file, and then it calls _setupAdminUser which tries to use it. Latter fails, and engine-setup rolls back the MAIN_TRANSACTION, including removing the file. I'd start debugging this issue by: 1. Patching _setupAuth to wait (e.g. using dialog.queryBoolean, search the source for examples) after it creates the file, so that I can investigate it 2. Patching _setupAdminUser to wait after it runs the tool, so that I can try to investigate the failure - e.g. run it myself under strace, if the existing logging is not enough. You can try using the otopi plugin wait_on_error for this, instead of patching. Good luck and best regards, -- Didi -- Didi