Whoops, sent before I finished replying- I'll look into first trying the Nightly, to see if that has the same behavior, and then trying the _setupAuth patching et. al. to see if I can narrow down what might be happening here.
From: Brent Saner <brent.saner@netfire.com>
Sent: Thursday, July 11, 2024 5:38 PM
To: users@ovirt.org <users@ovirt.org>
Subject: Re: [ovirt-users] Re: engine-setup fails: "Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute"
Looking into this, I can confirm that *after* the packages are installed and *before* engine-setup is run, /etc/ovirt-engine/aaa/ does not even exist:
# ls /etc/ovirt-engine/aaa
ls: cannot access '/etc/ovirt-engine/aaa': No such file or directory
Which may be expected, I'm unclear on that.
Further details inline below:
From: Yedidyah Bar David <didi@redhat.com>
Sent: Thursday, July 4, 2024 2:13 AM
To: Brent Saner <brent.saner@netfire.com>
Cc: users@ovirt.org <users@ovirt.org>
Subject: Re: [ovirt-users] Re: engine-setup fails: "Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute"
|
Caution: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
|
As a quick update to this:
# ovirt-aaa-jdbc-tool
Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false
Jun 19, 2024 7:28:14 PM org.ovirt.engine.extension.aaa.jdbc.binding.cli.Cli main
SEVERE: Unexpected Exception invoking Cli: Could not read properties from: /etc/ovirt-engine/aaa/internal.properties
Which is, of course, the same message in the log.
This is probably expected, since *engine-setup never actually created the file*:
Are you sure about this?
Presumably it was removed, but I do know the directory is not present before engine-setup is run and the file is not present after.
# ls -la /etc/ovirt-engine/aaa
total 4
drwxr-xr-x. 2 root root 6 Jun 19 19:27 .
drwxr-xr-x. 18 root root 4096 Jun 19 19:27 ..
#
I guess you checked the above only after engine-setup failed/finished, right?
At the time, yes; see above inlines; the directory itself does not even exist before engine-setup.
And:
2024-06-19 19:27:10,917+0000 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc plugin.execute:923 execute-output: ['/usr/share/ovirt-engine-extension-aaa-jdbc/dbscripts/schema.sh', '-s', '[REDACTED_REMOTE_DB_HOST]', '-p', '5432', '-u', '[REDACTED_REMOTE_DB_USER]',
'-d', '[REDACTED_REMOTE_DB_NAME]', '-e', 'aaa_jdbc', '-l', '/root/ovirt-engine-setup.log', '-c', 'apply'] stderr:
2024-06-19 19:27:10,917+0000 DEBUG otopi.transaction transaction._prepare:61 preparing 'File transaction for '/etc/ovirt-engine/aaa/internal.properties''
2024-06-19 19:27:10,917+0000 DEBUG otopi.filetransaction filetransaction.prepare:184 file '/etc/ovirt-engine/aaa/internal.properties' missing
Indeed
(SNIP)
If you grep ovirt-engine sources, you'll find internal.properties in:
packaging/setup/ovirt_engine_setup/engine/constants.py:
AAA_JDBC_CONFIG_DB = os.path.join(
OVIRT_ENGINE_SYSCONFDIR,
'aaa',
'internal.properties'
)
If you then grep for AAA_JDBC_CONFIG_DB, you see it in:
packaging/setup/plugins/ovirt-engine-setup/ovirt-engine/config/aaajdbc.py:
def _setupAuth(self):
self.environment[otopicons.CoreEnv.MAIN_TRANSACTION].append(
filetransaction.FileTransaction(
name=oenginecons.FileLocations.AAA_JDBC_CONFIG_DB,
...
visibleButUnsafe=True,
Forgot to mention: You can check otopi sources:src/otopi/filetransaction.py
to see what this means.
...
def _setupAdminUser(self):
toolArgs = (
oenginecons.FileLocations.AAA_JDBC_TOOL,
'--db-config=%s' % oenginecons.FileLocations.AAA_JDBC_CONFIG_DB,
)
...
@plugin.event(
stage=plugin.Stages.STAGE_MISC,
name=AAA_JDBC_SETUP_ADMIN_USER,
after=(
oengcommcons.Stages.DB_SCHEMA,
oengcommcons.Stages.DB_CONNECTION_AVAILABLE,
oenginecons.Stages.CONFIG_EXTENSIONS_UPGRADE,
),
before=(
oenginecons.Stages.CONFIG_AAA_ADMIN_USER_SETUP,
),
condition=lambda self: self.environment[
oenginecons.ConfigEnv.ADMIN_USER_AUTHZ_TYPE
] == self.AAA_JDBC_AUTHZ_TYPE,
)
def _misc(self):
# TODO: if we knew that aaa-jdbc package was upgraded by engine-setup
# TODO: we could display summary note that custom profiles have to be
# TODO: upgraded manually
self._setupSchema()
self._setupAuth()
self._setupAdminUser()
...
This means that:
At STAGE_MISC, _misc calls _setupAuth, which creates this file, and then it calls
_setupAdminUser which tries to use it. Latter fails, and engine-setup rolls back
the MAIN_TRANSACTION, including removing the file.
I'd start debugging this issue by:
1. Patching _setupAuth to wait (e.g. using dialog.queryBoolean, search the source
for examples) after it creates the file, so that I can investigate it
2. Patching _setupAdminUser to wait after it runs the tool, so that I can try to
investigate the failure - e.g. run it myself under strace, if the existing logging
is not enough.
You can try using the otopi plugin wait_on_error for this, instead of patching.
Good luck and best regards,
--
Didi
--
Didi
