Whoops, sent before I finished replying- I'll look into first trying the Nightly, to see if that has the same behavior, and then trying the _setupAuth patching et. al. to see if I can narrow down what might be happening here.


Brent Saner
SENIOR SYSTEMS ENGINEER
Follow us on LinkedIn!
brent.saner@netfire.com
855‑696‑3834 Ext. 110
www.netfire.com
From: Brent Saner <brent.saner@netfire.com>
Sent: Thursday, July 11, 2024 5:38 PM
To: users@ovirt.org <users@ovirt.org>
Subject: Re: [ovirt-users] Re: engine-setup fails: "Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute"
 

Looking into this, I can confirm that *after* the packages are installed and *before* engine-setup is run, /etc/ovirt-engine/aaa/ does not even exist:

# ls /etc/ovirt-engine/aaa
ls: cannot access '/etc/ovirt-engine/aaa': No such file or directory

Which may be expected, I'm unclear on that.

Further details inline below:


From: Yedidyah Bar David <didi@redhat.com>
Sent: Thursday, July 4, 2024 2:13 AM
To: Brent Saner <brent.saner@netfire.com>
Cc: users@ovirt.org <users@ovirt.org>
Subject: Re: [ovirt-users] Re: engine-setup fails: "Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute"
 
You don't often get email from didi@redhat.com. Learn why this is important
Caution: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.

On Thu, Jul 4, 2024 at 9:11 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Jun 19, 2024 at 10:38 PM Brent S. <brent.saner@netfire.com> wrote:
As a quick update to this:

# ovirt-aaa-jdbc-tool                                                                                                                                                                           
Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false                                                                                                                                                                                         
Jun 19, 2024 7:28:14 PM org.ovirt.engine.extension.aaa.jdbc.binding.cli.Cli main                                                                                                                                                             
SEVERE: Unexpected Exception invoking Cli: Could not read properties from: /etc/ovirt-engine/aaa/internal.properties

Which is, of course, the same message in the log.

This is probably expected, since *engine-setup never actually created the file*:

Are you sure about this?

Presumably it was removed, but I do know the directory is not present before engine-setup is run and the file is not present after.

 

# ls -la /etc/ovirt-engine/aaa
total 4
drwxr-xr-x.  2 root root    6 Jun 19 19:27 .
drwxr-xr-x. 18 root root 4096 Jun 19 19:27 ..
#

I guess you checked the above only after engine-setup failed/finished, right?

At the time, yes; see above inlines; the directory itself does not even exist before engine-setup.

 

And:

2024-06-19 19:27:10,917+0000 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc plugin.execute:923 execute-output: ['/usr/share/ovirt-engine-extension-aaa-jdbc/dbscripts/schema.sh', '-s', '[REDACTED_REMOTE_DB_HOST]', '-p', '5432', '-u', '[REDACTED_REMOTE_DB_USER]', '-d', '[REDACTED_REMOTE_DB_NAME]', '-e', 'aaa_jdbc', '-l', '/root/ovirt-engine-setup.log', '-c', 'apply'] stderr:


2024-06-19 19:27:10,917+0000 DEBUG otopi.transaction transaction._prepare:61 preparing 'File transaction for '/etc/ovirt-engine/aaa/internal.properties''
2024-06-19 19:27:10,917+0000 DEBUG otopi.filetransaction filetransaction.prepare:184 file '/etc/ovirt-engine/aaa/internal.properties' missing

Indeed

 (SNIP)

If you grep ovirt-engine sources, you'll find internal.properties in:

packaging/setup/ovirt_engine_setup/engine/constants.py:

    AAA_JDBC_CONFIG_DB = os.path.join(
        OVIRT_ENGINE_SYSCONFDIR,
        'aaa',
        'internal.properties'
    )

If you then grep for AAA_JDBC_CONFIG_DB, you see it in:

packaging/setup/plugins/ovirt-engine-setup/ovirt-engine/config/aaajdbc.py:

    def _setupAuth(self):
        self.environment[otopicons.CoreEnv.MAIN_TRANSACTION].append(
            filetransaction.FileTransaction(
                name=oenginecons.FileLocations.AAA_JDBC_CONFIG_DB,
...
                visibleButUnsafe=True,

Forgot to mention: You can check otopi sources:src/otopi/filetransaction.py
to see what this means.
 
...
    def _setupAdminUser(self):
        toolArgs = (
            oenginecons.FileLocations.AAA_JDBC_TOOL,
            '--db-config=%s' % oenginecons.FileLocations.AAA_JDBC_CONFIG_DB,
        )  
...
    @plugin.event(
        stage=plugin.Stages.STAGE_MISC,
        name=AAA_JDBC_SETUP_ADMIN_USER,
        after=(    
            oengcommcons.Stages.DB_SCHEMA,
            oengcommcons.Stages.DB_CONNECTION_AVAILABLE,
            oenginecons.Stages.CONFIG_EXTENSIONS_UPGRADE,
        ),
        before=(    
            oenginecons.Stages.CONFIG_AAA_ADMIN_USER_SETUP,
        ),
        condition=lambda self: self.environment[
            oenginecons.ConfigEnv.ADMIN_USER_AUTHZ_TYPE
        ] == self.AAA_JDBC_AUTHZ_TYPE,
    )
    def _misc(self):
        # TODO: if we knew that aaa-jdbc package was upgraded by engine-setup
        # TODO: we could display summary note that custom profiles have to be
        # TODO: upgraded manually
        self._setupSchema()
        self._setupAuth()
        self._setupAdminUser()
...

This means that:
At STAGE_MISC, _misc calls _setupAuth, which creates this file, and then it calls
_setupAdminUser which tries to use it. Latter fails, and engine-setup rolls back
the MAIN_TRANSACTION, including removing the file.

I'd start debugging this issue by:
1. Patching _setupAuth to wait (e.g. using dialog.queryBoolean, search the source
for examples) after it creates the file, so that I can investigate it
2. Patching _setupAdminUser to wait after it runs the tool, so that I can try to
investigate the failure - e.g. run it myself under strace, if the existing logging
is not enough.

You can try using the otopi plugin wait_on_error for this, instead of patching.

Good luck and best regards,
--
Didi


--
Didi