[ovirt-devel] HEADS UP: otopi - enforce priority and other changes

[Yedidyah Bar David]

Hi all,

In two days or so, we are going to merge [1].

Without it, if you have events (functions with a '@plugin.event'
decoration) that due to before=/after= requirements, otopi could not
comply with their priority=, it will continue, ignoring this and only
writing a message to the console if you set OTOPI_DEBUG=1.

With this change, it will fail.

Under normal circumstances, you'll see nothing in the console or the
log, just an exit code of 1.

If you set an environment variable OTOPI_DEBUG=1, it will emit a
"RuntimeError: Priorities were reversed during buildSequence" error,
see examples below.

You can restore the existing behavior by setting another env var
OTOPI_ALLOW_PRIORITY_OVERRIDE=1 .

Soon thereafter, we are going to merge [2]. This will replace the
algorithm that is used to decide what to run when. In principle, after
[1] is merged, they should be compatible. However, the new algorithm
does not allow overriding priorities. To revert to the old algorithm,
you can set env var OTOPI_ALLOW_LEGACY_BUILDSEQ=1 . The old algorithm
(and this option) is intended to be removed eventually, probably after
oVirt 4.0 is released.

Patches to make engine-setup (engine+dwh+reports) and host-deploy
compatible with the new behavior were already merged to master
branches. Still we most likely forgot some other tools/flows/etc., so
if you encounter such failures please open a bug against the relevant
tool.

On a more positive (?) side, we also recently added a new package to
master, called otopi-debug-plugins. It currently includes two plugins,
which are not enabled by default. One of them allows you to get a
prompt 'Press Enter to continue' on [ERROR], so it's easier to have a
look at the system before the tool might try to rollback or something,
the other allows you to enforce a failure at a certain stage, which
might be useful to debug/verify failures and especially rollbacks.
More details about these are in a new README file [3].

[1] https://gerrit.ovirt.org/55329
[2] https://gerrit.ovirt.org/28775
[3] https://gerrit.ovirt.org/gitweb?p=otopi.git;a=blob;f=README.debug-plugins

Examples:

1.
# ovirt-host-deploy
# echo $?
1

2.
# OTOPI_DEBUG=1 ovirt-host-deploy 2>&1 | tail
Priorities were reversed during buildSequence: method <bound method
?._init of <otopi.plugins.otopi.core.log.Plugin object at
0x7f9827c9fe10>> with priority 2000 appears after method <bound method
?._preinit of <otopi.plugins.ovirt_host_deploy.core.misc.Plugin object
at 0x1032990>> with priority 5000
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/__main__.py", line 88, in main
    installer.execute()
  File "/usr/lib/python2.7/site-packages/otopi/main.py", line 151, in execute
    self.context.buildSequence()
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 400,
in buildSequence
    raise RuntimeError(msg)
RuntimeError: Priorities were reversed during buildSequence: method
<bound method ?._init of <otopi.plugins.otopi.core.log.Plugin object
at 0x7f9827c9fe10>> with priority 2000 appears after method <bound
method ?._preinit of <otopi.plugins.ovirt_host_deploy.core.misc.Plugin
object at 0x1032990>> with priority 5000
('FATAL Internal error (main): Priorities were reversed during
buildSequence: method <bound method ?._init of
<otopi.plugins.otopi.core.log.Plugin object at 0x7f9827c9fe10>> with
priority 2000 appears after method <bound method ?._preinit of
<otopi.plugins.ovirt_host_deploy.core.misc.Plugin object at
0x1032990>> with priority 5000',)

3.
# OTOPI_DEBUG=1 OTOPI_ALLOW_PRIORITY_OVERRIDE=1 ovirt-host-deploy
...
Priorities were reversed during buildSequence: method <bound method
?._init of <otopi.plugins.otopi.core.log.Plugin object at
0x7fbe17f51e10>> with priority 2000 appears after method <bound method
?._preinit of <otopi.plugins.ovirt_host_deploy.core.misc.Plugin object
at 0x107f8d0>> with priority 5000
[ INFO  ] Stage: Initializing
          Continuing will configure this host for serving as
hypervisor. Are you sure you want to continue? (yes/no)

And with [2]:

4.
# OTOPI_DEBUG=1 ovirt-host-deploy
...
_toposortBuildSequence failed: Cyclic dependencies found
('FATAL Internal error (main): Cyclic dependencies found',)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/__main__.py", line 88, in main
    installer.execute()
  File "/usr/lib/python2.7/site-packages/otopi/main.py", line 151, in execute
    self.context.buildSequence()
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 665,
in buildSequence
    self._sequence = self._toposortBuildSequence()
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 624,
in _toposortBuildSequence
    raise RuntimeError(_('Cyclic dependencies found'))
RuntimeError: Cyclic dependencies found

5.
# OTOPI_DEBUG=1 OTOPI_ALLOW_PRIORITY_OVERRIDE=1 ovirt-host-deploy
(fails the same way as (4.))

6.
# OTOPI_ALLOW_LEGACY_BUILDSEQ=1 OTOPI_DEBUG=1 ovirt-host-deploy
(fails similarly to (2.) - you'll see more-or-less outputs of (4.)
then (2.) combined)

7.
# OTOPI_ALLOW_LEGACY_BUILDSEQ=1 OTOPI_DEBUG=1
OTOPI_ALLOW_PRIORITY_OVERRIDE=1 ovirt-host-deploy
(succeeds like (3.), but with part of the output of (4.) before that)

Best regards,
-- 
Didi