On Tue, Jan 15, 2019 at 11:38 AM Dafna Ron <dron@redhat.com> wrote:
>
> The last ovirt-engine package ran on Jan 11th and this failure happened on the 14th so the appliance package is suppose to be newer then the engine and should be running fine.
> if this is not the case, then I suggest that the maintainers of the project see what is needed to be updated and trigger a new package build (which is why I was alerting to the failure)
>
> Thanks,
> Dafna
>
>
> On Tue, Jan 15, 2019 at 6:51 AM Yedidyah Bar David <didi@redhat.com> wrote:
>>
>> On Mon, Jan 14, 2019 at 2:15 PM Dafna Ron <dron@redhat.com> wrote:
>> >
>> > Hi,
>> >
>> > we are failing test 001_upgrade_engine.test_initialize_engine on ovirt 4.2 for project ovirt-ansible-hosted-engine-setup.
>> >
>> > Patch reported by CQ:
>> >
>> > https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/commit/860f6a69d38719de663cb8ade13eb8bdff2d1e6f - Rename the role - replacing '-' with '_'
>> >
>> > Ido, can you please check this issue?
>> >
>> > failed build:
>> >
>> > http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3712/

This build has a single failure:

https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3712/testReport/junit/(root)/001_upgrade_engine/running_tests___upgrade_from_release_suite_el7_x86_64___test_initialize_engine/

which is in the upgrade_from_release suite, not hosted-engine.

However, I looked at the logs, and the root cause indeed seems to be
related to hosted-engine, as the CQ identified (Nice!), which is:

https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3712/artifact/upgrade-from-release-suite.el7.x86_64/lago_logs/lago.log/*view*/

2019-01-14 10:38:38,168::ssh.py::ssh::58::lago.ssh::DEBUG::Running
8d4833e8 on lago-upgrade-from-release-suite-4-2-engine: yum -y update
ovirt-*setup*
2019-01-14 10:38:39,732::ssh.py::ssh::81::lago.ssh::DEBUG::Command
8d4833e8 on lago-upgrade-from-release-suite-4-2-engine returned with 1
2019-01-14 10:38:39,733::ssh.py::ssh::89::lago.ssh::DEBUG::Command
8d4833e8 on lago-upgrade-from-release-suite-4-2-engine output:
 Loaded plugins: fastestmirror, versionlock
Determining fastest mirrors
Excluding 9 updates due to versionlock (use "yum versionlock status"
to show them)
Resolving Dependencies
--> Running transaction check
---> Package ovirt-ansible-engine-setup.noarch 0:1.1.5-1.el7 will be updated
---> Package ovirt-ansible-engine-setup.noarch
0:1.1.7-0.1.master.20181219093302.el7 will be an update
---> Package ovirt-ansible-hosted-engine-setup.noarch 0:1.0.2-1.el7
will be updated
---> Package ovirt-ansible-hosted-engine-setup.noarch
0:1.0.6-0.1.master.20190114102323.el7 will be an update
---> Package ovirt-engine-dwh-setup.noarch 0:4.2.4.3-1.el7 will be updated
---> Package ovirt-engine-dwh-setup.noarch
0:4.2.5-0.0.master.20180708055101.el7 will be an update
---> Package ovirt-engine-extension-aaa-ldap-setup.noarch
0:1.3.8-1.el7 will be updated
---> Package ovirt-engine-extension-aaa-ldap-setup.noarch
0:1.3.9-0.0.master.gitc39177c.el7 will be an update
--> Processing Dependency: ovirt-engine-extension-aaa-ldap =
1.3.9-0.0.master.gitc39177c.el7 for package:
ovirt-engine-extension-aaa-ldap-setup-1.3.9-0.0.master.gitc39177c.el7.noarch
---> Package ovirt-engine-setup.noarch 0:4.2.7.5-1.el7 will be updated
---> Package ovirt-engine-setup.noarch
0:4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 will be an update
---> Package ovirt-engine-setup-base.noarch 0:4.2.7.5-1.el7 will be updated
---> Package ovirt-engine-setup-base.noarch
0:4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 will be an update
--> Processing Dependency: ovirt-engine-lib >=
4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 for package:
ovirt-engine-setup-base-4.2.8.3-0.0.master.20190111080213.git7965c2e.el7.noarch
---> Package ovirt-engine-setup-plugin-ovirt-engine.noarch
0:4.2.7.5-1.el7 will be updated
---> Package ovirt-engine-setup-plugin-ovirt-engine.noarch
0:4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 will be an update
---> Package ovirt-engine-setup-plugin-ovirt-engine-common.noarch
0:4.2.7.5-1.el7 will be updated
---> Package ovirt-engine-setup-plugin-ovirt-engine-common.noarch
0:4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 will be an update
---> Package ovirt-engine-setup-plugin-vmconsole-proxy-helper.noarch
0:4.2.7.5-1.el7 will be updated
---> Package ovirt-engine-setup-plugin-vmconsole-proxy-helper.noarch
0:4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 will be an update
---> Package ovirt-engine-setup-plugin-websocket-proxy.noarch
0:4.2.7.5-1.el7 will be updated
---> Package ovirt-engine-setup-plugin-websocket-proxy.noarch
0:4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 will be an update
---> Package ovirt-imageio-proxy-setup.noarch 0:1.4.5-0.el7 will be updated
---> Package ovirt-imageio-proxy-setup.noarch 0:1.4.6-1.el7 will be an update
---> Package ovirt-setup-lib.noarch 0:1.1.5-1.el7 will be updated
---> Package ovirt-setup-lib.noarch
0:1.1.6-0.0.master.20180921125403.git90612e6.el7 will be an update
--> Running transaction check
---> Package ovirt-engine-extension-aaa-ldap.noarch 0:1.3.8-1.el7 will
be updated
---> Package ovirt-engine-extension-aaa-ldap.noarch
0:1.3.9-0.0.master.gitc39177c.el7 will be an update
---> Package ovirt-engine-lib.noarch 0:4.2.7.5-1.el7 will be updated
---> Package ovirt-engine-lib.noarch
0:4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package
       Arch   Version                                          Repository  Size
================================================================================
Updating:
 ovirt-ansible-engine-setup
       noarch 1.1.7-0.1.master.20181219093302.el7              alocalsync  15 k
 ovirt-ansible-hosted-engine-setup
       noarch 1.0.6-0.1.master.20190114102323.el7              alocalsync  45 k
 ovirt-engine-dwh-setup
       noarch 4.2.5-0.0.master.20180708055101.el7              alocalsync  78 k
 ovirt-engine-extension-aaa-ldap-setup
       noarch 1.3.9-0.0.master.gitc39177c.el7                  alocalsync  37 k
 ovirt-engine-setup
       noarch 4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 alocalsync  11 k
 ovirt-engine-setup-base
       noarch 4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 alocalsync 104 k
 ovirt-engine-setup-plugin-ovirt-engine
       noarch 4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 alocalsync 179 k
 ovirt-engine-setup-plugin-ovirt-engine-common
       noarch 4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 alocalsync 109 k
 ovirt-engine-setup-plugin-vmconsole-proxy-helper
       noarch 4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 alocalsync  30 k
 ovirt-engine-setup-plugin-websocket-proxy
       noarch 4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 alocalsync  28 k
 ovirt-imageio-proxy-setup
       noarch 1.4.6-1.el7                                      alocalsync  23 k
 ovirt-setup-lib
       noarch 1.1.6-0.0.master.20180921125403.git90612e6.el7   alocalsync  19 k
Updating for dependencies:
 ovirt-engine-extension-aaa-ldap
       noarch 1.3.9-0.0.master.gitc39177c.el7                  alocalsync 117 k
 ovirt-engine-lib
       noarch 4.2.8.3-0.0.master.20190111080213.git7965c2e.el7 alocalsync  31 k

Transaction Summary
================================================================================
Upgrade  12 Packages (+2 Dependent packages)

Total download size: 827 k
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
--------------------------------------------------------------------------------
Total                                              8.7 MB/s | 827 kB  00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction

2019-01-14 10:38:39,733::ssh.py::ssh::96::lago.ssh::DEBUG::Command
8d4833e8 on lago-upgrade-from-release-suite-4-2-engine  errors:
 Transaction couldn't start:
file /usr/share/ansible/roles/oVirt.hosted-engine-setup from install
of ovirt-ansible-hosted-engine-setup-1.0.6-0.1.master.20190114102323.el7.noarch
conflicts with file from package
ovirt-ansible-hosted-engine-setup-1.0.2-1.el7.noarch

So:

1. Yum failed to update ovirt-ansible-hosted-engine-setup, likely
because we (tried to) rename stuff there. Seems like we
tried handling it, based on the comment in the spec file, but perhaps
this didn't work. Ido? Adding also Simone.

This comes from here:
https://docs.fedoraproject.org/en-US/packaging-guidelines/Directory_Replacement/

"Due to a known limitation with RPM, it is not possible to replace a directory with any kind of file or symlink, nor is it possible to replace a symlink to a directory with a directory, without RPM producing file conflict errors while trying to install the package. For more information on the issues involved, refer to bug 447156 and bug 646523."

Due to galaxy 3.0 prescriptions, we should rename the role from from oVirt.hosted-engine-setup in ovirt.hosted_engine_setup and we are trying to keep a symlink there for backward compatibility.
The point is that rpm doesn't allow to replace a directory with a symlink during the upgrade; we should have a workaround in place but this is probably not enough.

Let's check what we can do on that.

2. We should probably patch OST to fail right after the failed
yum update, no need to run also engine-setup and confuse people...
Pushed this: https://gerrit.ovirt.org/96906 .

Best regards,

>> >
>> > error:
>> >
>> > [ INFO  ] Checking for an update for Setup...
>> >           An update for the Setup packages ovirt-engine-setup ovirt-engine-setup-plugin-websocket-proxy ovirt-engine-dwh-setup was found. Please update that package by running:
>> >           "yum update ovirt-engine-setup ovirt-engine-setup-plugin-websocket-proxy ovirt-engine-dwh-setup"
>> >           and then execute Setup again.
>> > [ ERROR ] Failed to execute stage 'Environment customization': Please update the Setup packages
>> > [ INFO  ] Stage: Clean up
>> >           Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20190114053840-4ysu6y.log
>> > [ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20190114053855-setup.conf'
>> > [ INFO  ] Stage: Pre-termination
>> > [ INFO  ] Stage: Termination
>> > [ ERROR ] Execution of setup failed
>> > ('FATAL Internal error (main): Please update the Setup packages',)
>> >
>> > lago.ssh: DEBUG: Command 8e6b18d0 on lago-upgrade-from-release-suite-4-2-engine  errors:
>> >  Traceback (most recent call last):
>> >   File "/usr/lib/python2.7/site-packages/otopi/__main__.py", line 88, in main
>> >     installer.execute()
>> >   File "/usr/lib/python2.7/site-packages/otopi/main.py", line 157, in execute
>> >     self.context.runSequence()
>> >   File "/usr/lib/python2.7/site-packages/otopi/context.py", line 771, in runSequence
>> >     util.raiseExceptionInformation(infos[0])
>> >   File "/usr/lib/python2.7/site-packages/otopi/util.py", line 81, in raiseExceptionInformation
>> >     exec('raise info[1], None, info[2]')
>> >   File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod
>> >     method['method']()
>> >   File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine-common/distro-rpm/packages.py", line 410, in _customization
>> >     raise RuntimeError(_('Please update the Setup packages'))
>> > RuntimeError: Please update the Setup packages
>>
>> I think this is self-explanatory, no?
>>
>> Most likely this happens because we have a new engine and the appliance
>> is not updated. If it's just a timing issue, should be solved without
>> intervention on next appliance build. You (CI) might consider, though,
>> forcing an appliance build and using it in each CQ batch that updates
>> the engine.
>>
>> If it's something else, need to check appliance build status.
>>
>> Best regards,
>> --
>> Didi



--
Didi