[ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ Aug. 8th 2017 ] [ 001_upgrade_engine.test_initialize_engine ]

[Yedidyah Bar David]

On Tue, Aug 8, 2017 at 12:27 PM, Yaniv Kaul <ykaul at redhat.com> wrote:
>
>
> On Tue, Aug 8, 2017 at 12:21 PM, Dafna Ron <dron at redhat.com> wrote:
>>
>> Hi,
>>
>> We see a sporadic failure in the upgrade test. from what I can see from
>> the log it is related to the firewalD package.
>>
>> Test failed: 001_upgrade_engine.test_initialize_engine
>>
>> Link to suspected patches:
>>
>> Link to Job:
>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/
>>
>> Link to all logs:
>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifact/

Downloaded all exported artifacts, unzipped, and:

>>
>>
>> Error snippet from the log:
>>
>> <error>
>>
>>
>>
>> 2017-08-08 04:09:11,006-0400 DEBUG otopi.plugins.otopi.network.firewalld
>> plugin.execute:926 execute-output: ('/bin/firewall-cmd', '--zone',
>> u'public', '--permanent', '--add-service', 'ovirt-postgres') stderr:
>> ESC[91mError: Action org.fedoraproject.FirewallD1.all is not
>> registeredESC[00m
>>
>> 2017-08-08 04:09:11,007-0400 DEBUG otopi.context
>> context._executeMethod:142 method exception
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in
>> _executeMethod
>>     method['method']()
>>   File "/usr/share/otopi/plugins/otopi/network/firewalld.py", line 334, in
>> _closeup
>>     '--add-service', service,
>>   File "/usr/lib/python2.7/site-packages/otopi/plugin.py", line 931, in
>> execute
>>     command=args[0],
>> RuntimeError: Command '/bin/firewall-cmd' failed to execute
>>
>> </error>

can't find this error.

In the console log [1] I see:

08:09:14 [upgrade-from-prevrelease-suit] Error occured, aborting
08:09:14 [upgrade-from-prevrelease-suit] Traceback (most recent call last):
08:09:14 [upgrade-from-prevrelease-suit]   File
"/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in
do_run
08:09:14 [upgrade-from-prevrelease-suit]
self.cli_plugins[args.ovirtverb].do_run(args)
08:09:14 [upgrade-from-prevrelease-suit]   File
"/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in
do_run
08:09:14 [upgrade-from-prevrelease-suit]     self._do_run(**vars(args))
08:09:14 [upgrade-from-prevrelease-suit]   File
"/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper
08:09:14 [upgrade-from-prevrelease-suit]     return func(*args, **kwargs)
08:09:14 [upgrade-from-prevrelease-suit]   File
"/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper
08:09:14 [upgrade-from-prevrelease-suit]     return func(*args,
prefix=prefix, **kwargs)
08:09:14 [upgrade-from-prevrelease-suit]   File
"/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in
do_ovirt_runtest
08:09:14 [upgrade-from-prevrelease-suit]     raise RuntimeError('Some
tests failed')
08:09:14 [upgrade-from-prevrelease-suit] RuntimeError: Some tests failed

But it's hard to understand what failed.

If I look around in the web interface, I can find this in [2].

So for some reason, downloading "all files in zip" misses many files.
Not sure why. IIRC we already had similar cases in the past.
When I press "(all files in zip)" in [3], I get a 134016 bytes
archive.zip, which only has inside it "basic-suit-master-el7" (and
"JenkinsTestedChangeList.dat"), no "upgrade-from-prevrelease-suit-master-el7"
or "upgrade-from-release-suit-master-el7". Please check.

[1] http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/consoleFull
[2] http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7/test_logs/upgrade-from-prevrelease-suite-master/post-001_upgrade_engine.py/lago-upgrade-from-prevrelease-suite-master-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-20170808040357-4iiirj.log
[3] http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifact/

>>
>>
>
> Google shows we are not the only one suffering from it - see [1] and the
> workarounds suggested.

The workaround [4] seems to have been to restart polkitd.
Do we want to do this?
Where? If in otopi, I'd consider this a somewhat ugly hack, and
would rather not, unless we have a clear and realistic reproducer.
If in CI, not sure where/when exactly.

Best,

[4] https://github.com/openshift/openshift-ansible/pull/3831/commits/457605bc7b8ba57b653eb14491f988a5565e3252

>
> Y.
> [1] https://github.com/openshift/openshift-ansible/issues/3213
>
>>
>>
>>
>>
>> _______________________________________________
>> Devel mailing list
>> Devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>
>
>
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>



-- 
Didi