[ OST Failure Report ] [ oVirt Master ] [ Aug. 8th 2017 ] [ 001_upgrade_engine.test_initialize_engine ]
4:21 a.m.
This is a multi-part message in MIME format.
--------------578D3CF4C53C01FCD1265694
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
**
*Hi, *
*
**
We see a sporadic failure in the upgrade test. from what I can see from
the log it is related to the firewalD package.
Test failed: *001_upgrade_engine.test_initialize_engine*
Link to suspected patches:
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifact/
Error snippet from the log:
<error>
*
*2017-08-08 04:09:11,006-0400 DEBUG
otopi.plugins.otopi.network.firewalld plugin.execute:926 execute-output:
('/bin/firewall-cmd', '--zone', u'public', '--permanent',
'--add-service', 'ovirt-postgres') stderr:
ESC[91mError: Action org.fedoraproject.FirewallD1.all is not
registeredESC[00m
2017-08-08 04:09:11,007-0400 DEBUG otopi.context
context._executeMethod:142 method exception
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in
_executeMethod
method['method']()
File "/usr/share/otopi/plugins/otopi/network/firewalld.py", line 334,
in _closeup
'--add-service', service,
File "/usr/lib/python2.7/site-packages/otopi/plugin.py", line 931, in
execute
command=args[0],
RuntimeError: Command '/bin/firewall-cmd' failed to execute
</error>
*
--------------578D3CF4C53C01FCD1265694
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-c11c-2c92-eac3-5aaf079f97bc">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Hi,
</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p>
</b><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-c11c-2c92-eac3-5aaf079f97bc">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">We
see a sporadic failure in the upgrade test. from what I can see from the log it is related
to the firewalD package.
</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Test
failed:
<b>001_upgrade_engine.test_initialize_engine</b></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to suspected patches: </span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to Job: <a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to all logs: <a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Error
snippet from the log: </span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p>
<br>
</b><br>
<b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-c11c-2c92-eac3-5aaf079f97bc">2017-08-08
04:09:11,006-0400 DEBUG otopi.plugins.otopi.network.firewalld
plugin.execute:926 execute-output: ('/bin/firewall-cmd',
'--zone', u'public', '--permanent',
'--add-service',
'ovirt-postgres') stderr:<br>
ESC[91mError: Action org.fedoraproject.FirewallD1.all is not
registeredESC[00m<br>
<br>
2017-08-08 04:09:11,007-0400 DEBUG otopi.context
context._executeMethod:142 method exception<br>
Traceback (most recent call last):<br>
File "/usr/lib/python2.7/site-packages/otopi/context.py", line
132, in _executeMethod<br>
method['method']()<br>
File "/usr/share/otopi/plugins/otopi/network/firewalld.py",
line 334, in _closeup<br>
'--add-service', service,<br>
File "/usr/lib/python2.7/site-packages/otopi/plugin.py", line
931, in execute<br>
command=args[0],<br>
RuntimeError: Command '/bin/firewall-cmd' failed to execute<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p>
<br>
</b></p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
</body>
</html>
--------------578D3CF4C53C01FCD1265694--
4:27 a.m.
On Tue, Aug 8, 2017 at 12:21 PM, Dafna Ron <dron(a)redhat.com> wrote:
*Hi, *
* We see a sporadic failure in the upgrade test. from what I can see from
the log it is related to the firewalD package. Test failed:
001_upgrade_engine.test_initialize_engine Link to suspected patches: Link
to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/> Link
to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifact/
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifa...
Error snippet from the log: <error> *
*2017-08-08 04:09:11,006-0400 DEBUG otopi.plugins.otopi.network.firewalld
plugin.execute:926 execute-output: ('/bin/firewall-cmd', '--zone',
u'public', '--permanent', '--add-service',
'ovirt-postgres') stderr:
ESC[91mError: Action org.fedoraproject.FirewallD1.all is not
registeredESC[00m 2017-08-08 04:09:11,007-0400 DEBUG otopi.context
context._executeMethod:142 method exception Traceback (most recent call
last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line
132, in _executeMethod method['method']() File
"/usr/share/otopi/plugins/otopi/network/firewalld.py", line 334, in
_closeup '--add-service', service, File
"/usr/lib/python2.7/site-packages/otopi/plugin.py", line 931, in execute
command=args[0], RuntimeError: Command '/bin/firewall-cmd' failed to
execute </error> *
Google shows we are not the only one suffering from it - see [1] and the
workarounds suggested.
Y.
[1] https://github.com/openshift/openshift-ansible/issues/3213
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
5:30 a.m.
On Tue, Aug 8, 2017 at 12:27 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
On Tue, Aug 8, 2017 at 12:21 PM, Dafna Ron <dron(a)redhat.com> wrote:
>
> Hi,
>
> We see a sporadic failure in the upgrade test. from what I can see from
> the log it is related to the firewalD package.
>
> Test failed: 001_upgrade_engine.test_initialize_engine
>
> Link to suspected patches:
>
> Link to Job:
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/
>
> Link to all logs:
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifact/
Downloaded all exported artifacts, unzipped, and:
>
>
> Error snippet from the log:
>
> <error>
>
>
>
> 2017-08-08 04:09:11,006-0400 DEBUG otopi.plugins.otopi.network.firewalld
> plugin.execute:926 execute-output: ('/bin/firewall-cmd', '--zone',
> u'public', '--permanent', '--add-service',
'ovirt-postgres') stderr:
> ESC[91mError: Action org.fedoraproject.FirewallD1.all is not
> registeredESC[00m
>
> 2017-08-08 04:09:11,007-0400 DEBUG otopi.context
> context._executeMethod:142 method exception
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in
> _executeMethod
> method['method']()
> File "/usr/share/otopi/plugins/otopi/network/firewalld.py", line 334, in
> _closeup
> '--add-service', service,
> File "/usr/lib/python2.7/site-packages/otopi/plugin.py", line 931, in
> execute
> command=args[0],
> RuntimeError: Command '/bin/firewall-cmd' failed to execute
>
> </error>
can't find this error.
In the console log [1] I see:
08:09:14 [upgrade-from-prevrelease-suit] Error occured, aborting
08:09:14 [upgrade-from-prevrelease-suit] Traceback (most recent call last):
08:09:14 [upgrade-from-prevrelease-suit] File
"/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in
do_run
08:09:14 [upgrade-from-prevrelease-suit]
self.cli_plugins[args.ovirtverb].do_run(args)
08:09:14 [upgrade-from-prevrelease-suit] File
"/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in
do_run
08:09:14 [upgrade-from-prevrelease-suit] self._do_run(**vars(args))
08:09:14 [upgrade-from-prevrelease-suit] File
"/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper
08:09:14 [upgrade-from-prevrelease-suit] return func(*args, **kwargs)
08:09:14 [upgrade-from-prevrelease-suit] File
"/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper
08:09:14 [upgrade-from-prevrelease-suit] return func(*args,
prefix=prefix, **kwargs)
08:09:14 [upgrade-from-prevrelease-suit] File
"/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in
do_ovirt_runtest
08:09:14 [upgrade-from-prevrelease-suit] raise RuntimeError('Some
tests failed')
08:09:14 [upgrade-from-prevrelease-suit] RuntimeError: Some tests failed
But it's hard to understand what failed.
If I look around in the web interface, I can find this in [2].
So for some reason, downloading "all files in zip" misses many files.
Not sure why. IIRC we already had similar cases in the past.
When I press "(all files in zip)" in [3], I get a 134016 bytes
archive.zip, which only has inside it "basic-suit-master-el7" (and
"JenkinsTestedChangeList.dat"), no
"upgrade-from-prevrelease-suit-master-el7"
or "upgrade-from-release-suit-master-el7". Please check.
[1] http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/consol...
[2]
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifa...
[3] http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifact/
>
>
Google shows we are not the only one suffering from it - see [1] and the
workarounds suggested.
The workaround [4] seems to have been to restart polkitd.
Do we want to do this?
Where? If in otopi, I'd consider this a somewhat ugly hack, and
would rather not, unless we have a clear and realistic reproducer.
If in CI, not sure where/when exactly.
Best,
[4]
https://github.com/openshift/openshift-ansible/pull/3831/commits/457605bc...
Y.
[1] https://github.com/openshift/openshift-ansible/issues/3213
>
>
>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
Didi
5:56 a.m.
There is something going on withe the artifacts logs, they are only
partially downloaded (I sent a mail to lago team to have a look).
Here is the link to the setup logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifa...
The error would appear in the ovirt-engine-setup-20170808040357-4iiirj.log:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifa...
Hope this helps.
On 08/08/2017 01:30 PM, Yedidyah Bar David wrote:
On Tue, Aug 8, 2017 at 12:27 PM, Yaniv Kaul <ykaul(a)redhat.com>
wrote:
>
> On Tue, Aug 8, 2017 at 12:21 PM, Dafna Ron <dron(a)redhat.com> wrote:
>> Hi,
>>
>> We see a sporadic failure in the upgrade test. from what I can see from
>> the log it is related to the firewalD package.
>>
>> Test failed: 001_upgrade_engine.test_initialize_engine
>>
>> Link to suspected patches:
>>
>> Link to Job:
>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/
>>
>> Link to all logs:
>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifact/
Downloaded all exported artifacts, unzipped, and:
>>
>> Error snippet from the log:
>>
>> <error>
>>
>>
>>
>> 2017-08-08 04:09:11,006-0400 DEBUG otopi.plugins.otopi.network.firewalld
>> plugin.execute:926 execute-output: ('/bin/firewall-cmd',
'--zone',
>> u'public', '--permanent', '--add-service',
'ovirt-postgres') stderr:
>> ESC[91mError: Action org.fedoraproject.FirewallD1.all is not
>> registeredESC[00m
>>
>> 2017-08-08 04:09:11,007-0400 DEBUG otopi.context
>> context._executeMethod:142 method exception
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132,
in
>> _executeMethod
>> method['method']()
>> File "/usr/share/otopi/plugins/otopi/network/firewalld.py", line 334,
in
>> _closeup
>> '--add-service', service,
>> File "/usr/lib/python2.7/site-packages/otopi/plugin.py", line 931,
in
>> execute
>> command=args[0],
>> RuntimeError: Command '/bin/firewall-cmd' failed to execute
>>
>> </error>
can't find this error.
In the console log [1] I see:
08:09:14 [upgrade-from-prevrelease-suit] Error occured, aborting
08:09:14 [upgrade-from-prevrelease-suit] Traceback (most recent call last):
08:09:14 [upgrade-from-prevrelease-suit] File
"/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in
do_run
08:09:14 [upgrade-from-prevrelease-suit]
self.cli_plugins[args.ovirtverb].do_run(args)
08:09:14 [upgrade-from-prevrelease-suit] File
"/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in
do_run
08:09:14 [upgrade-from-prevrelease-suit] self._do_run(**vars(args))
08:09:14 [upgrade-from-prevrelease-suit] File
"/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper
08:09:14 [upgrade-from-prevrelease-suit] return func(*args, **kwargs)
08:09:14 [upgrade-from-prevrelease-suit] File
"/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper
08:09:14 [upgrade-from-prevrelease-suit] return func(*args,
prefix=prefix, **kwargs)
08:09:14 [upgrade-from-prevrelease-suit] File
"/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in
do_ovirt_runtest
08:09:14 [upgrade-from-prevrelease-suit] raise RuntimeError('Some
tests failed')
08:09:14 [upgrade-from-prevrelease-suit] RuntimeError: Some tests failed
But it's hard to understand what failed.
If I look around in the web interface, I can find this in [2].
So for some reason, downloading "all files in zip" misses many files.
Not sure why. IIRC we already had similar cases in the past.
When I press "(all files in zip)" in [3], I get a 134016 bytes
archive.zip, which only has inside it "basic-suit-master-el7" (and
"JenkinsTestedChangeList.dat"), no
"upgrade-from-prevrelease-suit-master-el7"
or "upgrade-from-release-suit-master-el7". Please check.
[1] http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/consol...
[2]
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifa...
[3] http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1684/artifact/
>>
> Google shows we are not the only one suffering from it - see [1] and the
> workarounds suggested.
The workaround [4] seems to have been to restart polkitd.
Do we want to do this?
Where? If in otopi, I'd consider this a somewhat ugly hack, and
would rather not, unless we have a clear and realistic reproducer.
If in CI, not sure where/when exactly.
Best,
[4]
https://github.com/openshift/openshift-ansible/pull/3831/commits/457605bc...
> Y.
> [1] https://github.com/openshift/openshift-ansible/issues/3213
>
>>
>>
>>
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>
>
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
2451
days inactive
2451
days old
3 comments
3 participants
participants (3)
-
Dafna Ron -
Yaniv Kaul -
Yedidyah Bar David