[AUTOMATION BLOCKER] vdsm 3.5 build failing on Fedora 20
Hi, vdsm build is currently failing on Fedora 20: /usr/bin/pep8 --exclude="${exclude}" --filename '*.py,*.py.in' . ./tests/functional/networkTests.py:933:56: E127 continuation line over-indented for visual indent ./tests/functional/networkTests.py:1007:56: E127 continuation line over-indented for visual indent ./tests/functional/networkTests.py:1008:55: E127 continuation line over-indented for visual indent make[2]: Leaving directory `/builddir/build/BUILD/vdsm-4.16.21' make[2]: *** [check-local] Error 1 make[1]: Leaving directory `/builddir/build/BUILD/vdsm-4.16.21' make[1]: *** [check-am] Error 2 make: *** [check-recursive] Error 1 RPM build errors: error: Bad exit status from /var/tmp/rpm-tmp.wl7CZV (%check) Bad exit status from /var/tmp/rpm-tmp.wl7CZV (%check) Child return code was: 1 Please fix ASAP. -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
Hi, I keep seeing in engine.log (no another error anywhere else) this error. Does anyone knows what's wrong? Sometimes it takes days before it happens, sometimes if fails very frequently. 2015-07-03 09:12:40,056 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-10) [2428859b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM 'dev07' command failed: Policy reset 2015-07-03 09:12:40,056 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-10) [2428859b] Error: VDSGenericException: VDSNetworkException: Policy reset 2015-07-03 09:12:40,056 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-10) [2428859b] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Policy reset at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:30) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetupNetworksCommand(HostSetupNetworksCommand.java:253) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCommand(HostSetupNetworksCommand.java:171) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1212) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1356) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1979) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1392) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:375) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:459) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:441) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:397) [bll.jar:] … … 2015-07-03 09:12:40,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-10) [2428859b] Command 'PollVDSCommand(HostName = dev07, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='72456bb8-4afb-4104-b86e-ab692b4a6f2e'})' execution failed: VDSGenericException: VDSNetworkException: Policy reset thanks, mar.
Martin, This is policy reset when we perform setupNetworks. This is expected behavior but the stacktrace is confusing and we have a BZ [1] to improve it. The reason for it is that engine makes sure to keep connection alive at all times by using heartbeats. When we are about to configure network we know that there will be connectivity issue so before performing the operation we change heartbeat policy to ignore missing heartbeats. This operation enforces to reconnect using different policy and we see this nice stacktrace in the log. Thanks, Piotr [1] https://bugzilla.redhat.com/1235683 On Fri, Jul 3, 2015 at 9:17 AM, Martin Mucha <mmucha@redhat.com> wrote:
Hi,
I keep seeing in engine.log (no another error anywhere else) this error. Does anyone knows what's wrong? Sometimes it takes days before it happens, sometimes if fails very frequently.
2015-07-03 09:12:40,056 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-10) [2428859b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM 'dev07' command failed: Policy reset 2015-07-03 09:12:40,056 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-10) [2428859b] Error: VDSGenericException: VDSNetworkException: Policy reset 2015-07-03 09:12:40,056 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-10) [2428859b] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Policy reset at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:30) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetupNetworksCommand(HostSetupNetworksCommand.java:253) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCommand(HostSetupNetworksCommand.java:171) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1212) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1356) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1979) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1392) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:375) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:459) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:441) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:397) [bll.jar:] … …
2015-07-03 09:12:40,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-10) [2428859b] Command 'PollVDSCommand(HostName = dev07, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='72456bb8-4afb-4104-b86e-ab692b4a6f2e'})' execution failed: VDSGenericException: VDSNetworkException: Policy reset
thanks,
mar. _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
thanks for explanation, M. ----- Original Message -----
Martin,
This is policy reset when we perform setupNetworks. This is expected behavior but the stacktrace is confusing and we have a BZ [1] to improve it.
The reason for it is that engine makes sure to keep connection alive at all times by using heartbeats. When we are about to configure network we know that there will be connectivity issue so before performing the operation we change heartbeat policy to ignore missing heartbeats. This operation enforces to reconnect using different policy and we see this nice stacktrace in the log.
Thanks, Piotr
[1] https://bugzilla.redhat.com/1235683
On Fri, Jul 3, 2015 at 9:17 AM, Martin Mucha <mmucha@redhat.com> wrote:
Hi,
I keep seeing in engine.log (no another error anywhere else) this error. Does anyone knows what's wrong? Sometimes it takes days before it happens, sometimes if fails very frequently.
2015-07-03 09:12:40,056 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-10) [2428859b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM 'dev07' command failed: Policy reset 2015-07-03 09:12:40,056 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-10) [2428859b] Error: VDSGenericException: VDSNetworkException: Policy reset 2015-07-03 09:12:40,056 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-10) [2428859b] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Policy reset at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:30) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetupNetworksCommand(HostSetupNetworksCommand.java:253) [bll.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCommand(HostSetupNetworksCommand.java:171) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1212) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1356) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1979) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1392) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:375) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:459) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:441) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:397) [bll.jar:] … …
2015-07-03 09:12:40,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-10) [2428859b] Command 'PollVDSCommand(HostName = dev07, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='72456bb8-4afb-4104-b86e-ab692b4a6f2e'})' execution failed: VDSGenericException: VDSNetworkException: Policy reset
thanks,
mar. _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Fri, Jul 03, 2015 at 08:00:35AM +0200, Sandro Bonazzola wrote:
Hi, vdsm build is currently failing on Fedora 20:
/usr/bin/pep8 --exclude="${exclude}" --filename '*.py,*.py.in' . ./tests/functional/networkTests.py:933:56: E127 continuation line over-indented for visual indent ./tests/functional/networkTests.py:1007:56: E127 continuation line over-indented for visual indent ./tests/functional/networkTests.py:1008:55: E127 continuation line over-indented for visual indent make[2]: Leaving directory `/builddir/build/BUILD/vdsm-4.16.21' make[2]: *** [check-local] Error 1 make[1]: Leaving directory `/builddir/build/BUILD/vdsm-4.16.21' make[1]: *** [check-am] Error 2 make: *** [check-recursive] Error 1 RPM build errors: error: Bad exit status from /var/tmp/rpm-tmp.wl7CZV (%check) Bad exit status from /var/tmp/rpm-tmp.wl7CZV (%check) Child return code was: 1
Please fix ASAP.
I've posted https://gerrit.ovirt.org/#/c/43190/ to address this. Please review and help verify.
participants (4)
-
Dan Kenigsberg -
Martin Mucha -
Piotr Kliczewski -
Sandro Bonazzola