November 2017 - Devel - oVirt List Archives

Where is subprocess32 for EL7?
by Nir Soffer 17 Nov '17

17 Nov '17

I'm trying to update ioprocess to use subprocess32 instead of cpopen. The build fail on el7-* http://jenkins.ovirt.org/job/ioprocess_master_check-patch-el7-x86_64/262/co… http://jenkins.ovirt.org/job/ioprocess_master_check-patch-el7-ppc64le/32/ I think I'm missing a secret repository. Who can tell me where subprocess32 is hiding? Nir

3 4

sos 3.5 has been released
by Sandro Bonazzola 17 Nov '17

17 Nov '17

Hi, sos 3.5 has been released upstream. I packaged it for Fedora >= 26 (and waiting karma[fc26][fc27]) being 25 almost dead. Is there interest in packaging it for CentOS Virt SIG considering CentOS 7.4 is still shipping sos 3.4? Among the improvements in sos 3.5 there's support for PostgreSQL from SCL, the same we are using in oVirt 4.2. Or are we fine waiting until sos 3.5 will be packaged and shipped within RHEL? [fc26] https://bodhi.fedoraproject.org/updates/FEDORA-2017-bef483c3a0 [fc27] https://bodhi.fedoraproject.org/updates/FEDORA-2017-4bd9528c50 Thanks, -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>

1 0

Adding new features to ovirt.org
by John Marks 16 Nov '17

16 Nov '17

Hi, Thanks to everyone who helped with the feature categorization effort on the ovirt feature page <https://ovirt.org/develop/release-management/features/>. We've made significant progress, and thanks also to Eldan's UX work, the page is looking much better organized. There are a *few simple things* you can do to help keep the page organized, and to improve it. 1. Adding a new feature to ovirt.org: Please add your feature to a suitable category: - Gluster (The location in the repository is: /develop/release-management/features/gluster) - Infra (Same pattern as above) - Integration - Metrics - Network - Node - SLA - Storage - UX - VDSM - Virt 2. ‎To help us all find which release each feature is associated with, please add a link to your feature under *Features by Release <https://ovirt.org/develop/release-management/features/#features-by-release>. * The folder location in the source repository is under: source/develop/release-management/releases/x.x/feature.html.md x.x = release version 3. We want to continue to list all *existing features* under the correct category and under Features by Release <https://ovirt.org/develop/release-management/features/#features-by-release> . You can help by visiting this spreadsheet and completing the missing feature category and release version info: https://docs.google.com/spreadsheets/d/1c7Di6xz0N0j0rPFRzogBhErJj7rnTCatAZe… *Legend* Green*: *The feature is listed under the correct category and release *Brown**: *The feature needs to be moved to a different category / link needs to be added to the feature under the relevant release. Let me know if you have any questions. Thanks! John -- John Marks Community Manager & Technical Writer, oVirt redhat Israel Cell: +972 52 8644 491

4 6

[ OST Failure Report ] [ oVirt master ] [ 16-11-2017 ] [ 002_bootstrap.add_dc ]
by Dafna Ron 16 Nov '17

16 Nov '17

This is a multi-part message in MIME format. --------------D85A420919595D580C076AD7 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, we have a CQ failure in add_dc test. I can see at engine long that we have an issue with 'extension.aaa.jdbc.binding.api.AuthnExtension' I don't think that the patch reported was the actual cause of the failure. Yesterday we were failing on missing package ovirt-cockpit-sso which we added to our repos. and I think perhaps we now exposed an issue with sso support which was undiscovered before. ** *Link to suspected patches: * ** These are the reported patches: https://gerrit.ovirt.org/#/c/83790/ https://gerrit.ovirt.org/#/c/83779/ * ** Here is the original patch for the sso support: https://gerrit.ovirt.org/#/c/82997/ Link to Job: *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/ ** *Link to all logs:* *** http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/artifact/ (Relevant) error snippet from the log: <error> * 2017-11-15 16:50:07,115-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Loading extension 'internal-authn' 2017-11-15 16:50:07,118-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Extension 'internal-authn' loaded 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] Config BEGIN 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.aaa.authn.profile.name: internal 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.provides: org.ovirt.engine.api.extensions.aaa.Authn 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.class: org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.aaa.authn.authz.plugin: internal-authz 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.bindings.method: jbossmodule 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] config.datasource.file: /etc/ovirt-engine/aaa/internal.properties 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.name: internal-authn 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.module: org.ovirt.engine.extension.aaa.jdbc 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] Config END 2017-11-15 16:50:07,119-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Loading extension 'internal-authz' 2017-11-15 16:50:07,120-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Extension 'internal-authz' loaded 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] Config BEGIN 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.provides: org.ovirt.engine.api.extensions.aaa.Authz 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.class: org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthzExtension 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.bindings.method: jbossmodule 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] config.datasource.file: /etc/ovirt-engine/aaa/internal.properties 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.name: internal-authz 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.module: org.ovirt.engine.extension.aaa.jdbc 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] Config END 2017-11-15 16:50:07,121-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Initializing extension 'internal-authn' 2017-11-15 16:50:07,182-05 ERROR [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (ServerService Thread Pool -- 45) [] Unexpected Exception invoking: EXTENSION_INITIALIZE[e5ae1b7f-9104-4f23-a444-7b9175ff68d2] 2017-11-15 16:50:07,182-05 DEBUG [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (ServerService Thread Pool -- 45) [] Exception:: org.postgresql.util.PSQLException: FATAL: the database system is shutting down at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398) [postgresql.jar:] at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173) [postgresql.jar:] at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) [postgresql.jar:] at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136) [postgresql.jar:] at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) [postgresql.jar:] at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) [postgresql.jar:] at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) [postgresql.jar:] at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) [postgresql.jar:] at org.postgresql.Driver.makeConnection(Driver.java:393) [postgresql.jar:] at org.postgresql.Driver.connect(Driver.java:267) [postgresql.jar:] at java.sql.DriverManager.getConnection(DriverManager.java:664) [rt.jar:1.8.0_151] at java.sql.DriverManager.getConnection(DriverManager.java:247) [rt.jar:1.8.0_151] at org.ovirt.engine.extension.aaa.jdbc.core.datasource.DataSourceProvider$1.invoke(DataSourceProvider.java:86) [ovirt-engine-extension-aaa-jdbc.jar:] at com.sun.proxy.$Proxy177.getConnection(Unknown Source) at org.ovirt.engine.extension.aaa.jdbc.core.datasource.SchemaAwareDataSource.getConnection(SchemaAwareDataSource.java:41) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.ExtensionUtils.checkDbVersion(ExtensionUtils.java:146) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.doInit(AuthnExtension.java:114) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.invoke(AuthnExtension.java:56) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:49) at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:73) at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:109) at org.ovirt.engine.core.extensions.mgr.ExtensionsManager.initialize(ExtensionsManager.java:315) at org.ovirt.engine.core.sso.utils.SsoExtensionsManager.initialize(SsoExtensionsManager.java:91) at org.ovirt.engine.core.sso.utils.SsoExtensionsManager.<init>(SsoExtensionsManager.java:28) at org.ovirt.engine.core.sso.context.SsoContextListener.contextInitialized(SsoContextListener.java:28) at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:205) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:174) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:239) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:99) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.jboss.threads.JBossThread.run(JBossThread.java:320) 2017-11-15 16:50:07,185-05 ERROR [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Error in activating extension 'internal-authn': FATAL: the database system is shutting down 2017-11-15 16:50:07,185-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Class: class org.ovirt.engine.core.extensions.mgr.ExtensionInvokeCommandFailedException ** *</error>* ** --------------D85A420919595D580C076AD7 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, we have a CQ failure in add_dc test. I can see at engine long that we have an issue with 'extension.aaa.jdbc.binding.api.AuthnExtension' I don't think that the patch reported was the actual cause of the failure. Yesterday we were failing on missing package ovirt-cockpit-sso which we added to our repos. and I think perhaps we now exposed an issue with sso support which was undiscovered before. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">These are the reported patches: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83790/">https://gerrit.ovirt.org/#/c/83790/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83779/">https://gerrit.ovirt.org/#/c/83779/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> Here is the original patch for the sso support: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82997/">https://gerrit.ovirt.org/#/c/82997/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/</a><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/"> </a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">2017-11-15 16:50:07,115-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Loading extension 'internal-authn' 2017-11-15 16:50:07,118-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Extension 'internal-authn' loaded 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] Config BEGIN 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.aaa.authn.profile.name: internal 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.provides: org.ovirt.engine.api.extensions.aaa.Authn 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.class: org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.aaa.authn.authz.plugin: internal-authz 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.bindings.method: jbossmodule 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] config.datasource.file: /etc/ovirt-engine/aaa/internal.properties 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.name: internal-authn 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.module: org.ovirt.engine.extension.aaa.jdbc 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] Config END 2017-11-15 16:50:07,119-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Loading extension 'internal-authz' 2017-11-15 16:50:07,120-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Extension 'internal-authz' loaded 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] Config BEGIN 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.provides: org.ovirt.engine.api.extensions.aaa.Authz 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.class: org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthzExtension 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.bindings.method: jbossmodule 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] config.datasource.file: /etc/ovirt-engine/aaa/internal.properties 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.name: internal-authz 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.module: org.ovirt.engine.extension.aaa.jdbc 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] Config END 2017-11-15 16:50:07,121-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Initializing extension 'internal-authn' 2017-11-15 16:50:07,182-05 ERROR [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (ServerService Thread Pool -- 45) [] Unexpected Exception invoking: EXTENSION_INITIALIZE[e5ae1b7f-9104-4f23-a444-7b9175ff68d2] 2017-11-15 16:50:07,182-05 DEBUG [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (ServerService Thread Pool -- 45) [] Exception:: org.postgresql.util.PSQLException: FATAL: the database system is shutting down at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398) [postgresql.jar:] at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173) [postgresql.jar:] at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) [postgresql.jar:] at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136) [postgresql.jar:] at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) [postgresql.jar:] at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) [postgresql.jar:] at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) [postgresql.jar:] at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) [postgresql.jar:] at org.postgresql.Driver.makeConnection(Driver.java:393) [postgresql.jar:] at org.postgresql.Driver.connect(Driver.java:267) [postgresql.jar:] at java.sql.DriverManager.getConnection(DriverManager.java:664) [rt.jar:1.8.0_151] at java.sql.DriverManager.getConnection(DriverManager.java:247) [rt.jar:1.8.0_151] at org.ovirt.engine.extension.aaa.jdbc.core.datasource.DataSourceProvider$1.invoke(DataSourceProvider.java:86) [ovirt-engine-extension-aaa-jdbc.jar:] at com.sun.proxy.$Proxy177.getConnection(Unknown Source) at org.ovirt.engine.extension.aaa.jdbc.core.datasource.SchemaAwareDataSource.getConnection(SchemaAwareDataSource.java:41) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.ExtensionUtils.checkDbVersion(ExtensionUtils.java:146) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.doInit(AuthnExtension.java:114) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.invoke(AuthnExtension.java:56) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:49) at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:73) at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:109) at org.ovirt.engine.core.extensions.mgr.ExtensionsManager.initialize(ExtensionsManager.java:315) at org.ovirt.engine.core.sso.utils.SsoExtensionsManager.initialize(SsoExtensionsManager.java:91) at org.ovirt.engine.core.sso.utils.SsoExtensionsManager.<init>(SsoExtensionsManager.java:28) at org.ovirt.engine.core.sso.context.SsoContextListener.contextInitialized(SsoContextListener.java:28) at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:205) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:174) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:239) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:99) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.jboss.threads.JBossThread.run(JBossThread.java:320) 2017-11-15 16:50:07,185-05 ERROR [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Error in activating extension 'internal-authn': FATAL: the database system is shutting down 2017-11-15 16:50:07,185-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Class: class org.ovirt.engine.core.extensions.mgr.ExtensionInvokeCommandFailedException</pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------D85A420919595D580C076AD7--

4 4

[ OST Failure Report ] [ oVirt master ] [ 14-11-2017 ] [ 002_bootstrap.add_hosts ]
by Dafna Ron 15 Nov '17

15 Nov '17

This is a multi-part message in MIME format. --------------5185435FD38913B821CED967 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We had a failure in upgrade suite for 002_bootstrap.add_hosts. I am not seeing any error that can suggest on an issue in engine. I can see in the host messages host that we have stopped writing to the log for 15 minutes and it may suggest that there is something that is keeping the host from starting which causes us to fail the test on timeout. However, i can use some help in determining the cause for this failure and weather its connected to the bootstrap_add_host test in upgrade. ** *Link to suspected patches: As I said,**I do not think its related, but this is the patch that was reported. * * https://gerrit.ovirt.org/#/c/83854/ Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/* * * *Link to all logs:* *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifact/* * * *(Relevant) error snippet from the log: * * <error> ** _*Test error: *_** ** * Error Message False != True after 900 seconds Stacktrace Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/upgrade-from-release-suite-master/test-scenarios-after-upgrade/002_bootstrap.py", line 187, in add_hosts testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 900 seconds ** ** ** **_lago log: _* * *2017-11-13 15:31:23,212::log_utils.py::__enter__::600::lago.prefix::INFO::ESC[0mESC[0m 2017-11-13 15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:26,220::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:26,221::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:27,222::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 4 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:27,222::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 4 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:28,224::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:28,225::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:28,226::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:29,228::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:29,228::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:29,229::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:30,229::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:30,230::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:30,230::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 2 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:30,230::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:30,231::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:31,231::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:31,232::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:31,232::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:31,232::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:31,233::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:32,234::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:32,234::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:32,234::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:32,234::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:32,235::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:33,235::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:33,235::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:33,236::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:33,236::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:33,237::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,238::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:34,238::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:34,238::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,239::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,407::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,639::ssh.py::ssh::58::lago.ssh::DEBUG::Running bb49adfa on lago-upgrade-from-release-suite-master-host0: true 2017-11-13 15:31:34,665::ssh.py::ssh::81::lago.ssh::DEBUG::Command bb49adfa on lago-upgrade-from-release-suite-master-host0 returned with 0 2017-11-13 15:31:34,665::ssh.py::wait_for_ssh::153::lago.ssh::DEBUG::Wait succeeded for ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,665::log_utils.py::__exit__::611::lago.prefix::INFO::ESC[32mSuccessESC[0m (in 0:00:11) 2017-11-13 15:31:34,666::log_utils.py::__enter__::600::lago.prefix::INFO::ESC[0mESC[0m 2017-11-13 15:31:34,666::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,667::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 100 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,805::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,906::ssh.py::ssh::58::lago.ssh::DEBUG::Running bb724bf2 on lago-upgrade-from-release-suite-master-host0: bash -s < "set -xe DIST=$(uname -r | sed -r 's/^.*\.([^\.]+)\.[^\.]+$/\1/') * * * **_from host: _* * ions. Proceeding anyway. Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 13 10:31:55 lago-upgrade-from-release-suite-master-host0 yum[1294]: Updated: iptables-1.4.21-18.2.el7_4.x86_64 Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 3. Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Created slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Started Session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting Session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: New session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 4. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Created slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Started Session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting Session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: New session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 5. Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd: Starting Cleanup of Temporary Directories... Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd: Started Cleanup of Temporary Directories. Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]: DHCPREQUEST on eth0 to 192.168.200.1 port 67 (xid=0x564629cb) Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]: DHCPACK from 192.168.200.1 (xid=0x564629cb)** * * *</error>* ** --------------5185435FD38913B821CED967 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We had a failure in upgrade suite for 002_bootstrap.add_hosts. I am not seeing any error that can suggest on an issue in engine. I can see in the host messages host that we have stopped writing to the log for 15 minutes and it may suggest that there is something that is keeping the host from starting which causes us to fail the test on timeout. However, i can use some help in determining the cause for this failure and weather its connected to the bootstrap_add_host test in upgrade. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: As I said, I do not think its related, but this is the patch that was reported. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83854/">https://gerrit.ovirt.org/#/c/83854/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Test error: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <h3 style="box-sizing: border-box; color: rgb(51, 51, 51); font-family: Helvetica, Arial, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Error Message</h3> <pre style="box-sizing: border-box; white-space: pre-wrap; word-wrap: break-word; margin: 0px; color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">False != True after 900 seconds</pre> <h3 style="box-sizing: border-box; color: rgb(51, 51, 51); font-family: Helvetica, Arial, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Stacktrace</h3> <pre style="box-sizing: border-box; white-space: pre-wrap; word-wrap: break-word; margin: 0px; color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/upgrade-from-release-suite-master/test-scenarios-after-upgrade/002_bootstrap.py", line 187, in add_hosts testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 900 seconds</pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> lago log: 2017-11-13 15:31:23,212::log_utils.py::__enter__::600::lago.prefix::<a class="moz-txt-link-freetext" href="INFO::ESC">INFO::ESC</a>[0mESC[0m 2017-11-13 15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:26,220::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:26,221::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:27,222::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 4 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:27,222::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 4 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:28,224::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:28,225::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:28,226::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:29,228::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:29,228::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:29,229::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:30,229::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:30,230::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:30,230::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 2 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:30,230::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:30,231::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:31,231::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:31,232::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:31,232::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:31,232::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:31,233::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:32,234::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:32,234::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:32,234::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:32,234::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:32,235::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:33,235::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:33,235::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:33,236::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:33,236::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:33,237::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,238::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:34,238::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:34,238::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,239::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,407::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,639::ssh.py::<a class="moz-txt-link-freetext" href="ssh::58::lago.ssh::DEBUG::Running">ssh::58::lago.ssh::DEBUG::Running</a> bb49adfa on lago-upgrade-from-release-suite-master-host0: true 2017-11-13 15:31:34,665::ssh.py::<a class="moz-txt-link-freetext" href="ssh::81::lago.ssh::DEBUG::Command">ssh::81::lago.ssh::DEBUG::Command</a> bb49adfa on lago-upgrade-from-release-suite-master-host0 returned with 0 2017-11-13 15:31:34,665::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::153::lago.ssh::DEBUG::Wait">ssh::153::lago.ssh::DEBUG::Wait</a> succeeded for ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,665::log_utils.py::__exit__::611::lago.prefix::<a class="moz-txt-link-freetext" href="INFO::ESC">INFO::ESC</a>[32mSuccessESC[0m (in 0:00:11) 2017-11-13 15:31:34,666::log_utils.py::__enter__::600::lago.prefix::<a class="moz-txt-link-freetext" href="INFO::ESC">INFO::ESC</a>[0mESC[0m 2017-11-13 15:31:34,666::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,667::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 100 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,805::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,906::ssh.py::<a class="moz-txt-link-freetext" href="ssh::58::lago.ssh::DEBUG::Running">ssh::58::lago.ssh::DEBUG::Running</a> bb724bf2 on lago-upgrade-from-release-suite-master-host0: bash -s < "set -xe DIST=$(uname -r | sed -r 's/^.*\.([^\.]+)\.[^\.]+$/\1/') from host: ions. Proceeding anyway. Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 13 10:31:55 lago-upgrade-from-release-suite-master-host0 yum[1294]: Updated: iptables-1.4.21-18.2.el7_4.x86_64 Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 3. Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Created slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Started Session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting Session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: New session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 4. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Created slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Started Session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting Session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: New session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 5. Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd: Starting Cleanup of Temporary Directories... Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd: Started Cleanup of Temporary Directories. Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]: DHCPREQUEST on eth0 to 192.168.200.1 port 67 (xid=0x564629cb) Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]: DHCPACK from 192.168.200.1 (xid=0x564629cb) <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------5185435FD38913B821CED967--

3 4

more aggressive static asset caching in ovirt-engine
by Greg Sheremeta 15 Nov '17

15 Nov '17

Hi, I made some changes [1][2] to ovirt-engine that make it now more aggressively (and correctly) cache all static assets. We were previously caching some things, but not all. (Thanks to ykaul for noticing.) The upside is reduced server hits and faster page loads. The downside is you'll have to clear your browser cache more often as you are developing. (You can use a browser plugin, incognito mode, F12 > disable cache, etc. if you don't want to keep opening the dialog.) [1] https://gerrit.ovirt.org/#/c/83879/ [2] https://gerrit.ovirt.org/#/c/83871/ Best wishes, Greg -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

2 2

[ OST Failure Report ] [ oVirt master ] [ 15-11-2017 ] [ 001_upgrade_engine.test_initialize_engine ]
by Dafna Ron 15 Nov '17

15 Nov '17

This is a multi-part message in MIME format. --------------1EB3285B75C820E3A1990192 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We have a few alerts from cq all pointing to the same patch. Adding Sandro since he's working on fixing the packages issue and resolve this. ** *Link to suspected patches: **https://gerrit.ovirt.org/#/c/83852/* ** ** *Link to Job:* *** * *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3824* http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3829/testRepo… http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3833/testRepo… http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3839/testRepo… http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3844/testRepo… http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3850/* * * * *Link to all logs:* * ** http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3824/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3829/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3833/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3839/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3844/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3850/artifact/ (Relevant) error snippet from the log: <error> * Removing obsolete ovirt-engine-setup-plugin-websocket-proxy-4.1.7.7-0.0.master.20171114102253.git9267e4c.el7.centos.noarch.rpm cockpit-ovirt-dashboard-0.10.9 FAILED (1/145): cockpit-ovirt-das 0% [ ] 0.0 B/s | 0 B --:-- ETA cockpit-ovirt-uiplugin-0.10.9- FAILED (1/145): cockpit-ovirt-uip 0% [ ] 0.0 B/s | 0 B --:-- ETA Public key for cockpit-ovirt-uiplugin-0.10.9-1.el7.centos.noarch.rpm is not installed ovirt-ansible-roles-1.0.4-1.el FAILED ovirt-engine-setup-plugin-live FAILED ovirt-engine-yarn-0.19.1-2.el7 FAILED ** *</error>* ** --------------1EB3285B75C820E3A1990192 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We have a few alerts from cq all pointing to the same patch. Adding Sandro since he's working on fixing the packages issue and resolve this. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83852/">https://gerrit.ovirt.org/#/c/83852/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3824">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3824</a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3829/testRepo…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3829/testRepo…</a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3833/testRepo…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3833/testRepo…</a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3839/testRepo…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3839/testRepo…</a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3844/testRepo…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3844/testRepo…</a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3850/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3850/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3824/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3824/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3829/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3829/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3833/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3833/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3839/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3839/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3844/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3844/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3850/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3850/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> Removing obsolete ovirt-engine-setup-plugin-websocket-proxy-4.1.7.7-0.0.master.20171114102253.git9267e4c.el7.centos.noarch.rpm <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"> cockpit-ovirt-dashboard-0.10.9 FAILED (1/145): cockpit-ovirt-das 0% [ ] 0.0 B/s | 0 B --:-- ETA cockpit-ovirt-uiplugin-0.10.9- FAILED (1/145): cockpit-ovirt-uip 0% [ ] 0.0 B/s | 0 B --:-- ETA Public key for cockpit-ovirt-uiplugin-0.10.9-1.el7.centos.noarch.rpm is not installed ovirt-ansible-roles-1.0.4-1.el FAILED ovirt-engine-setup-plugin-live FAILED ovirt-engine-yarn-0.19.1-2.el7 FAILED </pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------1EB3285B75C820E3A1990192--

2 1

speed up engine development with SKIP_CHECKS
by Greg Sheremeta 15 Nov '17

15 Nov '17

Hi, I've added a SKIP_CHECKS flag to the ovirt-engine build here [1]. Thanks to Roy for the idea [2]. TL;DR: you'll save roughly 2 minutes 10 seconds per full compile with SKIP_CHECKS. You'll save more (3 min 10 sec) if you don't need a 'clean'. You'll save the most (about 3 min 20 sec) if you can skip rebuilding GWT. Here are some statistics (3 runs per command, for averages sake) My test machine is a Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz 256GB Samsung SSD (old, using SATA) 32GB DDR3 @ 1333 MT/s --- fresh master checkout no clean, no gwt, skip checks: make install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=0 SKIP_CHECKS=1 1:04.38 total 1:06.04 total 1:04.41 total same as above, but now change one non-GWT class in the utils project make install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=0 SKIP_CHECKS=1 1:05.49 total Ok, pretend we recompiled and our utils class change had an error. Same as above, change the one non-GWT class in the utils project again, and use the resume (-pl) functionality: make install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=0 SKIP_CHECKS=1 EXTRA_BUILD_FLAGS="-pl org.ovirt.engine.core:utils" 17.958 total (yes, 18 seconds, obviously a world record for any engine build) [The resume piece is EXTRA_BUILD_FLAGS="-pl org.ovirt.engine.ui:webadmin" -- I personally need to use this more often!] --- no clean, no gwt, run checks (but skip unit tests): make install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=0 BUILD_UT=0 3:07.66 total 3:16.69 total 3:14.28 total clean, no gwt, skip checks: make clean install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=0 SKIP_CHECKS=1 4:17.74 total 4:15.76 total 4:13.90 total no clean, gwt (1 permutation for chrome), skip checks: make install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=1 SKIP_CHECKS=1 DEV_EXTRA_BUILD_FLAGS_GWT_DEFAULTS="-Dgwt.userAgent=safari" 4:25.63 total 4:26.71 total 4:30.69 total no clean, gwt (1 permutation for chrome), run checks but skip unit tests: make install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=1 BUILD_UT=0 DEV_EXTRA_BUILD_FLAGS_GWT_DEFAULTS="-Dgwt.userAgent=safari" 6:44.23 total And the whole enchilada [3] ... clean, gwt (1 permutation for chrome), run checks, run unit tests: 11:52.25 total ... wow :) Hope this helps you have a faster change-compile-check cycle :) Best wishes, Greg [1] https://gerrit.ovirt.org/#/c/83961/ [2] https://www.mail-archive.com/devel@ovirt.org/msg10377.html [3] https://idioms.thefreedictionary.com/whole+enchilada -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

3 2

[ OST Failure Report ] [ oVirt master ] [ 2017-11-13 ] [add_master_storage_domain]
by Barak Korren 13 Nov '17

13 Nov '17

Test failed: [ add_master_storage_domain ] Link to suspected patches: - https://gerrit.ovirt.org/#/c/83849/4 This seems to be a fairly consistent regression as all patches that follow the patch above exhibit the same issue when tested, this list of similarly failing patches includes: - https://gerrit.ovirt.org/#/c/83940/2 - https://gerrit.ovirt.org/#/c/83608/7 - https://gerrit.ovirt.org/#/c/83815/3 The following patch is an exception because it simply failed to build: - https://gerrit.ovirt.org/#/c/83370/9 Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3770/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3770/artifact… Error snippet from log: <error> Fault reason is "Operation Failed". Fault detail is "[Failed to attach Storage due to an error on the Data Center master Storage Domain. -Please activate the master Storage Domain first.]". HTTP response code is 409. </error> -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

3 8

failing check-patch on ovirt-engine
by Greg Sheremeta 13 Nov '17

13 Nov '17

Hi, I'm seeing a weird issue in ovirt-engine check-patch. There's no way this failure is related to my patch [ https://gerrit.ovirt.org/#/c/83950/] which is a 1 character typo fix in the constants file. Anyone have any ideas or also seeing this? It seems random, too -- sometimes 1 or 2 of the jobs will succeed [el7, fc25] but the other will fail. Etc. *14:48:08* + automation/packaging-setup-tests.sh*14:48:08* + trap popd 0*14:48:08* +++ readlink -f automation/packaging-setup-tests.sh*14:48:08* ++ dirname /home/jenkins/workspace/ovirt-engine_master_check-patch-fc26-x86_64/ovirt-engine/automation/packaging-setup-tests.sh*14:48:08* + pushd /home/jenkins/workspace/ovirt-engine_master_check-patch-fc26-x86_64/ovirt-engine/automation/..*14:48:08* ~ ~*14:48:08* + export PYTHONPATH=packaging/pythonlib:packaging/setup:*14:48:08* + PYTHONPATH=packaging/pythonlib:packaging/setup:*14:48:08* + python -m pytest packaging/setup*14:48:08* ============================= test session starts ==============================*14:48:08* platform linux2 -- Python 2.7.14, pytest-3.2.3, py-1.4.34, pluggy-0.4.0*14:48:08* rootdir: /home/jenkins/workspace/ovirt-engine_master_check-patch-fc26-x86_64/ovirt-engine, inifile:*14:48:08* collected 0 items / 1 errors*14:48:08* *14:48:08* ==================================== ERRORS ====================================*14:48:08* ERROR collecting packaging/setup/tests/ovirt_engine_setup/engine_common/test_database.py *14:48:08* ImportError while importing test module '/home/jenkins/workspace/ovirt-engine_master_check-patch-fc26-x86_64/ovirt-engine/packaging/setup/tests/ovirt_engine_setup/engine_common/test_database.py'.*14:48:08* Hint: make sure your test modules/packages have valid Python names.*14:48:08* Traceback:*14:48:08* packaging/setup/tests/ovirt_engine_setup/engine_common/test_database.py:19: in <module>*14:48:08* import ovirt_engine_setup.engine_common.database as under_test # isort:skip # noqa: E402*14:48:08* packaging/setup/ovirt_engine_setup/engine_common/database.py:30: in <module>*14:48:08* from otopi import base*14:48:08* E ImportError: No module named otopi*14:48:08* !!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!*14:48:08* =========================== 1 error in 0.33 seconds ============================*14:48:08* + popd*14:48:08* ~ ---------- Forwarded message ---------- From: Code Review <gerrit(a)ovirt.org> Date: Sun, Nov 12, 2017 at 10:31 AM Subject: Change in ovirt-engine[master]: engine: typo fix snaphot -> snapshot To: Greg Sheremeta <gshereme(a)redhat.com> Jenkins CI *posted comments* on this change. View Change <https://gerrit.ovirt.org/83950> Patch set 1:Continuous-Integration -1 Build Failed http://jenkins.ovirt.org/job/ovirt-engine_master_check- patch-fc25-x86_64/17231/ : FAILURE http://jenkins.ovirt.org/job/ovirt-engine_master_check- patch-fc26-x86_64/157/ : FAILURE http://jenkins.ovirt.org/job/ovirt-engine_master_check- patch-el7-x86_64/33193/ : SUCCESS To view, visit change 83950 <https://gerrit.ovirt.org/83950>. To unsubscribe, visit settings <https://gerrit.ovirt.org/settings>. Gerrit-Project: ovirt-engine Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88b4d75855c0abe2f0eddd8b282c172205db2e42 Gerrit-Change-Number: 83950 Gerrit-PatchSet: 1 Gerrit-Owner: Greg Sheremeta <gshereme(a)redhat.com> Gerrit-Reviewer: Alexander Wels <awels(a)redhat.com> Gerrit-Reviewer: Greg Sheremeta <gshereme(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Tal Nisan <tnisan(a)redhat.com> Gerrit-Reviewer: Vojtech Szocs <vszocs(a)redhat.com> Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org> Gerrit-Comment-Date: Sun, 12 Nov 2017 15:31:27 +0000 Gerrit-HasComments: No -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

3 2