Re: [CQ]: 105811,4 (ovirt-engine) failed "ovirt-master" system tests
On Thu, Dec 19, 2019 at 3:07 AM oVirt Jenkins <jenkins@ovirt.org> wrote:
Change 105811,4 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/105811/4
For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/
lago log [1] has: 2019-12-19 01:03:18,289::ssh.py::ssh::58::lago.ssh::DEBUG::Running 57dde5d6 on lago-basic-suite-master-engine: /usr/bin/systemctl stop ovirt-engine 2019-12-19 01:03:20,379::ssh.py::ssh::81::lago.ssh::DEBUG::Command 57dde5d6 on lago-basic-suite-master-engine returned with 0 2019-12-19 01:03:20,381::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:b30be6a0-81f5-4611-b638-2cc46f8dd28f:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:03:20,512::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:b30be6a0-81f5-4611-b638-2cc46f8dd28f:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:03:20,777::ssh.py::ssh::58::lago.ssh::DEBUG::Running 5959a5f8 on lago-basic-suite-master-engine: /usr/bin/systemctl status --lines=0 ovirt-engine 2019-12-19 01:03:20,828::ssh.py::ssh::81::lago.ssh::DEBUG::Command 5959a5f8 on lago-basic-suite-master-engine returned with 3 2019-12-19 01:03:20,828::ssh.py::ssh::89::lago.ssh::DEBUG::Command 5959a5f8 on lago-basic-suite-master-engine output: ◠ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled; vendor preset: disabled) Active: inactive (dead) since Wed 2019-12-18 20:03:20 EST; 455ms ago Process: 24166 ExecStart=/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify $EXTRA_ARGS start (code=exited, status=0/SUCCESS) Main PID: 24166 (code=exited, status=0/SUCCESS) 2019-12-19 01:03:20,829::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:6b4dab7c-64f3-43e5-ae85-2fe6246d9376:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:03:20,998::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:6b4dab7c-64f3-43e5-ae85-2fe6246d9376:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:03:21,214::ssh.py::ssh::58::lago.ssh::DEBUG::Running 599c5b96 on lago-basic-suite-master-engine: /usr/bin/systemctl start ovirt-engine 2019-12-19 01:04:13,735::ssh.py::ssh::81::lago.ssh::DEBUG::Command 599c5b96 on lago-basic-suite-master-engine returned with 0 2019-12-19 01:04:13,737::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:5b775d53-c53c-4d15-9cae-2806c44079b7:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:04:13,971::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:5b775d53-c53c-4d15-9cae-2806c44079b7:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:04:14,488::ssh.py::ssh::58::lago.ssh::DEBUG::Running 795d5098 on lago-basic-suite-master-engine: /usr/bin/systemctl status --lines=0 ovirt-engine 2019-12-19 01:04:15,039::ssh.py::ssh::81::lago.ssh::DEBUG::Command 795d5098 on lago-basic-suite-master-engine returned with 0 2019-12-19 01:04:15,040::ssh.py::ssh::89::lago.ssh::DEBUG::Command 795d5098 on lago-basic-suite-master-engine output: ◠ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2019-12-18 20:04:13 EST; 1s ago Main PID: 5718 (ovirt-engine.py) CGroup: /system.slice/ovirt-engine.service ├─5718 /usr/bin/python2 /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify start └─6073 ovirt-engine --add-modules java.se -server -XX:+TieredCompilation -Xms1024M -Xmx1024M -Xss1M -Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djsse.enableSNIExtension=false -Dresteasy.preferJacksonOverJsonB=true -Djackson.deserialization.whitelist.packages=org,com,java,javax -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/ovirt-engine/dump -Djava.util.logging.manager=org.jboss.logmanager -Dlogging.configuration=file:///var/lib/ovirt-engine/jboss_runtime/config/ovirt-engine-logging.properties -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.server.default.config=ovirt-engine -Djboss.home.dir=/usr/share/ovirt-engine-wildfly -Djboss.server.base.dir=/usr/share/ovirt-engine -Djboss.server.data.dir=/var/lib/ovirt-engine -Djboss.server.log.dir=/var/log/ovirt-engine -Djboss.server.config.dir=/var/lib/ovirt-engine/jboss_runtime/config -Djboss.server.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp -Djboss.controller.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp -jar /usr/share/ovirt-engine-wildfly/jboss-modules.jar -mp /usr/share/ovirt-engine-wildfly-overlay/modules:/usr/share/ovirt-engine/modules/common:/usr/share/ovirt-engine-extension-aaa-jdbc/modules:/usr/share/ovirt-engine-extension-aaa-ldap/modules:/usr/share/ovirt-engine-wildfly/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -c ovirt-engine.xml 2019-12-19 01:04:15,040::log_utils.py::end_log_task::670::nose::INFO:: # add_ldap_provider: [32mSuccess [0m (in 0:01:08) 2019-12-19 01:05:12,552::log_utils.py::start_log_task::655::nose::INFO:: # Failure: RuntimeError (test api call failed): [0m [0m So probably the engine didn't manage to start yet and connecting to its api failed [2]. Perhaps wait, after starting it, until it's really up (checking the api or health)? [1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/artifac... [2] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/testRep...
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/P3BLXLG63LC7PN...
-- Didi
On Thu, Dec 19, 2019 at 11:49 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Thu, Dec 19, 2019 at 3:07 AM oVirt Jenkins <jenkins@ovirt.org> wrote:
Change 105811,4 (ovirt-engine) is probably the reason behind recent
system test
failures in the "ovirt-master" change queue and needs to be fixed.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/105811/4
For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/
lago log [1] has:
2019-12-19 01:03:18,289::ssh.py::ssh::58::lago.ssh::DEBUG::Running 57dde5d6 on lago-basic-suite-master-engine: /usr/bin/systemctl stop ovirt-engine 2019-12-19 01:03:20,379::ssh.py::ssh::81::lago.ssh::DEBUG::Command 57dde5d6 on lago-basic-suite-master-engine returned with 0 2019-12-19 01:03:20,381::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:b30be6a0-81f5-4611-b638-2cc46f8dd28f:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:03:20,512::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:b30be6a0-81f5-4611-b638-2cc46f8dd28f:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:03:20,777::ssh.py::ssh::58::lago.ssh::DEBUG::Running 5959a5f8 on lago-basic-suite-master-engine: /usr/bin/systemctl status --lines=0 ovirt-engine 2019-12-19 01:03:20,828::ssh.py::ssh::81::lago.ssh::DEBUG::Command 5959a5f8 on lago-basic-suite-master-engine returned with 3 2019-12-19 01:03:20,828::ssh.py::ssh::89::lago.ssh::DEBUG::Command 5959a5f8 on lago-basic-suite-master-engine output: â— ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled; vendor preset: disabled) Active: inactive (dead) since Wed 2019-12-18 20:03:20 EST; 455ms ago Process: 24166 ExecStart=/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify $EXTRA_ARGS start (code=exited, status=0/SUCCESS) Main PID: 24166 (code=exited, status=0/SUCCESS)
2019-12-19 01:03:20,829::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:6b4dab7c-64f3-43e5-ae85-2fe6246d9376:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:03:20,998::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:6b4dab7c-64f3-43e5-ae85-2fe6246d9376:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:03:21,214::ssh.py::ssh::58::lago.ssh::DEBUG::Running 599c5b96 on lago-basic-suite-master-engine: /usr/bin/systemctl start ovirt-engine 2019-12-19 01:04:13,735::ssh.py::ssh::81::lago.ssh::DEBUG::Command 599c5b96 on lago-basic-suite-master-engine returned with 0 2019-12-19 01:04:13,737::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:5b775d53-c53c-4d15-9cae-2806c44079b7:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:04:13,971::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:5b775d53-c53c-4d15-9cae-2806c44079b7:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:04:14,488::ssh.py::ssh::58::lago.ssh::DEBUG::Running 795d5098 on lago-basic-suite-master-engine: /usr/bin/systemctl status --lines=0 ovirt-engine 2019-12-19 01:04:15,039::ssh.py::ssh::81::lago.ssh::DEBUG::Command 795d5098 on lago-basic-suite-master-engine returned with 0 2019-12-19 01:04:15,040::ssh.py::ssh::89::lago.ssh::DEBUG::Command 795d5098 on lago-basic-suite-master-engine output: ◠ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2019-12-18 20:04:13 EST; 1s ago Main PID: 5718 (ovirt-engine.py) CGroup: /system.slice/ovirt-engine.service ├─5718 /usr/bin/python2 /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify start └─6073 ovirt-engine --add-modules java.se -server -XX:+TieredCompilation -Xms1024M -Xmx1024M -Xss1M -Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djsse.enableSNIExtension=false -Dresteasy.preferJacksonOverJsonB=true -Djackson.deserialization.whitelist.packages=org,com,java,javax -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/ovirt-engine/dump -Djava.util.logging.manager=org.jboss.logmanager
-Dlogging.configuration=file:///var/lib/ovirt-engine/jboss_runtime/config/ovirt-engine-logging.properties -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.server.default.config=ovirt-engine -Djboss.home.dir=/usr/share/ovirt-engine-wildfly -Djboss.server.base.dir=/usr/share/ovirt-engine -Djboss.server.data.dir=/var/lib/ovirt-engine -Djboss.server.log.dir=/var/log/ovirt-engine -Djboss.server.config.dir=/var/lib/ovirt-engine/jboss_runtime/config -Djboss.server.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp -Djboss.controller.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp -jar /usr/share/ovirt-engine-wildfly/jboss-modules.jar -mp
/usr/share/ovirt-engine-wildfly-overlay/modules:/usr/share/ovirt-engine/modules/common:/usr/share/ovirt-engine-extension-aaa-jdbc/modules:/usr/share/ovirt-engine-extension-aaa-ldap/modules:/usr/share/ovirt-engine-wildfly/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -c ovirt-engine.xml
2019-12-19 01:04:15,040::log_utils.py::end_log_task::670::nose::INFO:: # add_ldap_provider: [32mSuccess [0m (in 0:01:08) 2019-12-19 01:05:12,552::log_utils.py::start_log_task::655::nose::INFO:: # Failure: RuntimeError (test api call failed): [0m [0m
So probably the engine didn't manage to start yet and connecting to its api failed [2]. Perhaps wait, after starting it, until it's really up (checking the api or health)?
Maybe, but above change should not affect anyhow engine startup. And also newer CQ tests were green: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17808/ https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17810/ https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17811/ So it seems to me as some single occasion env failure ...
[1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/artifac... [2] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/testRep...
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/P3BLXLG63LC7PN...
-- Didi
-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.
On Fri, Dec 20, 2019 at 11:00 AM Martin Perina <mperina@redhat.com> wrote:
On Thu, Dec 19, 2019 at 11:49 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Thu, Dec 19, 2019 at 3:07 AM oVirt Jenkins <jenkins@ovirt.org> wrote:
Change 105811,4 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/105811/4
For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/
lago log [1] has:
2019-12-19 01:03:18,289::ssh.py::ssh::58::lago.ssh::DEBUG::Running 57dde5d6 on lago-basic-suite-master-engine: /usr/bin/systemctl stop ovirt-engine 2019-12-19 01:03:20,379::ssh.py::ssh::81::lago.ssh::DEBUG::Command 57dde5d6 on lago-basic-suite-master-engine returned with 0 2019-12-19 01:03:20,381::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:b30be6a0-81f5-4611-b638-2cc46f8dd28f:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:03:20,512::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:b30be6a0-81f5-4611-b638-2cc46f8dd28f:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:03:20,777::ssh.py::ssh::58::lago.ssh::DEBUG::Running 5959a5f8 on lago-basic-suite-master-engine: /usr/bin/systemctl status --lines=0 ovirt-engine 2019-12-19 01:03:20,828::ssh.py::ssh::81::lago.ssh::DEBUG::Command 5959a5f8 on lago-basic-suite-master-engine returned with 3 2019-12-19 01:03:20,828::ssh.py::ssh::89::lago.ssh::DEBUG::Command 5959a5f8 on lago-basic-suite-master-engine output: â— ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled; vendor preset: disabled) Active: inactive (dead) since Wed 2019-12-18 20:03:20 EST; 455ms ago Process: 24166 ExecStart=/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify $EXTRA_ARGS start (code=exited, status=0/SUCCESS) Main PID: 24166 (code=exited, status=0/SUCCESS)
2019-12-19 01:03:20,829::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:6b4dab7c-64f3-43e5-ae85-2fe6246d9376:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:03:20,998::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:6b4dab7c-64f3-43e5-ae85-2fe6246d9376:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:03:21,214::ssh.py::ssh::58::lago.ssh::DEBUG::Running 599c5b96 on lago-basic-suite-master-engine: /usr/bin/systemctl start ovirt-engine 2019-12-19 01:04:13,735::ssh.py::ssh::81::lago.ssh::DEBUG::Command 599c5b96 on lago-basic-suite-master-engine returned with 0 2019-12-19 01:04:13,737::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:5b775d53-c53c-4d15-9cae-2806c44079b7:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:04:13,971::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:5b775d53-c53c-4d15-9cae-2806c44079b7:Get ssh client for lago-basic-suite-master-engine: 2019-12-19 01:04:14,488::ssh.py::ssh::58::lago.ssh::DEBUG::Running 795d5098 on lago-basic-suite-master-engine: /usr/bin/systemctl status --lines=0 ovirt-engine 2019-12-19 01:04:15,039::ssh.py::ssh::81::lago.ssh::DEBUG::Command 795d5098 on lago-basic-suite-master-engine returned with 0 2019-12-19 01:04:15,040::ssh.py::ssh::89::lago.ssh::DEBUG::Command 795d5098 on lago-basic-suite-master-engine output: ◠ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2019-12-18 20:04:13 EST; 1s ago Main PID: 5718 (ovirt-engine.py) CGroup: /system.slice/ovirt-engine.service ├─5718 /usr/bin/python2 /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify start └─6073 ovirt-engine --add-modules java.se -server -XX:+TieredCompilation -Xms1024M -Xmx1024M -Xss1M -Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djsse.enableSNIExtension=false -Dresteasy.preferJacksonOverJsonB=true -Djackson.deserialization.whitelist.packages=org,com,java,javax -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/ovirt-engine/dump -Djava.util.logging.manager=org.jboss.logmanager -Dlogging.configuration=file:///var/lib/ovirt-engine/jboss_runtime/config/ovirt-engine-logging.properties -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.server.default.config=ovirt-engine -Djboss.home.dir=/usr/share/ovirt-engine-wildfly -Djboss.server.base.dir=/usr/share/ovirt-engine -Djboss.server.data.dir=/var/lib/ovirt-engine -Djboss.server.log.dir=/var/log/ovirt-engine -Djboss.server.config.dir=/var/lib/ovirt-engine/jboss_runtime/config -Djboss.server.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp -Djboss.controller.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp -jar /usr/share/ovirt-engine-wildfly/jboss-modules.jar -mp /usr/share/ovirt-engine-wildfly-overlay/modules:/usr/share/ovirt-engine/modules/common:/usr/share/ovirt-engine-extension-aaa-jdbc/modules:/usr/share/ovirt-engine-extension-aaa-ldap/modules:/usr/share/ovirt-engine-wildfly/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -c ovirt-engine.xml
2019-12-19 01:04:15,040::log_utils.py::end_log_task::670::nose::INFO:: # add_ldap_provider: [32mSuccess [0m (in 0:01:08) 2019-12-19 01:05:12,552::log_utils.py::start_log_task::655::nose::INFO:: # Failure: RuntimeError (test api call failed): [0m [0m
So probably the engine didn't manage to start yet and connecting to its api failed [2]. Perhaps wait, after starting it, until it's really up (checking the api or health)?
Maybe, but above change should not affect anyhow engine startup. And also newer CQ tests were green:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17808/ https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17810/ https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17811/
So it seems to me as some single occasion env failure ...
The fact that starting the engine and then immediately trying to use it does not always work, is definitely not a single occasion: https://bugzilla.redhat.com/show_bug.cgi?id=1075013
[1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/artifac... [2] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/testRep...
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/P3BLXLG63LC7PN...
-- Didi
-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.
-- Didi
participants (2)
-
Martin Perina -
Yedidyah Bar David