On Thu, Dec 19, 2019 at 11:49 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Thu, Dec 19, 2019 at 3:07 AM oVirt Jenkins <jenkins@ovirt.org> wrote:
>
> Change 105811,4 (ovirt-engine) is probably the reason behind recent system test
> failures in the "ovirt-master" change queue and needs to be fixed.
>
> This change had been removed from the testing queue. Artifacts build from this
> change will not be released until it is fixed.
>
> For further details about the change see:
> https://gerrit.ovirt.org/#/c/105811/4
>
> For failed test results see:
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/

lago log [1] has:

2019-12-19 01:03:18,289::ssh.py::ssh::58::lago.ssh::DEBUG::Running
57dde5d6 on lago-basic-suite-master-engine: /usr/bin/systemctl stop
ovirt-engine
2019-12-19 01:03:20,379::ssh.py::ssh::81::lago.ssh::DEBUG::Command
57dde5d6 on lago-basic-suite-master-engine returned with 0
2019-12-19 01:03:20,381::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:b30be6a0-81f5-4611-b638-2cc46f8dd28f:Get ssh client for
lago-basic-suite-master-engine:
2019-12-19 01:03:20,512::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:b30be6a0-81f5-4611-b638-2cc46f8dd28f:Get ssh client for
lago-basic-suite-master-engine:
2019-12-19 01:03:20,777::ssh.py::ssh::58::lago.ssh::DEBUG::Running
5959a5f8 on lago-basic-suite-master-engine: /usr/bin/systemctl status
--lines=0 ovirt-engine
2019-12-19 01:03:20,828::ssh.py::ssh::81::lago.ssh::DEBUG::Command
5959a5f8 on lago-basic-suite-master-engine returned with 3
2019-12-19 01:03:20,828::ssh.py::ssh::89::lago.ssh::DEBUG::Command
5959a5f8 on lago-basic-suite-master-engine output:
 â—  ovirt-engine.service - oVirt Engine
   Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service;
enabled; vendor preset: disabled)
   Active: inactive (dead) since Wed 2019-12-18 20:03:20 EST; 455ms ago
  Process: 24166
ExecStart=/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py
--redirect-output --systemd=notify $EXTRA_ARGS start (code=exited,
status=0/SUCCESS)
 Main PID: 24166 (code=exited, status=0/SUCCESS)

2019-12-19 01:03:20,829::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:6b4dab7c-64f3-43e5-ae85-2fe6246d9376:Get ssh client for
lago-basic-suite-master-engine:
2019-12-19 01:03:20,998::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:6b4dab7c-64f3-43e5-ae85-2fe6246d9376:Get ssh client for
lago-basic-suite-master-engine:
2019-12-19 01:03:21,214::ssh.py::ssh::58::lago.ssh::DEBUG::Running
599c5b96 on lago-basic-suite-master-engine: /usr/bin/systemctl start
ovirt-engine
2019-12-19 01:04:13,735::ssh.py::ssh::81::lago.ssh::DEBUG::Command
599c5b96 on lago-basic-suite-master-engine returned with 0
2019-12-19 01:04:13,737::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:5b775d53-c53c-4d15-9cae-2806c44079b7:Get ssh client for
lago-basic-suite-master-engine:
2019-12-19 01:04:13,971::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:5b775d53-c53c-4d15-9cae-2806c44079b7:Get ssh client for
lago-basic-suite-master-engine:
2019-12-19 01:04:14,488::ssh.py::ssh::58::lago.ssh::DEBUG::Running
795d5098 on lago-basic-suite-master-engine: /usr/bin/systemctl status
--lines=0 ovirt-engine
2019-12-19 01:04:15,039::ssh.py::ssh::81::lago.ssh::DEBUG::Command
795d5098 on lago-basic-suite-master-engine returned with 0
2019-12-19 01:04:15,040::ssh.py::ssh::89::lago.ssh::DEBUG::Command
795d5098 on lago-basic-suite-master-engine output:
 â—  ovirt-engine.service - oVirt Engine
   Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service;
enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-12-18 20:04:13 EST; 1s ago
 Main PID: 5718 (ovirt-engine.py)
   CGroup: /system.slice/ovirt-engine.service
           â”œâ”€5718 /usr/bin/python2
/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py
--redirect-output --systemd=notify start
           â””─6073 ovirt-engine --add-modules java.se -server
-XX:+TieredCompilation -Xms1024M -Xmx1024M -Xss1M
-Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000
-Djsse.enableSNIExtension=false -Dresteasy.preferJacksonOverJsonB=true
-Djackson.deserialization.whitelist.packages=org,com,java,javax
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/log/ovirt-engine/dump
-Djava.util.logging.manager=org.jboss.logmanager
-Dlogging.configuration=file:///var/lib/ovirt-engine/jboss_runtime/config/ovirt-engine-logging.properties
-Dorg.jboss.resolver.warning=true
-Djboss.modules.system.pkgs=org.jboss.byteman
-Djboss.server.default.config=ovirt-engine
-Djboss.home.dir=/usr/share/ovirt-engine-wildfly
-Djboss.server.base.dir=/usr/share/ovirt-engine
-Djboss.server.data.dir=/var/lib/ovirt-engine
-Djboss.server.log.dir=/var/log/ovirt-engine
-Djboss.server.config.dir=/var/lib/ovirt-engine/jboss_runtime/config
-Djboss.server.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp
-Djboss.controller.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp
-jar /usr/share/ovirt-engine-wildfly/jboss-modules.jar -mp
/usr/share/ovirt-engine-wildfly-overlay/modules:/usr/share/ovirt-engine/modules/common:/usr/share/ovirt-engine-extension-aaa-jdbc/modules:/usr/share/ovirt-engine-extension-aaa-ldap/modules:/usr/share/ovirt-engine-wildfly/modules
-jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -c
ovirt-engine.xml

2019-12-19 01:04:15,040::log_utils.py::end_log_task::670::nose::INFO::
 # add_ldap_provider:  [32mSuccess [0m (in 0:01:08)
2019-12-19 01:05:12,552::log_utils.py::start_log_task::655::nose::INFO::
 # Failure: RuntimeError (test api call failed):  [0m [0m

So probably the engine didn't manage to start yet and connecting to
its api failed [2]. Perhaps wait, after starting it, until it's really
up (checking the api or health)?

Maybe, but above change should not affect anyhow engine startup. And also newer CQ tests were green:


So it seems to me as some single occasion env failure ...

[1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-master/post-099_aaa-ldap.py/lago_logs/lago.log/*view*/
[2] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/testReport/junit/nose.failure/Failure/runTest/

> _______________________________________________
> Infra mailing list -- infra@ovirt.org
> To unsubscribe send an email to infra-leave@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/P3BLXLG63LC7PNFGYW6UI6H3R7A6XNV3/



--
Didi



--
Martin Perina
Manager, Software Engineering
Red Hat Czech s.r.o.