[ovirt-devel] Re: [CQ]: 105811,4 (ovirt-engine) failed "ovirt-master" system tests

On Thu, Dec 19, 2019 at 11:49 AM Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > > On Thu, Dec 19, 2019 at 3:07 AM oVirt Jenkins <jenkins(a)ovirt.org&gt; wrote: > > > > Change 105811,4 (ovirt-engine) is probably the reason behind recent system test > > failures in the "ovirt-master" change queue and needs to be fixed. > > > > This change had been removed from the testing queue. Artifacts build from this > > change will not be released until it is fixed. > > > > For further details about the change see: > > https://gerrit.ovirt.org/#/c/105811/4 > > > > For failed test results see: > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/ > > lago log [1] has: > > 2019-12-19 01:03:18,289::ssh.py::ssh::58::lago.ssh::DEBUG::Running > 57dde5d6 on lago-basic-suite-master-engine: /usr/bin/systemctl stop > ovirt-engine > 2019-12-19 01:03:20,379::ssh.py::ssh::81::lago.ssh::DEBUG::Command > 57dde5d6 on lago-basic-suite-master-engine returned with 0 > 2019-12-19 01:03:20,381::log_utils.py::__enter__::600::lago.ssh::DEBUG::start > task:b30be6a0-81f5-4611-b638-2cc46f8dd28f:Get ssh client for > lago-basic-suite-master-engine: > 2019-12-19 01:03:20,512::log_utils.py::__exit__::611::lago.ssh::DEBUG::end > task:b30be6a0-81f5-4611-b638-2cc46f8dd28f:Get ssh client for > lago-basic-suite-master-engine: > 2019-12-19 01:03:20,777::ssh.py::ssh::58::lago.ssh::DEBUG::Running > 5959a5f8 on lago-basic-suite-master-engine: /usr/bin/systemctl status > --lines=0 ovirt-engine > 2019-12-19 01:03:20,828::ssh.py::ssh::81::lago.ssh::DEBUG::Command > 5959a5f8 on lago-basic-suite-master-engine returned with 3 > 2019-12-19 01:03:20,828::ssh.py::ssh::89::lago.ssh::DEBUG::Command > 5959a5f8 on lago-basic-suite-master-engine output: > ◠ovirt-engine.service - oVirt Engine > Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; > enabled; vendor preset: disabled) > Active: inactive (dead) since Wed 2019-12-18 20:03:20 EST; 455ms ago > Process: 24166 > ExecStart=/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py > --redirect-output --systemd=notify $EXTRA_ARGS start (code=exited, > status=0/SUCCESS) > Main PID: 24166 (code=exited, status=0/SUCCESS) > > 2019-12-19 01:03:20,829::log_utils.py::__enter__::600::lago.ssh::DEBUG::start > task:6b4dab7c-64f3-43e5-ae85-2fe6246d9376:Get ssh client for > lago-basic-suite-master-engine: > 2019-12-19 01:03:20,998::log_utils.py::__exit__::611::lago.ssh::DEBUG::end > task:6b4dab7c-64f3-43e5-ae85-2fe6246d9376:Get ssh client for > lago-basic-suite-master-engine: > 2019-12-19 01:03:21,214::ssh.py::ssh::58::lago.ssh::DEBUG::Running > 599c5b96 on lago-basic-suite-master-engine: /usr/bin/systemctl start > ovirt-engine > 2019-12-19 01:04:13,735::ssh.py::ssh::81::lago.ssh::DEBUG::Command > 599c5b96 on lago-basic-suite-master-engine returned with 0 > 2019-12-19 01:04:13,737::log_utils.py::__enter__::600::lago.ssh::DEBUG::start > task:5b775d53-c53c-4d15-9cae-2806c44079b7:Get ssh client for > lago-basic-suite-master-engine: > 2019-12-19 01:04:13,971::log_utils.py::__exit__::611::lago.ssh::DEBUG::end > task:5b775d53-c53c-4d15-9cae-2806c44079b7:Get ssh client for > lago-basic-suite-master-engine: > 2019-12-19 01:04:14,488::ssh.py::ssh::58::lago.ssh::DEBUG::Running > 795d5098 on lago-basic-suite-master-engine: /usr/bin/systemctl status > --lines=0 ovirt-engine > 2019-12-19 01:04:15,039::ssh.py::ssh::81::lago.ssh::DEBUG::Command > 795d5098 on lago-basic-suite-master-engine returned with 0 > 2019-12-19 01:04:15,040::ssh.py::ssh::89::lago.ssh::DEBUG::Command > 795d5098 on lago-basic-suite-master-engine output: > ◠ovirt-engine.service - oVirt Engine > Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; > enabled; vendor preset: disabled) > Active: active (running) since Wed 2019-12-18 20:04:13 EST; 1s ago > Main PID: 5718 (ovirt-engine.py) > CGroup: /system.slice/ovirt-engine.service > ├─5718 /usr/bin/python2 > /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py > --redirect-output --systemd=notify start > └─6073 ovirt-engine --add-modules java.se -server > -XX:+TieredCompilation -Xms1024M -Xmx1024M -Xss1M > -Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=3600000 > -Dsun.rmi.dgc.server.gcInterval=3600000 > -Djsse.enableSNIExtension=false -Dresteasy.preferJacksonOverJsonB=true > -Djackson.deserialization.whitelist.packages=org,com,java,javax > -XX:+HeapDumpOnOutOfMemoryError > -XX:HeapDumpPath=/var/log/ovirt-engine/dump > -Djava.util.logging.manager=org.jboss.logmanager > -Dlogging.configuration=file:///var/lib/ovirt-engine/jboss_runtime/config/ovirt-engine-logging.properties > -Dorg.jboss.resolver.warning=true > -Djboss.modules.system.pkgs=org.jboss.byteman > -Djboss.server.default.config=ovirt-engine > -Djboss.home.dir=/usr/share/ovirt-engine-wildfly > -Djboss.server.base.dir=/usr/share/ovirt-engine > -Djboss.server.data.dir=/var/lib/ovirt-engine > -Djboss.server.log.dir=/var/log/ovirt-engine > -Djboss.server.config.dir=/var/lib/ovirt-engine/jboss_runtime/config > -Djboss.server.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp > -Djboss.controller.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp > -jar /usr/share/ovirt-engine-wildfly/jboss-modules.jar -mp > /usr/share/ovirt-engine-wildfly-overlay/modules:/usr/share/ovirt-engine/modules/common:/usr/share/ovirt-engine-extension-aaa-jdbc/modules:/usr/share/ovirt-engine-extension-aaa-ldap/modules:/usr/share/ovirt-engine-wildfly/modules > -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -c > ovirt-engine.xml > > 2019-12-19 01:04:15,040::log_utils.py::end_log_task::670::nose::INFO:: > # add_ldap_provider: [32mSuccess [0m (in 0:01:08) > 2019-12-19 01:05:12,552::log_utils.py::start_log_task::655::nose::INFO:: > # Failure: RuntimeError (test api call failed): [0m [0m > > So probably the engine didn't manage to start yet and connecting to > its api failed [2]. Perhaps wait, after starting it, until it's really > up (checking the api or health)? Maybe, but above change should not affect anyhow engine startup. And also newer CQ tests were green: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17808/ https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17810/ https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17811/ So it seems to me as some single occasion env failure ...

> > > [1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/arti... > [2] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17803/test... > > > _______________________________________________ > > Infra mailing list -- infra(a)ovirt.org > > To unsubscribe send an email to infra-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/P3BLXLG63LC... > > > > -- > Didi > -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.