[ovirt-devel] Re: Health check endpoint in the engine hanging forever on CheckDBConnection

On 27 Nov 2020, at 12:04, Artur Socha <asocha(a)redhat.com&gt; wrote: In this run there is ovirt-engine-* used with git c2c805a2662, however, vdsm-jsonrpc-client is in version 1.6.0 which has a breaking change of removing reactive streams support in favor to java.util.concurrent.FLOW [1] which is handled by engine's commit ff3aa4da956 [2] [1] https://gerrit.ovirt.org/#/c/vdsm-jsonrpc-java/+/109916/ <https://gerrit.ovirt.org/#/c/vdsm-jsonrpc-java/+/109916/> [2] https://gerrit.ovirt.org/#/c/ovirt-engine/+/112347/ <https://gerrit.ovirt.org/#/c/ovirt-engine/+/112347/> Artur On Fri, Nov 27, 2020 at 11:49 AM Artur Socha <asocha(a)redhat.com <mailto:asocha@redhat.com>> wrote: I can see something I was recently touching. Removal of reactive stream from vdsm-jsonrpc-java. Checking... Artur On Fri, Nov 27, 2020 at 11:47 AM Marcin Sobczyk <msobczyk(a)redhat.com <mailto:msobczyk@redhat.com>> wrote: On 11/27/20 11:24 AM, Martin Perina wrote: > Hi, > > the health status is pretty stupid simple call to database: > > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... > <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... > <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... > <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... > https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/com... <https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/com... > <https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/com... <https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/com... > > So it should definitely not hang forever unless there is some serious > issue in the engine start up or PostgreSQL database. Could you please > share logs? Especially interesting would server.log and engine.log > from /var/log/ovirt-engine Well, after some discussion with Artur and trying some workarounds, the problem magically disappeared on my servers, but there's one OST gating run in CI that suffered from the same problem: https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_g... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_g... https://jenkins.ovirt.org/job/ovirt-system-tests_gate/937/artifact/basic-... <https://jenkins.ovirt.org/job/ovirt-system-tests_gate/937/artifact/basic-... https://jenkins.ovirt.org/job/ovirt-system-tests_gate/937/artifact/basic-... <https://jenkins.ovirt.org/job/ovirt-system-tests_gate/937/artifact/basic-... > > Martin > > > On Thu, Nov 26, 2020 at 4:24 PM Marcin Sobczyk <msobczyk(a)redhat.com <mailto:msobczyk@redhat.com> > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: > > Hi, > > I'm trying out (I think) the latest build of ovirt-engine in OST > [1] and > the basic suite > fails when we do engine reconfiguration and then restart the > service [2]. > After restarting we wait on the health check endpoint status here [3]. > This however ends with a timeout. > I tried running manually: > > curl -D - http://engine/ovirt-engine/services/health <http://engine/ovirt-engine/services/health> > <http://engine/ovirt-engine/services/health <http://engine/ovirt-engine/services/health>> > > and that command also hangs forever. > In the engine log the last related entry seem to be: > > 2020-11-26 16:20:15,761+01 DEBUG > [org.ovirt.engine.core.utils.servlet.LocaleFilter] (default > task-7) [] > Incoming locale 'en-US'. Filter determined locale to be 'en-US' > 2020-11-26 16:20:15,761+01 DEBUG > [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] > Health > Status servlet: entry > 2020-11-26 16:20:15,761+01 DEBUG > [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] > Calling CheckDBConnection query > > Has anyone else also encountered that? > > Regards, Marcin > > [1] > ovirt-engine-4.4.4.3-0.0.master.20201126133903.gitc2c805a2662.el8.noarch > [2] > https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0... <https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0... > <https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0... <https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0... > [3] > https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0... <https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0... > <https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0... <https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0... > > > > -- > Martin Perina > Manager, Software Engineering > Red Hat Czech s.r.o. -- Artur Socha Senior Software Engineer, RHV Red Hat -- Artur Socha Senior Software Engineer, RHV Red Hat _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/DTY2FJHJ3YB...