[ovirt-devel] Re: Health check endpoint in the engine hanging forever on CheckDBConnection

I can see something I was recently touching. Removal of reactive stream from vdsm-jsonrpc-java. Checking... Artur On Fri, Nov 27, 2020 at 11:47 AM Marcin Sobczyk <msobczyk(a)redhat.com&gt; wrote: > > > On 11/27/20 11:24 AM, Martin Perina wrote: > > Hi, > > > > the health status is pretty stupid simple call to database: > > > > > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... > > < > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... > > > > > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... > > < > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... > > > > > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... > > < > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... > > > > > https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/com... > > < > https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/com... > > > > > > So it should definitely not hang forever unless there is some serious > > issue in the engine start up or PostgreSQL database. Could you please > > share logs? Especially interesting would server.log and engine.log > > from /var/log/ovirt-engine > Well, after some discussion with Artur and trying some workarounds, the > problem magically > disappeared on my servers, but there's one OST gating run in CI that > suffered from the same problem: > > > https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_g... > > > https://jenkins.ovirt.org/job/ovirt-system-tests_gate/937/artifact/basic-... > > > https://jenkins.ovirt.org/job/ovirt-system-tests_gate/937/artifact/basic-... > > > > > > Martin > > > > > > On Thu, Nov 26, 2020 at 4:24 PM Marcin Sobczyk <msobczyk(a)redhat.com > > <mailto:msobczyk@redhat.com>> wrote: > > > > Hi, > > > > I'm trying out (I think) the latest build of ovirt-engine in OST > > [1] and > > the basic suite > > fails when we do engine reconfiguration and then restart the > > service [2]. > > After restarting we wait on the health check endpoint status here > [3]. > > This however ends with a timeout. > > I tried running manually: > > > > curl -D - http://engine/ovirt-engine/services/health > > <http://engine/ovirt-engine/services/health> > > > > and that command also hangs forever. > > In the engine log the last related entry seem to be: > > > > 2020-11-26 16:20:15,761+01 DEBUG > > [org.ovirt.engine.core.utils.servlet.LocaleFilter] (default > > task-7) [] > > Incoming locale 'en-US'. Filter determined locale to be 'en-US' > > 2020-11-26 16:20:15,761+01 DEBUG > > [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] > > Health > > Status servlet: entry > > 2020-11-26 16:20:15,761+01 DEBUG > > [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] > > Calling CheckDBConnection query > > > > Has anyone else also encountered that? > > > > Regards, Marcin > > > > [1] > > > ovirt-engine-4.4.4.3-0.0.master.20201126133903.gitc2c805a2662.el8.noarch > > [2] > > > https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0... > > < > https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0... > > > > [3] > > > https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0... > > < > https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0... > > > > > > > > > > -- > > Martin Perina > > Manager, Software Engineering > > Red Hat Czech s.r.o. > > -- Artur Socha Senior Software Engineer, RHV Red Hat