Hi, the health status is pretty stupid simple call to database: https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/se... https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bl... https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/da... https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/common... So it should definitely not hang forever unless there is some serious issue in the engine start up or PostgreSQL database. Could you please share logs? Especially interesting would server.log and engine.log from /var/log/ovirt-engine Martin On Thu, Nov 26, 2020 at 4:24 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:

Hi,

I'm trying out (I think) the latest build of ovirt-engine in OST [1] and the basic suite fails when we do engine reconfiguration and then restart the service [2]. After restarting we wait on the health check endpoint status here [3]. This however ends with a timeout. I tried running manually:

curl -D - http://engine/ovirt-engine/services/health

and that command also hangs forever. In the engine log the last related entry seem to be:

2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.utils.servlet.LocaleFilter] (default task-7) [] Incoming locale 'en-US'. Filter determined locale to be 'en-US' 2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] Health Status servlet: entry 2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] Calling CheckDBConnection query

Has anyone else also encountered that?

Regards, Marcin

[1] ovirt-engine-4.4.4.3-0.0.master.20201126133903.gitc2c805a2662.el8.noarch [2]

https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9c... [3]

https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9c...

-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

I can see something I was recently touching. Removal of reactive stream from vdsm-jsonrpc-java. Checking... Artur On Fri, Nov 27, 2020 at 11:47 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:

On 11/27/20 11:24 AM, Martin Perina wrote:

...

Hi,

the health status is pretty stupid simple call to database:

https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/se...

...

< https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/se...

https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bl...

...

< https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bl...

https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/da...

...

< https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/da...

...

< https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/common...

So it should definitely not hang forever unless there is some serious issue in the engine start up or PostgreSQL database. Could you please share logs? Especially interesting would server.log and engine.log from /var/log/ovirt-engine Well, after some discussion with Artur and trying some workarounds, the

https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/common... problem magically disappeared on my servers, but there's one OST gating run in CI that suffered from the same problem:

https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_gate...

https://jenkins.ovirt.org/job/ovirt-system-tests_gate/937/artifact/basic-sui...

https://jenkins.ovirt.org/job/ovirt-system-tests_gate/937/artifact/basic-sui...

...

Martin

On Thu, Nov 26, 2020 at 4:24 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:

Hi,

I'm trying out (I think) the latest build of ovirt-engine in OST [1] and the basic suite fails when we do engine reconfiguration and then restart the service [2]. After restarting we wait on the health check endpoint status here

[3].

...

This however ends with a timeout. I tried running manually:

curl -D - http://engine/ovirt-engine/services/health <http://engine/ovirt-engine/services/health>

and that command also hangs forever. In the engine log the last related entry seem to be:

2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.utils.servlet.LocaleFilter] (default task-7) [] Incoming locale 'en-US'. Filter determined locale to be 'en-US' 2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] Health Status servlet: entry 2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] Calling CheckDBConnection query

Has anyone else also encountered that?

Regards, Marcin

[1]

ovirt-engine-4.4.4.3-0.0.master.20201126133903.gitc2c805a2662.el8.noarch

...

[2]

https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9c...

...

<

https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9c...

...

[3]

https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9c...

...

<

https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9c...

...

-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

-- Artur Socha Senior Software Engineer, RHV Red Hat

It would be good to notify on devel list when there are breaking changes across multiple components, development envs usually do not update everything. If there is an actual incompatibility like in this case, it would also be good to not only require new version, but properly conflict with an old one. Currently official composes fail too because old engines have vdsm-jsonrpc-java >= 1.5.4 which pulls in 1.6.0 just fine Thanks, michal

On 27 Nov 2020, at 12:04, Artur Socha <asocha@redhat.com> wrote:

In this run there is ovirt-engine-* used with git c2c805a2662, however, vdsm-jsonrpc-client is in version 1.6.0 which has a breaking change of removing reactive streams support in favor to java.util.concurrent.FLOW [1] which is handled by engine's commit ff3aa4da956 [2]

[1] https://gerrit.ovirt.org/#/c/vdsm-jsonrpc-java/+/109916/ <https://gerrit.ovirt.org/#/c/vdsm-jsonrpc-java/+/109916/> [2] https://gerrit.ovirt.org/#/c/ovirt-engine/+/112347/ <https://gerrit.ovirt.org/#/c/ovirt-engine/+/112347/>

Artur

On Fri, Nov 27, 2020 at 11:49 AM Artur Socha <asocha@redhat.com <mailto:asocha@redhat.com>> wrote: I can see something I was recently touching. Removal of reactive stream from vdsm-jsonrpc-java. Checking... Artur

On Fri, Nov 27, 2020 at 11:47 AM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:

...

Hi,

the health status is pretty stupid simple call to database:

https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/se... <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/services/src/main/java/org/ovirt/engine/core/services/HealthStatus.java> <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/se... <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/services/src/main/java/org/ovirt/engine/core/services/HealthStatus.java>> https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bl... <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/CheckDBConnectionQuery.java#L21> <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bl... <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/CheckDBConnectionQuery.java#L21>> https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/da... <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/dal/src/main/java/org/ovirt/engine/core/dal/dbbroker/DbConnectionUtil.java#L33> <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/da... <https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/dal/src/main/java/org/ovirt/engine/core/dal/dbbroker/DbConnectionUtil.java#L33>> https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/common... <https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/common_sp.sql#L421> <https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/common... <https://github.com/oVirt/ovirt-engine/blob/master/packaging/dbscripts/common_sp.sql#L421>>

So it should definitely not hang forever unless there is some serious issue in the engine start up or PostgreSQL database. Could you please share logs? Especially interesting would server.log and engine.log from /var/log/ovirt-engine Well, after some discussion with Artur and trying some workarounds, the

On 11/27/20 11:24 AM, Martin Perina wrote: problem magically disappeared on my servers, but there's one OST gating run in CI that suffered from the same problem:

https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_gate... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_gate/detail/ovirt-system-tests_gate/937/pipeline#step-240-log-1226>

https://jenkins.ovirt.org/job/ovirt-system-tests_gate/937/artifact/basic-sui... <https://jenkins.ovirt.org/job/ovirt-system-tests_gate/937/artifact/basic-suit-master.el7.x86_64/test_logs/basic-suite-master/lago-basic-suite-master-engine/_var_log/ovirt-engine/server.log/*view*/>

https://jenkins.ovirt.org/job/ovirt-system-tests_gate/937/artifact/basic-sui... <https://jenkins.ovirt.org/job/ovirt-system-tests_gate/937/artifact/basic-suit-master.el7.x86_64/test_logs/basic-suite-master/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log/*view*/>

...

Martin

On Thu, Nov 26, 2020 at 4:24 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote:

Hi,

I'm trying out (I think) the latest build of ovirt-engine in OST [1] and the basic suite fails when we do engine reconfiguration and then restart the service [2]. After restarting we wait on the health check endpoint status here [3]. This however ends with a timeout. I tried running manually:

curl -D - http://engine/ovirt-engine/services/health <http://engine/ovirt-engine/services/health> <http://engine/ovirt-engine/services/health <http://engine/ovirt-engine/services/health>>

and that command also hangs forever. In the engine log the last related entry seem to be:

2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.utils.servlet.LocaleFilter] (default task-7) [] Incoming locale 'en-US'. Filter determined locale to be 'en-US' 2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] Health Status servlet: entry 2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] Calling CheckDBConnection query

Has anyone else also encountered that?

Regards, Marcin

[1] ovirt-engine-4.4.4.3-0.0.master.20201126133903.gitc2c805a2662.el8.noarch [2] https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9c... <https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9cda2d94e36e2be/basic-suite-master/test-scenarios/test_001_initialize_engine.py#L88> <https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9c... <https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9cda2d94e36e2be/basic-suite-master/test-scenarios/test_001_initialize_engine.py#L88>> [3] https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9c... <https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9cda2d94e36e2be/ost_utils/ost_utils/pytest/fixtures/engine.py#L160> <https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9c... <https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9cda2d94e36e2be/ost_utils/ost_utils/pytest/fixtures/engine.py#L160>>

-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

-- Artur Socha Senior Software Engineer, RHV Red Hat

-- Artur Socha Senior Software Engineer, RHV Red Hat _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/DTY2FJHJ3YBI4Z...