On Fri, Dec 24, 2021 at 1:06 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:

Hi All,

OST currently fails all the time during engine setup. Here's a piece of ansible log that's seen repeatedly and I think describes the problem:

11:07:54 E "engine-config", 11:07:54 E "-s", 11:07:54 E "OvfUpdateIntervalInMinutes=10" 11:07:54 E ], 11:07:54 E "delta": "0:00:01.142926", 11:07:54 E "end": "2021-12-24 11:06:37.894810", 11:07:54 E "invocation": { 11:07:54 E "module_args": { 11:07:54 E "_raw_params": "engine-config -s OvfUpdateIntervalInMinutes='10' ", 11:07:54 E "_uses_shell": false, 11:07:54 E "argv": null, 11:07:54 E "chdir": null, 11:07:54 E "creates": null, 11:07:54 E "executable": null, 11:07:54 E "removes": null, 11:07:54 E "stdin": null, 11:07:54 E "stdin_add_newline": true, 11:07:54 E "strip_empty_ends": true, 11:07:54 E "warn": false 11:07:54 E } 11:07:54 E }, 11:07:54 E "item": { 11:07:54 E "key": "OvfUpdateIntervalInMinutes", 11:07:54 E "value": "10" 11:07:54 E }, 11:07:54 E "msg": "non-zero return code", 11:07:54 E "rc": 1, 11:07:54 E "start": "2021-12-24 11:06:36.751884", 11:07:54 E "stderr": "Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false", 11:07:54 E "stderr_lines": [ 11:07:54 E "Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false" 11:07:54 E ], 11:07:54 E "stdout": "Error loading module from /usr/share/ovirt-engine/modules/common/org/springframework/main/module.xml", 11:07:54 E "stdout_lines": [ 11:07:54 E "Error loading module from /usr/share/ovirt-engine/modules/common/org/springframework/main/module.xml"

We do set some config values for the engine in OST when running engine-setup. I tried commenting these out, but then engine failed health check anyway:

"Status code was 503 and not [200]: HTTP Error 503: Service Unavailable"

The last working set of OST images was the one from Dec 23, 2021 2:05:08 AM. The first broken one is from Dec 24, 2021 2:05:09 AM. The shipped ovirt-engine RPMs doesn't seem to contain any important changes for these two sets, but AFAICS the newer ovirt-dependencies RPM did take in a couple of patches that look suspicious [1][2][3]. The patches were merged on November 16th, but it seems they were first used in that broken set from Dec 24 (the one from Dec 23 seems to contain ovirt-dependencies RPM based on this [4] commit).

I wanted to try out an older version of ovirt-dependencies, but I think they were wiped out from resources.ovirt.org.

I will disable cyclic el8stream OST runs for now, cause all of them fail. If there's anyone working and able to make a build with those patches reverted and test that out, please ping me and I'll re-enable them.

Regards, Marcin

[1] https://gerrit.ovirt.org/c/ovirt-dependencies/+/114699 [2] https://gerrit.ovirt.org/c/ovirt-dependencies/+/113877

This is the one that broke us - it replaced a set of shell scripts with maven. These scripts also created: # ls -l /usr/share/java/ovirt-dependencies/4.4 total 0 lrwxrwxrwx. 1 root root 24 Nov 4 18:00 gwt-servlet.jar -> ../gwt-servlet-2.9.0.jar lrwxrwxrwx. 1 root root 31 Nov 4 18:00 spring-aop.jar -> ../spring-aop-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 33 Nov 4 18:00 spring-beans.jar -> ../spring-beans-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 35 Nov 4 18:00 spring-context.jar -> ../spring-context-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 32 Nov 4 18:00 spring-core.jar -> ../spring-core-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 38 Nov 4 18:00 spring-expression.jar -> ../spring-expression-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 38 Nov 4 18:00 spring-instrument.jar -> ../spring-instrument-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 32 Nov 4 18:00 spring-jdbc.jar -> ../spring-jdbc-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 32 Nov 4 18:00 spring-test.jar -> ../spring-test-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 30 Nov 4 18:00 spring-tx.jar -> ../spring-tx-5.0.4.RELEASE.jar But now there is no '4.4' directory. The engine spec still has: %global ovirt_dependencies ovirt-dependencies/4.4 despite being at 4.5 and despite also ovirt-dependencies being bumped to 4.5 a few days ago. Not sure what the exact plans are regarding this change - if it's just a bug, I'll let Martin/Artur fix (by perhaps adding a 4.5 directory with links and point the 4.5 engine there, or make it provide both - not sure what the plans are). I did not try to revert, anyway, because quite many other related patches were merged since, including the move to github/copr. For now, I pushed this simple workaround to OST [1]: # ln -s . /usr/share/java/ovirt-dependencies/4.4 # ls -l /usr/share/java/ovirt-dependencies/ total 4724 lrwxrwxrwx. 1 root root 1 Dec 27 08:28 4.4 -> . -rw-r--r--. 1 root root 360096 Dec 26 23:39 spring-aop.jar -rw-r--r--. 1 root root 654084 Dec 26 23:39 spring-beans.jar -rw-r--r--. 1 root root 1079129 Dec 26 23:39 spring-context.jar -rw-r--r--. 1 root root 1216476 Dec 26 23:39 spring-core.jar -rw-r--r--. 1 root root 263472 Dec 26 23:39 spring-expression.jar -rw-r--r--. 1 root root 1452 Dec 26 23:39 spring-instrument.jar -rw-r--r--. 1 root root 15898 Dec 26 23:39 spring-jcl.jar -rw-r--r--. 1 root root 391565 Dec 26 23:39 spring-jdbc.jar -rw-r--r--. 1 root root 590394 Dec 26 23:39 spring-test.jar -rw-r--r--. 1 root root 249289 Dec 26 23:39 spring-tx.jar With this: # ls -l /usr/share/ovirt-engine/modules/common/org/springframework/main/ total 4 -rw-r--r--. 1 root root 737 Dec 23 15:29 module.xml lrwxrwxrwx. 1 root root 53 Dec 23 15:42 spring-aop.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-aop.jar lrwxrwxrwx. 1 root root 53 Dec 23 15:42 spring-asm.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-asm.jar lrwxrwxrwx. 1 root root 55 Dec 23 15:42 spring-beans.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-beans.jar lrwxrwxrwx. 1 root root 57 Dec 23 15:42 spring-context.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-context.jar lrwxrwxrwx. 1 root root 54 Dec 23 15:42 spring-core.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-core.jar lrwxrwxrwx. 1 root root 60 Dec 23 15:42 spring-expression.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-expression.jar lrwxrwxrwx. 1 root root 60 Dec 23 15:42 spring-instrument.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-instrument.jar lrwxrwxrwx. 1 root root 54 Dec 23 15:42 spring-jdbc.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-jdbc.jar lrwxrwxrwx. 1 root root 52 Dec 23 15:42 spring-tx.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-tx.jar This unbreaks all the links despite one: spring-asm.jar is still missing. No idea why. It's missing also on my slightly older setup from ovirt-dependencies commit ca19602 (and engine 3326bfea06). I guess it might be unrelated. I am testing [1] now locally. It already passed test_add_cpu_profile so seems promising. Best regards, [1] https://gerrit.ovirt.org/c/ovirt-system-tests/+/118150

[3] https://gerrit.ovirt.org/c/ovirt-dependencies/+/114654 [4] https://gerrit.ovirt.org/c/ovirt-dependencies/+/117459 _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MON3B5NSSXBL2B...

-- Didi