master and 4.1 dwhd Fails to Start (was ovirt_experimental_master Fails: PostgreSQL is not Accessible During Engine Setup)
Hello All. We checked this with did and postgres connection error is not an error (although it prints stacktrace... cannot we not print stacktraces, please, for antyhing that we handle in code... it is really confusing when you need to find the root cause). The test is checking for dwhd to be up using systemd: testlib.assert_true_within_short( lambda: engine.service('ovirt-engine-dwhd').alive() ) that runs: /usr/bin/systemctl status --lines=0 ovirt-engine-dwhd lago.ssh: DEBUG: Command 90e98548 on lago-basic-suite-master-engine returned with 3 lago.ssh: DEBUG: Command 90e98548 on lago-basic-suite-master-engine output: ● ovirt-engine-dwhd.service - oVirt Engine Data Warehouse Loaded: loaded (/usr/lib/systemd/system/ovirt-engine-dwhd.service; enabled; vendor preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Tue 2017-01-17 07:33:23 EST; 3min 4s ago Main PID: 22448 (code=exited, status=1/FAILURE) CGroup: /system.slice/ovirt-engine-dwhd.service dwhd log [1] has the following error: Error: Could not find or load main class ovirt_engine_dwh.historyetl_4_2.HistoryETL so this looks to be the actual problem. The latest job failed with this is [2]. This also affects 4.1, e.g. [3]. [1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4791/artifa... [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4791/ [3] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/287 -- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
Hello All. With Shirly help we found that 4.1 was using master and when ovirt_engine_dwh was branched today 4.1 job was not updated. This resulted in 4.1 repo be poisoned with 4.1 dwh rpm. I manually cleared it and waiting for test results, looks like we have a pile up on Jenkins due to lot of patches merged. For master I believe there are still some problems introduced so it will continue to fail "Error: Could not find or load main class ovirt_engine_dwh.historyetl_4_2.HistoryETL" till we find and fix the root cause. Anton. On Tue, Jan 17, 2017 at 2:18 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello All.
We checked this with did and postgres connection error is not an error (although it prints stacktrace... cannot we not print stacktraces, please, for antyhing that we handle in code... it is really confusing when you need to find the root cause).
The test is checking for dwhd to be up using systemd:
testlib.assert_true_within_short( lambda: engine.service('ovirt-engine-dwhd').alive() )
that runs:
/usr/bin/systemctl status --lines=0 ovirt-engine-dwhd lago.ssh: DEBUG: Command 90e98548 on lago-basic-suite-master-engine returned with 3 lago.ssh: DEBUG: Command 90e98548 on lago-basic-suite-master-engine output: ● ovirt-engine-dwhd.service - oVirt Engine Data Warehouse Loaded: loaded (/usr/lib/systemd/system/ovirt-engine-dwhd.service; enabled; vendor preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Tue 2017-01-17 07:33:23 EST; 3min 4s ago Main PID: 22448 (code=exited, status=1/FAILURE) CGroup: /system.slice/ovirt-engine-dwhd.service
dwhd log [1] has the following error:
Error: Could not find or load main class ovirt_engine_dwh.historyetl_4_ 2.HistoryETL
so this looks to be the actual problem. The latest job failed with this is [2]. This also affects 4.1, e.g. [3].
[1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/4791/artifact/exported-artifacts/basic_suite_master. sh-el7/exported-artifacts/test_logs/basic-suite-master/ post-001_initialize_engine.py/lago-basic-suite-master- engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4791/ [3] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/287
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
I fixed the master bug. Is dwh in master still failing? Best regards, Shirly Radco BI Software Engineer Red Hat Israel Ltd. 34 Jerusalem Road Building A, 4th floor Ra'anana, Israel 4350109 On Tue, Jan 17, 2017 at 5:30 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello All.
With Shirly help we found that 4.1 was using master and when ovirt_engine_dwh was branched today 4.1 job was not updated. This resulted in 4.1 repo be poisoned with 4.1 dwh rpm. I manually cleared it and waiting for test results, looks like we have a pile up on Jenkins due to lot of patches merged.
For master I believe there are still some problems introduced so it will continue to fail "Error: Could not find or load main class ovirt_engine_dwh.historyetl_4_2.HistoryETL" till we find and fix the root cause.
Anton.
On Tue, Jan 17, 2017 at 2:18 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello All.
We checked this with did and postgres connection error is not an error (although it prints stacktrace... cannot we not print stacktraces, please, for antyhing that we handle in code... it is really confusing when you need to find the root cause).
The test is checking for dwhd to be up using systemd:
testlib.assert_true_within_short( lambda: engine.service('ovirt-engine-dwhd').alive() )
that runs:
/usr/bin/systemctl status --lines=0 ovirt-engine-dwhd lago.ssh: DEBUG: Command 90e98548 on lago-basic-suite-master-engine returned with 3 lago.ssh: DEBUG: Command 90e98548 on lago-basic-suite-master-engine output: ● ovirt-engine-dwhd.service - oVirt Engine Data Warehouse Loaded: loaded (/usr/lib/systemd/system/ovirt-engine-dwhd.service; enabled; vendor preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Tue 2017-01-17 07:33:23 EST; 3min 4s ago Main PID: 22448 (code=exited, status=1/FAILURE) CGroup: /system.slice/ovirt-engine-dwhd.service
dwhd log [1] has the following error:
Error: Could not find or load main class ovirt_engine_dwh.historyetl_4_ 2.HistoryETL
so this looks to be the actual problem. The latest job failed with this is [2]. This also affects 4.1, e.g. [3].
[1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma ster/4791/artifact/exported-artifacts/basic_suite_master.sh- el7/exported-artifacts/test_logs/basic-suite-master/post- 001_initialize_engine.py/lago-basic-suite-master-engine/_ var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimenta l_master/4791/ [3] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/287
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
Hello Shirly. Thanks for your fix. No. Today there are no things of dwh related failures in both master and 4.1 Anton. On Tue, Jan 17, 2017 at 9:26 PM, Shirly Radco <sradco@redhat.com> wrote:
I fixed the master bug. Is dwh in master still failing?
Best regards,
Shirly Radco
BI Software Engineer Red Hat Israel Ltd. 34 Jerusalem Road Building A, 4th floor Ra'anana, Israel 4350109
On Tue, Jan 17, 2017 at 5:30 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello All.
With Shirly help we found that 4.1 was using master and when ovirt_engine_dwh was branched today 4.1 job was not updated. This resulted in 4.1 repo be poisoned with 4.1 dwh rpm. I manually cleared it and waiting for test results, looks like we have a pile up on Jenkins due to lot of patches merged.
For master I believe there are still some problems introduced so it will continue to fail "Error: Could not find or load main class ovirt_engine_dwh.historyetl_4_2.HistoryETL" till we find and fix the root cause.
Anton.
On Tue, Jan 17, 2017 at 2:18 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello All.
We checked this with did and postgres connection error is not an error (although it prints stacktrace... cannot we not print stacktraces, please, for antyhing that we handle in code... it is really confusing when you need to find the root cause).
The test is checking for dwhd to be up using systemd:
testlib.assert_true_within_short( lambda: engine.service('ovirt-engine-dwhd').alive() )
that runs:
/usr/bin/systemctl status --lines=0 ovirt-engine-dwhd lago.ssh: DEBUG: Command 90e98548 on lago-basic-suite-master-engine returned with 3 lago.ssh: DEBUG: Command 90e98548 on lago-basic-suite-master-engine output: ● ovirt-engine-dwhd.service - oVirt Engine Data Warehouse Loaded: loaded (/usr/lib/systemd/system/ovirt-engine-dwhd.service; enabled; vendor preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Tue 2017-01-17 07:33:23 EST; 3min 4s ago Main PID: 22448 (code=exited, status=1/FAILURE) CGroup: /system.slice/ovirt-engine-dwhd.service
dwhd log [1] has the following error:
Error: Could not find or load main class ovirt_engine_dwh.historyetl_4_ 2.HistoryETL
so this looks to be the actual problem. The latest job failed with this is [2]. This also affects 4.1, e.g. [3].
[1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma ster/4791/artifact/exported-artifacts/basic_suite_master.sh- el7/exported-artifacts/test_logs/basic-suite-master/post-001 _initialize_engine.py/lago-basic-suite-master-engine/_var_ log/ovirt-engine-dwh/ovirt-engine-dwhd.log [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimenta l_master/4791/ [3] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/287
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
On Wed, Jan 18, 2017 at 3:09 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello Shirly.
Thanks for your fix. No. Today there are no things of dwh related failures in both master and 4.1
Would be great if it could be published, so us mere mortals could enjoy the fix and run o-s-t as well :-) Y.
Anton.
On Tue, Jan 17, 2017 at 9:26 PM, Shirly Radco <sradco@redhat.com> wrote:
I fixed the master bug. Is dwh in master still failing?
Best regards,
Shirly Radco
BI Software Engineer Red Hat Israel Ltd. 34 Jerusalem Road Building A, 4th floor Ra'anana, Israel 4350109
On Tue, Jan 17, 2017 at 5:30 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello All.
With Shirly help we found that 4.1 was using master and when ovirt_engine_dwh was branched today 4.1 job was not updated. This resulted in 4.1 repo be poisoned with 4.1 dwh rpm. I manually cleared it and waiting for test results, looks like we have a pile up on Jenkins due to lot of patches merged.
For master I believe there are still some problems introduced so it will continue to fail "Error: Could not find or load main class ovirt_engine_dwh.historyetl_4_2.HistoryETL" till we find and fix the root cause.
Anton.
On Tue, Jan 17, 2017 at 2:18 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello All.
We checked this with did and postgres connection error is not an error (although it prints stacktrace... cannot we not print stacktraces, please, for antyhing that we handle in code... it is really confusing when you need to find the root cause).
The test is checking for dwhd to be up using systemd:
testlib.assert_true_within_short( lambda: engine.service('ovirt-engine-dwhd').alive() )
that runs:
/usr/bin/systemctl status --lines=0 ovirt-engine-dwhd lago.ssh: DEBUG: Command 90e98548 on lago-basic-suite-master-engine returned with 3 lago.ssh: DEBUG: Command 90e98548 on lago-basic-suite-master-engine output: ● ovirt-engine-dwhd.service - oVirt Engine Data Warehouse Loaded: loaded (/usr/lib/systemd/system/ovirt-engine-dwhd.service; enabled; vendor preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Tue 2017-01-17 07:33:23 EST; 3min 4s ago Main PID: 22448 (code=exited, status=1/FAILURE) CGroup: /system.slice/ovirt-engine-dwhd.service
dwhd log [1] has the following error:
Error: Could not find or load main class ovirt_engine_dwh.historyetl_4_ 2.HistoryETL
so this looks to be the actual problem. The latest job failed with this is [2]. This also affects 4.1, e.g. [3].
[1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma ster/4791/artifact/exported-artifacts/basic_suite_master.sh- el7/exported-artifacts/test_logs/basic-suite-master/post-001 _initialize_engine.py/lago-basic-suite-master-engine/_var_lo g/ovirt-engine-dwh/ovirt-engine-dwhd.log [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimenta l_master/4791/ [3] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/287
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Wed, Jan 18, 2017 at 10:59 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Wed, Jan 18, 2017 at 3:09 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello Shirly.
Thanks for your fix. No. Today there are no things of dwh related failures in both master and 4.1
Would be great if it could be published, so us mere mortals could enjoy the fix and run o-s-t as well :-)
What do you mean? once the fix is merged it is published to latest.tested repos. The default 4.1 repo for O-S-T is using the 4.1-pre repo, so until that one is not refreshed you won't see the fix.
Y.
Anton.
On Tue, Jan 17, 2017 at 9:26 PM, Shirly Radco <sradco@redhat.com> wrote:
I fixed the master bug. Is dwh in master still failing?
Best regards,
Shirly Radco
BI Software Engineer Red Hat Israel Ltd. 34 Jerusalem Road Building A, 4th floor Ra'anana, Israel 4350109
On Tue, Jan 17, 2017 at 5:30 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello All.
With Shirly help we found that 4.1 was using master and when ovirt_engine_dwh was branched today 4.1 job was not updated. This resulted in 4.1 repo be poisoned with 4.1 dwh rpm. I manually cleared it and waiting for test results, looks like we have a pile up on Jenkins due to lot of patches merged.
For master I believe there are still some problems introduced so it will continue to fail "Error: Could not find or load main class ovirt_engine_dwh.historyetl_4_2.HistoryETL" till we find and fix the root cause.
Anton.
On Tue, Jan 17, 2017 at 2:18 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello All.
We checked this with did and postgres connection error is not an error (although it prints stacktrace... cannot we not print stacktraces, please, for antyhing that we handle in code... it is really confusing when you need to find the root cause).
The test is checking for dwhd to be up using systemd:
testlib.assert_true_within_short( lambda: engine.service('ovirt-engine-dwhd').alive() )
that runs:
/usr/bin/systemctl status --lines=0 ovirt-engine-dwhd lago.ssh: DEBUG: Command 90e98548 on lago-basic-suite-master-engine returned with 3 lago.ssh: DEBUG: Command 90e98548 on lago-basic-suite-master-engine output: ● ovirt-engine-dwhd.service - oVirt Engine Data Warehouse Loaded: loaded (/usr/lib/systemd/system/ovirt-engine-dwhd.service; enabled; vendor preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Tue 2017-01-17 07:33:23 EST; 3min 4s ago Main PID: 22448 (code=exited, status=1/FAILURE) CGroup: /system.slice/ovirt-engine-dwhd.service
dwhd log [1] has the following error:
Error: Could not find or load main class ovirt_engine_dwh.historyetl_4_ 2.HistoryETL
so this looks to be the actual problem. The latest job failed with this is [2]. This also affects 4.1, e.g. [3].
[1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma ster/4791/artifact/exported-artifacts/basic_suite_master.sh- el7/exported-artifacts/test_logs/basic-suite-master/post-001 _initialize_engine.py/lago-basic-suite-master-engine/_var_lo g/ovirt-engine-dwh/ovirt-engine-dwhd.log [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimenta l_master/4791/ [3] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/287
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
participants (4)
-
Anton Marchukov -
Eyal Edri -
Shirly Radco -
Yaniv Kaul