[ OST Failure Report ] [ oVirt Master ] [ upgrade suites failed! ] [ 15/10/17 ]
Hi all, The following tests are failing both of the upgrade suites in OST (upgrade_from_release and upgrade_from_prevrelease). *Link to console:* ovirt-master_change-queue-tester/3146/console <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> *Link to test logs:* - upgrade-from-release-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> - upgrade-from-prevrelease-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 *Please note that every patch that is based on top of the patch above was not deployed to the tested repo.* *Error snippet from engine setup log:* 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies self._clearZombies() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies 'Failed to clear zombie commands. ' RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem -- DANIEL BELENKY RHV DEVOPS EMEA VIRTUALIZATION R&D <https://red.ht/sig>
can someone address this issue? every patch to *ovirt-engine* that is based on top of this patch is failing OST and* won't deploy to the tested repo*. On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
Hi all, The following tests are failing both of the upgrade suites in OST (upgrade_from_release and upgrade_from_prevrelease).
*Link to console:* ovirt-master_change-queue-tester/3146/console <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> *Link to test logs:* - upgrade-from-release-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> - upgrade-from-prevrelease-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 *Please note that every patch that is based on top of the patch above was not deployed to the tested repo.*
*Error snippet from engine setup log:*
2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies self._clearZombies() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies 'Failed to clear zombie commands. ' RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
-- DANIEL BELENKY RHV DEVOPS EMEA VIRTUALIZATION R&D <https://red.ht/sig>
On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
can someone address this issue? every patch to *ovirt-engine* that is based on top of this patch is failing OST and* won't deploy to the tested repo*.
On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
Hi all, The following tests are failing both of the upgrade suites in OST (upgrade_from_release and upgrade_from_prevrelease).
*Link to console:* ovirt-master_change-queue-tester/3146/console <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> *Link to test logs:* - upgrade-from-release-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> - upgrade-from-prevrelease-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 *Please note that every patch that is based on top of the patch above was not deployed to the tested repo.*
*Error snippet from engine setup log:*
Please add a direct link next time, if possible. This is it: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/... And a bit above the snippet below, there is: 2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.upgrade.asynctasks plugin.execute:926 execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr: /usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies self._clearZombies() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies 'Failed to clear zombie commands. ' RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem
With [1], taskcleaner.sh sources generate-pgpass.sh .
generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, is not yet upgraded (at the point of above failure). generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a path relative to "$0". In master it does not, but we now upgrade and it does. This, in principle, is the core of the bug: A file, such as generate-pgpass.sh, that's supposed to be sourced from some other files, should not by itself source other files that are relative to "$0", because it can't know what "$0" is - it's the path of the script sourcing it, not of itself. It seems like luckily we were not affected by this in 4.1, because all of the files that sourced generate-pgpass.sh were together with it in the same directory. But with [1], taskcleaner does too now, and is in a different directory. Not sure what's the best solution: - revert [1] (and introduce it later on, in 4.3) - patch 4.1's generate-pgpass.sh and require the fixed 4.1 version in 4.2 setup - Somehow trick everything to work together? Not sure. Seems like you can't set $0. [1] https://gerrit.ovirt.org/82511 Regards,
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
can someone address this issue? every patch to *ovirt-engine* that is based on top of this patch is failing OST and* won't deploy to the tested repo*.
On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
Hi all, The following tests are failing both of the upgrade suites in OST (upgrade_from_release and upgrade_from_prevrelease).
*Link to console:* ovirt-master_change-queue-tester/3146/console <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> *Link to test logs:* - upgrade-from-release-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> - upgrade-from-prevrelease-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 *Please note that every patch that is based on top of the patch above was not deployed to the tested repo.*
*Error snippet from engine setup log:*
Please add a direct link next time, if possible. This is it:
http://jenkins.ovirt.org/job/ovirt-master_change-queue- tester/3146/artifact/exported-artifacts/upgrade-from- release-suit-master-el7/test_logs/upgrade-from-release- suite-master/post-001_upgrade_engine.py/lago-upgrade-from- release-suite-master-engine/_var_log/ovirt-engine/setup/ ovirt-engine-setup-20171013222617-73f0df.log
And a bit above the snippet below, there is:
2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_ setup.ovirt_engine.upgrade.asynctasks plugin.execute:926 execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr:
/usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory
2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies self._clearZombies() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies 'Failed to clear zombie commands. ' RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem
With [1], taskcleaner.sh sources generate-pgpass.sh .
generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, is not yet upgraded (at the point of above failure).
generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a path relative to "$0". In master it does not, but we now upgrade and it does.
This, in principle, is the core of the bug:
A file, such as generate-pgpass.sh, that's supposed to be sourced from some other files, should not by itself source other files that are relative to "$0", because it can't know what "$0" is - it's the path of the script sourcing it, not of itself.
It seems like luckily we were not affected by this in 4.1, because all of the files that sourced generate-pgpass.sh were together with it in the same directory. But with [1], taskcleaner does too now, and is in a different directory.
Not sure what's the best solution:
- revert [1] (and introduce it later on, in 4.3)
- patch 4.1's generate-pgpass.sh and require the fixed 4.1 version in 4.2 setup
- Somehow trick everything to work together? Not sure. Seems like you can't set $0.
The same bug exists with unlock_entity: https://gerrit.ovirt.org/82615 So we should probably revert both.
Regards,
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
-- Didi
On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
can someone address this issue? every patch to *ovirt-engine* that is based on top of this patch is failing OST and* won't deploy to the tested repo*.
On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
Hi all, The following tests are failing both of the upgrade suites in OST (upgrade_from_release and upgrade_from_prevrelease).
*Link to console:* ovirt-master_change-queue-tester/3146/console <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> *Link to test logs:* - upgrade-from-release-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> - upgrade-from-prevrelease-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 *Please note that every patch that is based on top of the patch above was not deployed to the tested repo.*
*Error snippet from engine setup log:*
Please add a direct link next time, if possible. This is it:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste r/3146/artifact/exported-artifacts/upgrade-from-release- suit-master-el7/test_logs/upgrade-from-release-suite- master/post-001_upgrade_engine.py/lago-upgrade-from-release- suite-master-engine/_var_log/ovirt-engine/setup/ovirt- engine-setup-20171013222617-73f0df.log
And a bit above the snippet below, there is:
2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set up.ovirt_engine.upgrade.asynctasks plugin.execute:926 execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr:
/usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory
2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies self._clearZombies() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies 'Failed to clear zombie commands. ' RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem
With [1], taskcleaner.sh sources generate-pgpass.sh .
generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, is not yet upgraded (at the point of above failure).
generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a path relative to "$0". In master it does not, but we now upgrade and it does.
This, in principle, is the core of the bug:
A file, such as generate-pgpass.sh, that's supposed to be sourced from some other files, should not by itself source other files that are relative to "$0", because it can't know what "$0" is - it's the path of the script sourcing it, not of itself.
It seems like luckily we were not affected by this in 4.1, because all of the files that sourced generate-pgpass.sh were together with it in the same directory. But with [1], taskcleaner does too now, and is in a different directory.
Not sure what's the best solution:
- revert [1] (and introduce it later on, in 4.3)
- patch 4.1's generate-pgpass.sh and require the fixed 4.1 version in 4.2 setup
- Somehow trick everything to work together? Not sure. Seems like you can't set $0.
The same bug exists with unlock_entity:
https://gerrit.ovirt.org/82615
So we should probably revert both.
I tend to agree. Doesn't mean we cannot fix this for 4.2, but let's revert for the meantime. Y.
Regards,
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Didi, thanks for addressing this issue and the detailed explanation. I'll make sure I attach all relevant logs next time, sorry for that! On Mon, Oct 16, 2017 at 10:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
can someone address this issue? every patch to *ovirt-engine* that is based on top of this patch is failing OST and* won't deploy to the tested repo*.
On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
Hi all, The following tests are failing both of the upgrade suites in OST (upgrade_from_release and upgrade_from_prevrelease).
*Link to console:* ovirt-master_change-queue-tester/3146/console <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> *Link to test logs:* - upgrade-from-release-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> - upgrade-from-prevrelease-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 *Please note that every patch that is based on top of the patch above was not deployed to the tested repo.*
*Error snippet from engine setup log:*
Please add a direct link next time, if possible. This is it:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste r/3146/artifact/exported-artifacts/upgrade-from-release-suit -master-el7/test_logs/upgrade-from-release-suite-master/ post-001_upgrade_engine.py/lago-upgrade-from-release-suit e-master-engine/_var_log/ovirt-engine/setup/ovirt-engine- setup-20171013222617-73f0df.log
And a bit above the snippet below, there is:
2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set up.ovirt_engine.upgrade.asynctasks plugin.execute:926 execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr:
/usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory
2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies self._clearZombies() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies 'Failed to clear zombie commands. ' RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem
With [1], taskcleaner.sh sources generate-pgpass.sh .
generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, is not yet upgraded (at the point of above failure).
generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a path relative to "$0". In master it does not, but we now upgrade and it does.
This, in principle, is the core of the bug:
A file, such as generate-pgpass.sh, that's supposed to be sourced from some other files, should not by itself source other files that are relative to "$0", because it can't know what "$0" is - it's the path of the script sourcing it, not of itself.
It seems like luckily we were not affected by this in 4.1, because all of the files that sourced generate-pgpass.sh were together with it in the same directory. But with [1], taskcleaner does too now, and is in a different directory.
Not sure what's the best solution:
- revert [1] (and introduce it later on, in 4.3)
- patch 4.1's generate-pgpass.sh and require the fixed 4.1 version in 4.2 setup
- Somehow trick everything to work together? Not sure. Seems like you can't set $0.
The same bug exists with unlock_entity:
https://gerrit.ovirt.org/82615
So we should probably revert both.
I tend to agree. Doesn't mean we cannot fix this for 4.2, but let's revert for the meantime. Y.
Regards,
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- DANIEL BELENKY RHV DEVOPS EMEA VIRTUALIZATION R&D <https://red.ht/sig>
Fix on review https://gerrit.ovirt.org/#/c/82799/ On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
can someone address this issue? every patch to *ovirt-engine* that is based on top of this patch is failing OST and* won't deploy to the tested repo*.
On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
Hi all, The following tests are failing both of the upgrade suites in OST (upgrade_from_release and upgrade_from_prevrelease).
*Link to console:* ovirt-master_change-queue-tester/3146/console <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> *Link to test logs:* - upgrade-from-release-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> - upgrade-from-prevrelease-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 *Please note that every patch that is based on top of the patch above was not deployed to the tested repo.*
*Error snippet from engine setup log:*
Please add a direct link next time, if possible. This is it:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste r/3146/artifact/exported-artifacts/upgrade-from-release-suit -master-el7/test_logs/upgrade-from-release-suite-master/ post-001_upgrade_engine.py/lago-upgrade-from-release-suit e-master-engine/_var_log/ovirt-engine/setup/ovirt-engine- setup-20171013222617-73f0df.log
And a bit above the snippet below, there is:
2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set up.ovirt_engine.upgrade.asynctasks plugin.execute:926 execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr:
/usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory
2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies self._clearZombies() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies 'Failed to clear zombie commands. ' RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem
With [1], taskcleaner.sh sources generate-pgpass.sh .
generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, is not yet upgraded (at the point of above failure).
generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a path relative to "$0". In master it does not, but we now upgrade and it does.
This, in principle, is the core of the bug:
A file, such as generate-pgpass.sh, that's supposed to be sourced from some other files, should not by itself source other files that are relative to "$0", because it can't know what "$0" is - it's the path of the script sourcing it, not of itself.
It seems like luckily we were not affected by this in 4.1, because all of the files that sourced generate-pgpass.sh were together with it in the same directory. But with [1], taskcleaner does too now, and is in a different directory.
Not sure what's the best solution:
- revert [1] (and introduce it later on, in 4.3)
- patch 4.1's generate-pgpass.sh and require the fixed 4.1 version in 4.2 setup
- Somehow trick everything to work together? Not sure. Seems like you can't set $0.
The same bug exists with unlock_entity:
https://gerrit.ovirt.org/82615
So we should probably revert both.
I tend to agree. Doesn't mean we cannot fix this for 4.2, but let's revert for the meantime. Y.
Regards,
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova <mvoglova@redhat.com> wrote:
Fix on review https://gerrit.ovirt.org/#/c/82799/
That's indeed a related patch, but not sure how it solves current failure.
On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
can someone address this issue? every patch to *ovirt-engine* that is based on top of this patch is failing OST and* won't deploy to the tested repo*.
On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
Hi all, The following tests are failing both of the upgrade suites in OST (upgrade_from_release and upgrade_from_prevrelease).
*Link to console:* ovirt-master_change-queue-tester/3146/console <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> *Link to test logs:* - upgrade-from-release-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> - upgrade-from-prevrelease-suit-master-el7 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 *Please note that every patch that is based on top of the patch above was not deployed to the tested repo.*
*Error snippet from engine setup log:*
Please add a direct link next time, if possible. This is it:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste r/3146/artifact/exported-artifacts/upgrade-from-release-suit -master-el7/test_logs/upgrade-from-release-suite-master/post -001_upgrade_engine.py/lago-upgrade-from-release-suite- master-engine/_var_log/ovirt-engine/setup/ovirt-engine-setu p-20171013222617-73f0df.log
And a bit above the snippet below, there is:
2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set up.ovirt_engine.upgrade.asynctasks plugin.execute:926 execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr:
/usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory
2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies self._clearZombies() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies 'Failed to clear zombie commands. ' RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem
With [1], taskcleaner.sh sources generate-pgpass.sh .
generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, is not yet upgraded (at the point of above failure).
generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a path relative to "$0". In master it does not, but we now upgrade and it does.
This, in principle, is the core of the bug:
A file, such as generate-pgpass.sh, that's supposed to be sourced from some other files, should not by itself source other files that are relative to "$0", because it can't know what "$0" is - it's the path of the script sourcing it, not of itself.
It seems like luckily we were not affected by this in 4.1, because all of the files that sourced generate-pgpass.sh were together with it in the same directory. But with [1], taskcleaner does too now, and is in a different directory.
Not sure what's the best solution:
- revert [1] (and introduce it later on, in 4.3)
- patch 4.1's generate-pgpass.sh and require the fixed 4.1 version in 4.2 setup
E.g.: https://gerrit.ovirt.org/82800
- Somehow trick everything to work together? Not sure. Seems like you can't set $0.
The same bug exists with unlock_entity:
https://gerrit.ovirt.org/82615
So we should probably revert both.
I tend to agree. Doesn't mean we cannot fix this for 4.2, but let's revert for the meantime. Y.
Regards,
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova <mvoglova@redhat.com> wrote:
Fix on review https://gerrit.ovirt.org/#/c/82799/
That's indeed a related patch, but not sure how it solves current failure.
Let's copy generate-pgpass.sh to packaging/setup/dbutils and source it from this location for taskcleaner.sh and unlock_entity.sh
On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
can someone address this issue? every patch to *ovirt-engine* that is based on top of this patch is failing OST and* won't deploy to the tested repo*.
On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
> Hi all, > The following tests are failing both of the upgrade suites in OST > (upgrade_from_release and upgrade_from_prevrelease). > > *Link to console:* ovirt-master_change-queue-tester/3146/console > <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> > *Link to test logs:* > - upgrade-from-release-suit-master-el7 > <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> > - upgrade-from-prevrelease-suit-master-el7 > <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> > *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 > *Please note that every patch that is based on top of the patch > above was not deployed to the tested repo.* > > *Error snippet from engine setup log:* >
Please add a direct link next time, if possible. This is it:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste r/3146/artifact/exported-artifacts/upgrade-from-release-suit -master-el7/test_logs/upgrade-from-release-suite-master/post -001_upgrade_engine.py/lago-upgrade-from-release-suite-maste r-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup- 20171013222617-73f0df.log
And a bit above the snippet below, there is:
2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set up.ovirt_engine.upgrade.asynctasks plugin.execute:926 execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr:
/usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory
2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception
> Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod > method['method']() > File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies > self._clearZombies() > File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies > 'Failed to clear zombie commands. ' > RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem > 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem > > With [1], taskcleaner.sh sources generate-pgpass.sh .
generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, is not yet upgraded (at the point of above failure).
generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a path relative to "$0". In master it does not, but we now upgrade and it does.
This, in principle, is the core of the bug:
A file, such as generate-pgpass.sh, that's supposed to be sourced from some other files, should not by itself source other files that are relative to "$0", because it can't know what "$0" is - it's the path of the script sourcing it, not of itself.
It seems like luckily we were not affected by this in 4.1, because all of the files that sourced generate-pgpass.sh were together with it in the same directory. But with [1], taskcleaner does too now, and is in a different directory.
Not sure what's the best solution:
- revert [1] (and introduce it later on, in 4.3)
- patch 4.1's generate-pgpass.sh and require the fixed 4.1 version in 4.2 setup
E.g.:
https://gerrit.ovirt.org/82800
- Somehow trick everything to work together? Not sure. Seems like you can't set $0.
The same bug exists with unlock_entity:
https://gerrit.ovirt.org/82615
So we should probably revert both.
I tend to agree. Doesn't mean we cannot fix this for 4.2, but let's revert for the meantime. Y.
Regards,
-- > > DANIEL BELENKY > > RHV DEVOPS > > EMEA VIRTUALIZATION R&D > <https://red.ht/sig> >
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Mon, Oct 16, 2017 at 9:44 AM, Martin Perina <mperina@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova <mvoglova@redhat.com> wrote:
Fix on review https://gerrit.ovirt.org/#/c/82799/
That's indeed a related patch, but not sure how it solves current failure.
Let's copy generate-pgpass.sh to packaging/setup/dbutils and source it from this location for taskcleaner.sh and unlock_entity.sh
After offline discussion merging https://gerrit.ovirt.org/82800 and then https://gerrit.ovirt.org/#/c/82799/ will fix the issue. Both patches are needed.
On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky@redhat.com> wrote:
> can someone address this issue? every patch to *ovirt-engine* that > is based on top of this patch is failing OST and* won't deploy to > the tested repo*. > > On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky@redhat.com > > wrote: > >> Hi all, >> The following tests are failing both of the upgrade suites in OST >> (upgrade_from_release and upgrade_from_prevrelease). >> >> *Link to console:* ovirt-master_change-queue-tester/3146/console >> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> >> *Link to test logs:* >> - upgrade-from-release-suit-master-el7 >> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> >> - upgrade-from-prevrelease-suit-master-el7 >> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> >> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 >> *Please note that every patch that is based on top of the patch >> above was not deployed to the tested repo.* >> >> *Error snippet from engine setup log:* >> > Please add a direct link next time, if possible. This is it:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste r/3146/artifact/exported-artifacts/upgrade-from-release-suit -master-el7/test_logs/upgrade-from-release-suite-master/post -001_upgrade_engine.py/lago-upgrade-from-release-suite-maste r-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-2017 1013222617-73f0df.log
And a bit above the snippet below, there is:
2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set up.ovirt_engine.upgrade.asynctasks plugin.execute:926 execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr:
/usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory
2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >> method['method']() >> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies >> self._clearZombies() >> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies >> 'Failed to clear zombie commands. ' >> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem >> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem >> >> With [1], taskcleaner.sh sources generate-pgpass.sh .
generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, is not yet upgraded (at the point of above failure).
generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a path relative to "$0". In master it does not, but we now upgrade and it does.
This, in principle, is the core of the bug:
A file, such as generate-pgpass.sh, that's supposed to be sourced from some other files, should not by itself source other files that are relative to "$0", because it can't know what "$0" is - it's the path of the script sourcing it, not of itself.
It seems like luckily we were not affected by this in 4.1, because all of the files that sourced generate-pgpass.sh were together with it in the same directory. But with [1], taskcleaner does too now, and is in a different directory.
Not sure what's the best solution:
- revert [1] (and introduce it later on, in 4.3)
- patch 4.1's generate-pgpass.sh and require the fixed 4.1 version in 4.2 setup
E.g.:
https://gerrit.ovirt.org/82800
- Somehow trick everything to work together? Not sure. Seems like you can't set $0.
The same bug exists with unlock_entity:
https://gerrit.ovirt.org/82615
So we should probably revert both.
I tend to agree. Doesn't mean we cannot fix this for 4.2, but let's revert for the meantime. Y.
Regards,
> -- >> >> DANIEL BELENKY >> >> RHV DEVOPS >> >> EMEA VIRTUALIZATION R&D >> <https://red.ht/sig> >> > > > > -- > > DANIEL BELENKY > > RHV DEVOPS > > EMEA VIRTUALIZATION R&D > <https://red.ht/sig> > > _______________________________________________ > Devel mailing list > Devel@ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel >
-- Didi
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Mon, Oct 16, 2017 at 11:01 AM, Miroslava Voglova <mvoglova@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:44 AM, Martin Perina <mperina@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova <mvoglova@redhat.com
wrote:
Fix on review https://gerrit.ovirt.org/#/c/82799/
That's indeed a related patch, but not sure how it solves current failure.
Let's copy generate-pgpass.sh to packaging/setup/dbutils and source it from this location for taskcleaner.sh and unlock_entity.sh
After offline discussion merging https://gerrit.ovirt.org/82800 and then https://gerrit.ovirt.org/#/c/82799/ will fix the issue. Both patches are needed.
So this means: Merge https://gerrit.ovirt.org/82800 . We might want to open a real 4.1 bug for this. Build 4.1.7 (or 4.1.8?) with it . Patch 4.2 engine-setup to require tools-4.1.7. Adding Sandro.
On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <didi@redhat.com > wrote:
> On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky@redhat.com > > wrote: > >> can someone address this issue? every patch to *ovirt-engine* that >> is based on top of this patch is failing OST and* won't deploy to >> the tested repo*. >> >> On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky < >> dbelenky@redhat.com> wrote: >> >>> Hi all, >>> The following tests are failing both of the upgrade suites in OST >>> (upgrade_from_release and upgrade_from_prevrelease). >>> >>> *Link to console:* ovirt-master_change-queue-tester/3146/console >>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> >>> *Link to test logs:* >>> - upgrade-from-release-suit-master-el7 >>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> >>> - upgrade-from-prevrelease-suit-master-el7 >>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> >>> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 >>> *Please note that every patch that is based on top of the patch >>> above was not deployed to the tested repo.* >>> >>> *Error snippet from engine setup log:* >>> >> > Please add a direct link next time, if possible. This is it: > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste > r/3146/artifact/exported-artifacts/upgrade-from-release-suit > -master-el7/test_logs/upgrade-from-release-suite-master/post > -001_upgrade_engine.py/lago-upgrade-from-release-suite-maste > r-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-2017 > 1013222617-73f0df.log > > And a bit above the snippet below, there is: > > 2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set > up.ovirt_engine.upgrade.asynctasks plugin.execute:926 > execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', > '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', > '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', > '-r', '-Z') stderr: > > /usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory > > > 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception >>> Traceback (most recent call last): >>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >>> method['method']() >>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies >>> self._clearZombies() >>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies >>> 'Failed to clear zombie commands. ' >>> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem >>> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem >>> >>> > With [1], taskcleaner.sh sources generate-pgpass.sh . > > generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, > is not > yet upgraded (at the point of above failure). > > generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a > path > relative to "$0". In master it does not, but we now upgrade and it > does. > > This, in principle, is the core of the bug: > > A file, such as generate-pgpass.sh, that's supposed to be sourced > from some other files, should not by itself source other files > that are relative to "$0", because it can't know what "$0" is - it's > the path of the script sourcing it, not of itself. > > It seems like luckily we were not affected by this in 4.1, because > all of the files that sourced generate-pgpass.sh were together with > it in the same directory. But with [1], taskcleaner does too now, > and is in a different directory. > > Not sure what's the best solution: > > - revert [1] (and introduce it later on, in 4.3) > > - patch 4.1's generate-pgpass.sh and require the fixed 4.1 version > in 4.2 setup >
E.g.:
https://gerrit.ovirt.org/82800
> - Somehow trick everything to work together? Not sure. Seems like > you can't set $0. > > [1] https://gerrit.ovirt.org/82511 >
The same bug exists with unlock_entity:
https://gerrit.ovirt.org/82615
So we should probably revert both.
I tend to agree. Doesn't mean we cannot fix this for 4.2, but let's revert for the meantime. Y.
> > > Regards, > > >> -- >>> >>> DANIEL BELENKY >>> >>> RHV DEVOPS >>> >>> EMEA VIRTUALIZATION R&D >>> <https://red.ht/sig> >>> >> >> >> >> -- >> >> DANIEL BELENKY >> >> RHV DEVOPS >> >> EMEA VIRTUALIZATION R&D >> <https://red.ht/sig> >> >> _______________________________________________ >> Devel mailing list >> Devel@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > > > > -- > Didi >
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
2017-10-16 10:30 GMT+02:00 Yedidyah Bar David <didi@redhat.com>:
On Mon, Oct 16, 2017 at 11:01 AM, Miroslava Voglova <mvoglova@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:44 AM, Martin Perina <mperina@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova < mvoglova@redhat.com> wrote:
Fix on review https://gerrit.ovirt.org/#/c/82799/
That's indeed a related patch, but not sure how it solves current failure.
Let's copy generate-pgpass.sh to packaging/setup/dbutils and source it from this location for taskcleaner.sh and unlock_entity.sh
After offline discussion merging https://gerrit.ovirt.org/82800 and then https://gerrit.ovirt.org/#/c/82799/ will fix the issue. Both patches are needed.
So this means:
Merge https://gerrit.ovirt.org/82800 . We might want to open a real 4.1 bug for this.
Agreed, let's open a real 4.1.7 bug to track this and get proper verification by QE.
Build 4.1.7 (or 4.1.8?) with it .
Let's point to 4.1.7
Patch 4.2 engine-setup to require tools-4.1.7.
well, tools >= 4.1.7 :-)
Adding Sandro.
On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi@redhat.com > wrote:
> On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David < > didi@redhat.com> wrote: > >> On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky < >> dbelenky@redhat.com> wrote: >> >>> can someone address this issue? every patch to *ovirt-engine* that >>> is based on top of this patch is failing OST and* won't deploy to >>> the tested repo*. >>> >>> On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky < >>> dbelenky@redhat.com> wrote: >>> >>>> Hi all, >>>> The following tests are failing both of the upgrade suites in OST >>>> (upgrade_from_release and upgrade_from_prevrelease). >>>> >>>> *Link to console:* ovirt-master_change-queue-tester/3146/console >>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> >>>> *Link to test logs:* >>>> - upgrade-from-release-suit-master-el7 >>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> >>>> - upgrade-from-prevrelease-suit-master-el7 >>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> >>>> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 >>>> *Please note that every patch that is based on top of the patch >>>> above was not deployed to the tested repo.* >>>> >>>> *Error snippet from engine setup log:* >>>> >>> >> Please add a direct link next time, if possible. This is it: >> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste >> r/3146/artifact/exported-artifacts/upgrade-from-release-suit >> -master-el7/test_logs/upgrade-from-release-suite-master/post >> -001_upgrade_engine.py/lago-upgrade-from-release-suite-maste >> r-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-2017 >> 1013222617-73f0df.log >> >> And a bit above the snippet below, there is: >> >> 2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set >> up.ovirt_engine.upgrade.asynctasks plugin.execute:926 >> execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', >> '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', >> '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', >> '-r', '-Z') stderr: >> >> /usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory >> >> >> 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception >>>> Traceback (most recent call last): >>>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >>>> method['method']() >>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies >>>> self._clearZombies() >>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies >>>> 'Failed to clear zombie commands. ' >>>> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>> >>>> >> With [1], taskcleaner.sh sources generate-pgpass.sh . >> >> generate-pgpass.sh is in ovirt-engine-tools, which in upgrade >> flows, is not >> yet upgraded (at the point of above failure). >> >> generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a >> path >> relative to "$0". In master it does not, but we now upgrade and it >> does. >> >> This, in principle, is the core of the bug: >> >> A file, such as generate-pgpass.sh, that's supposed to be sourced >> from some other files, should not by itself source other files >> that are relative to "$0", because it can't know what "$0" is - it's >> the path of the script sourcing it, not of itself. >> >> It seems like luckily we were not affected by this in 4.1, because >> all of the files that sourced generate-pgpass.sh were together with >> it in the same directory. But with [1], taskcleaner does too now, >> and is in a different directory. >> >> Not sure what's the best solution: >> >> - revert [1] (and introduce it later on, in 4.3) >> >> - patch 4.1's generate-pgpass.sh and require the fixed 4.1 version >> in 4.2 setup >> >
E.g.:
https://gerrit.ovirt.org/82800
>> - Somehow trick everything to work together? Not sure. Seems like >> you can't set $0. >> >> [1] https://gerrit.ovirt.org/82511 >> > > The same bug exists with unlock_entity: > > https://gerrit.ovirt.org/82615 > > So we should probably revert both. >
I tend to agree. Doesn't mean we cannot fix this for 4.2, but let's revert for the meantime. Y.
> > >> >> >> Regards, >> >> >>> -- >>>> >>>> DANIEL BELENKY >>>> >>>> RHV DEVOPS >>>> >>>> EMEA VIRTUALIZATION R&D >>>> <https://red.ht/sig> >>>> >>> >>> >>> >>> -- >>> >>> DANIEL BELENKY >>> >>> RHV DEVOPS >>> >>> EMEA VIRTUALIZATION R&D >>> <https://red.ht/sig> >>> >>> _______________________________________________ >>> Devel mailing list >>> Devel@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >> >> >> >> -- >> Didi >> > > > > -- > Didi > > _______________________________________________ > Devel mailing list > Devel@ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel >
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
-- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>
Hey, I see that both of the patches that were supposed to fix the issue ( https://gerrit.ovirt.org/82800 and https://gerrit.ovirt.org/#/c/82799/) were merged, but the issue remains. On Mon, Oct 16, 2017 at 11:45 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2017-10-16 10:30 GMT+02:00 Yedidyah Bar David <didi@redhat.com>:
On Mon, Oct 16, 2017 at 11:01 AM, Miroslava Voglova <mvoglova@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:44 AM, Martin Perina <mperina@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova < mvoglova@redhat.com> wrote:
Fix on review https://gerrit.ovirt.org/#/c/82799/
That's indeed a related patch, but not sure how it solves current failure.
Let's copy generate-pgpass.sh to packaging/setup/dbutils and source it from this location for taskcleaner.sh and unlock_entity.sh
After offline discussion merging https://gerrit.ovirt.org/82800 and then https://gerrit.ovirt.org/#/c/82799/ will fix the issue. Both patches are needed.
So this means:
Merge https://gerrit.ovirt.org/82800 . We might want to open a real 4.1 bug for this.
Agreed, let's open a real 4.1.7 bug to track this and get proper verification by QE.
Build 4.1.7 (or 4.1.8?) with it .
Let's point to 4.1.7
Patch 4.2 engine-setup to require tools-4.1.7.
well, tools >= 4.1.7 :-)
Adding Sandro.
On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
> > > On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David < > didi@redhat.com> wrote: > >> On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David < >> didi@redhat.com> wrote: >> >>> On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky < >>> dbelenky@redhat.com> wrote: >>> >>>> can someone address this issue? every patch to *ovirt-engine* that >>>> is based on top of this patch is failing OST and* won't deploy >>>> to the tested repo*. >>>> >>>> On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky < >>>> dbelenky@redhat.com> wrote: >>>> >>>>> Hi all, >>>>> The following tests are failing both of the upgrade suites in >>>>> OST (upgrade_from_release and upgrade_from_prevrelease). >>>>> >>>>> *Link to console:* ovirt-master_change-queue-tester/3146/console >>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> >>>>> *Link to test logs:* >>>>> - upgrade-from-release-suit-master-el7 >>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> >>>>> - upgrade-from-prevrelease-suit-master-el7 >>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> >>>>> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 >>>>> *Please note that every patch that is based on top of the patch >>>>> above was not deployed to the tested repo.* >>>>> >>>>> *Error snippet from engine setup log:* >>>>> >>>> >>> Please add a direct link next time, if possible. This is it: >>> >>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste >>> r/3146/artifact/exported-artifacts/upgrade-from-release-suit >>> -master-el7/test_logs/upgrade-from-release-suite-master/post >>> -001_upgrade_engine.py/lago-upgrade-from-release-suite-maste >>> r-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-2017 >>> 1013222617-73f0df.log >>> >>> And a bit above the snippet below, there is: >>> >>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set >>> up.ovirt_engine.upgrade.asynctasks plugin.execute:926 >>> execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', >>> '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', >>> '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', >>> '-r', '-Z') stderr: >>> >>> /usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory >>> >>> >>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception >>>>> Traceback (most recent call last): >>>>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >>>>> method['method']() >>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies >>>>> self._clearZombies() >>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies >>>>> 'Failed to clear zombie commands. ' >>>>> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>>> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>>> >>>>> >>> With [1], taskcleaner.sh sources generate-pgpass.sh . >>> >>> generate-pgpass.sh is in ovirt-engine-tools, which in upgrade >>> flows, is not >>> yet upgraded (at the point of above failure). >>> >>> generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using >>> a path >>> relative to "$0". In master it does not, but we now upgrade and it >>> does. >>> >>> This, in principle, is the core of the bug: >>> >>> A file, such as generate-pgpass.sh, that's supposed to be sourced >>> from some other files, should not by itself source other files >>> that are relative to "$0", because it can't know what "$0" is - >>> it's >>> the path of the script sourcing it, not of itself. >>> >>> It seems like luckily we were not affected by this in 4.1, because >>> all of the files that sourced generate-pgpass.sh were together with >>> it in the same directory. But with [1], taskcleaner does too now, >>> and is in a different directory. >>> >>> Not sure what's the best solution: >>> >>> - revert [1] (and introduce it later on, in 4.3) >>> >>> - patch 4.1's generate-pgpass.sh and require the fixed 4.1 version >>> in 4.2 setup >>> >>
E.g.:
https://gerrit.ovirt.org/82800
>>> - Somehow trick everything to work together? Not sure. Seems like >>> you can't set $0. >>> >>> [1] https://gerrit.ovirt.org/82511 >>> >> >> The same bug exists with unlock_entity: >> >> https://gerrit.ovirt.org/82615 >> >> So we should probably revert both. >> > > I tend to agree. > Doesn't mean we cannot fix this for 4.2, but let's revert for the > meantime. > Y. > > >> >> >>> >>> >>> Regards, >>> >>> >>>> -- >>>>> >>>>> DANIEL BELENKY >>>>> >>>>> RHV DEVOPS >>>>> >>>>> EMEA VIRTUALIZATION R&D >>>>> <https://red.ht/sig> >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> DANIEL BELENKY >>>> >>>> RHV DEVOPS >>>> >>>> EMEA VIRTUALIZATION R&D >>>> <https://red.ht/sig> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>> >>> >>> -- >>> Didi >>> >> >> >> >> -- >> Didi >> >> _______________________________________________ >> Devel mailing list >> Devel@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > > > _______________________________________________ > Devel mailing list > Devel@ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel >
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>
-- DANIEL BELENKY RHV DEVOPS EMEA VIRTUALIZATION R&D <https://red.ht/sig>
On Tue, Oct 17, 2017 at 1:17 PM, Daniel Belenky <dbelenky@redhat.com> wrote:
Hey, I see that both of the patches that were supposed to fix the issue ( https://gerrit.ovirt.org/82800 and https://gerrit.ovirt.org/#/c/82799/) were merged, but the issue remains.
Are we sure that when testing upgrade from 4.1 to master, we are using 4.1.7.XXX from latest tested as a base version? Because it we will try to upgrade from any previous version, the upgrade will always fail. I'm asking about it, because we wasn't able to find a way how to force manual OST to use latest tested as base version, it always use latest released 4.1.6. That's why Miroslava had to verify that manually
On Mon, Oct 16, 2017 at 11:45 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2017-10-16 10:30 GMT+02:00 Yedidyah Bar David <didi@redhat.com>:
On Mon, Oct 16, 2017 at 11:01 AM, Miroslava Voglova <mvoglova@redhat.com
wrote:
On Mon, Oct 16, 2017 at 9:44 AM, Martin Perina <mperina@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova < mvoglova@redhat.com> wrote:
> Fix on review https://gerrit.ovirt.org/#/c/82799/ >
That's indeed a related patch, but not sure how it solves current failure.
Let's copy generate-pgpass.sh to packaging/setup/dbutils and source it from this location for taskcleaner.sh and unlock_entity.sh
After offline discussion merging https://gerrit.ovirt.org/82800 and then https://gerrit.ovirt.org/#/c/82799/ will fix the issue. Both patches are needed.
So this means:
Merge https://gerrit.ovirt.org/82800 . We might want to open a real 4.1 bug for this.
Agreed, let's open a real 4.1.7 bug to track this and get proper verification by QE.
Build 4.1.7 (or 4.1.8?) with it .
Let's point to 4.1.7
Patch 4.2 engine-setup to require tools-4.1.7.
well, tools >= 4.1.7 :-)
Adding Sandro.
> > On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul@redhat.com> > wrote: > >> >> >> On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David < >> didi@redhat.com> wrote: >> >>> On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David < >>> didi@redhat.com> wrote: >>> >>>> On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky < >>>> dbelenky@redhat.com> wrote: >>>> >>>>> can someone address this issue? every patch to *ovirt-engine* that >>>>> is based on top of this patch is failing OST and* won't deploy >>>>> to the tested repo*. >>>>> >>>>> On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky < >>>>> dbelenky@redhat.com> wrote: >>>>> >>>>>> Hi all, >>>>>> The following tests are failing both of the upgrade suites in >>>>>> OST (upgrade_from_release and upgrade_from_prevrelease). >>>>>> >>>>>> *Link to console:* ovirt-master_change-q >>>>>> ueue-tester/3146/console >>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console> >>>>>> *Link to test logs:* >>>>>> - upgrade-from-release-suit-master-el7 >>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7> >>>>>> - upgrade-from-prevrelease-suit-master-el7 >>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7> >>>>>> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 >>>>>> *Please note that every patch that is based on top of the patch >>>>>> above was not deployed to the tested repo.* >>>>>> >>>>>> *Error snippet from engine setup log:* >>>>>> >>>>> >>>> Please add a direct link next time, if possible. This is it: >>>> >>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste >>>> r/3146/artifact/exported-artifacts/upgrade-from-release-suit >>>> -master-el7/test_logs/upgrade-from-release-suite-master/post >>>> -001_upgrade_engine.py/lago-upgrade-from-release-suite-maste >>>> r-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-2017 >>>> 1013222617-73f0df.log >>>> >>>> And a bit above the snippet below, there is: >>>> >>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set >>>> up.ovirt_engine.upgrade.asynctasks plugin.execute:926 >>>> execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', >>>> '-l', '/var/log/ovirt-engine/setup/o >>>> virt-engine-setup-20171013222617-73f0df.log', '-u', 'engine', >>>> '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr: >>>> >>>> /usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory >>>> >>>> >>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception >>>>>> Traceback (most recent call last): >>>>>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >>>>>> method['method']() >>>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies >>>>>> self._clearZombies() >>>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies >>>>>> 'Failed to clear zombie commands. ' >>>>>> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>>>> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>>>> >>>>>> >>>> With [1], taskcleaner.sh sources generate-pgpass.sh . >>>> >>>> generate-pgpass.sh is in ovirt-engine-tools, which in upgrade >>>> flows, is not >>>> yet upgraded (at the point of above failure). >>>> >>>> generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using >>>> a path >>>> relative to "$0". In master it does not, but we now upgrade and >>>> it does. >>>> >>>> This, in principle, is the core of the bug: >>>> >>>> A file, such as generate-pgpass.sh, that's supposed to be sourced >>>> from some other files, should not by itself source other files >>>> that are relative to "$0", because it can't know what "$0" is - >>>> it's >>>> the path of the script sourcing it, not of itself. >>>> >>>> It seems like luckily we were not affected by this in 4.1, because >>>> all of the files that sourced generate-pgpass.sh were together >>>> with >>>> it in the same directory. But with [1], taskcleaner does too now, >>>> and is in a different directory. >>>> >>>> Not sure what's the best solution: >>>> >>>> - revert [1] (and introduce it later on, in 4.3) >>>> >>>> - patch 4.1's generate-pgpass.sh and require the fixed 4.1 version >>>> in 4.2 setup >>>> >>> E.g.:
https://gerrit.ovirt.org/82800
> >>>> - Somehow trick everything to work together? Not sure. Seems like >>>> you can't set $0. >>>> >>>> [1] https://gerrit.ovirt.org/82511 >>>> >>> >>> The same bug exists with unlock_entity: >>> >>> https://gerrit.ovirt.org/82615 >>> >>> So we should probably revert both. >>> >> >> I tend to agree. >> Doesn't mean we cannot fix this for 4.2, but let's revert for the >> meantime. >> Y. >> >> >>> >>> >>>> >>>> >>>> Regards, >>>> >>>> >>>>> -- >>>>>> >>>>>> DANIEL BELENKY >>>>>> >>>>>> RHV DEVOPS >>>>>> >>>>>> EMEA VIRTUALIZATION R&D >>>>>> <https://red.ht/sig> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> DANIEL BELENKY >>>>> >>>>> RHV DEVOPS >>>>> >>>>> EMEA VIRTUALIZATION R&D >>>>> <https://red.ht/sig> >>>>> >>>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>> >>>> >>>> >>>> -- >>>> Didi >>>> >>> >>> >>> >>> -- >>> Didi >>> >>> _______________________________________________ >>> Devel mailing list >>> Devel@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >> >> >> _______________________________________________ >> Devel mailing list >> Devel@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > >
-- Didi
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D <https://red.ht/sig>
On Tue, Oct 17, 2017 at 3:23 PM, Martin Perina <mperina@redhat.com> wrote:
On Tue, Oct 17, 2017 at 1:17 PM, Daniel Belenky <dbelenky@redhat.com> wrote:
Hey, I see that both of the patches that were supposed to fix the issue (https://gerrit.ovirt.org/82800 and https://gerrit.ovirt.org/#/c/82799/) were merged, but the issue remains.
Are we sure that when testing upgrade from 4.1 to master, we are using 4.1.7.XXX from latest tested as a base version? Because it we will try to upgrade from any previous version, the upgrade
will always fail. I'm asking about it, because we wasn't able to find a way how to force manual OST to use latest tested as base version, it always use latest released 4.1.6. That's why Miroslava had to verify that manually
This build upgraded 4.0 -> 4.2. Daniel will disable it. I also pushed a patch to prevent it: https://gerrit.ovirt.org/82860 .
On Mon, Oct 16, 2017 at 11:45 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2017-10-16 10:30 GMT+02:00 Yedidyah Bar David <didi@redhat.com>:
On Mon, Oct 16, 2017 at 11:01 AM, Miroslava Voglova <mvoglova@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:44 AM, Martin Perina <mperina@redhat.com> wrote:
On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi@redhat.com> wrote: > > On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova <mvoglova@redhat.com> wrote: >> >> Fix on review https://gerrit.ovirt.org/#/c/82799/ > > > That's indeed a related patch, but not sure how it solves current failure.
Let's copy generate-pgpass.sh to packaging/setup/dbutils and source it from this location for taskcleaner.sh and unlock_entity.sh
After offline discussion merging https://gerrit.ovirt.org/82800 and then https://gerrit.ovirt.org/#/c/82799/ will fix the issue. Both patches are needed.
So this means:
Merge https://gerrit.ovirt.org/82800 . We might want to open a real 4.1 bug for this.
Agreed, let's open a real 4.1.7 bug to track this and get proper verification by QE.
Build 4.1.7 (or 4.1.8?) with it .
Let's point to 4.1.7
Patch 4.2 engine-setup to require tools-4.1.7.
well, tools >= 4.1.7 :-)
Adding Sandro.
> >> >> >> On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote: >>> >>> >>> >>> On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi@redhat.com> wrote: >>>> >>>> On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <didi@redhat.com> wrote: >>>>> >>>>> On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky@redhat.com> wrote: >>>>>> >>>>>> can someone address this issue? every patch to ovirt-engine that is based on top of this patch is failing OST and won't deploy to the tested repo. >>>>>> >>>>>> On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky@redhat.com> wrote: >>>>>>> >>>>>>> Hi all, >>>>>>> The following tests are failing both of the upgrade suites in OST (upgrade_from_release and upgrade_from_prevrelease). >>>>>>> >>>>>>> Link to console: ovirt-master_change-queue-tester/3146/console >>>>>>> Link to test logs: >>>>>>> - upgrade-from-release-suit-master-el7 >>>>>>> - upgrade-from-prevrelease-suit-master-el7 >>>>>>> Suspected patch: https://gerrit.ovirt.org/#/c/82615/5 >>>>>>> Please note that every patch that is based on top of the patch above was not deployed to the tested repo. >>>>>>> >>>>>>> Error snippet from engine setup log: >>>>> >>>>> >>>>> Please add a direct link next time, if possible. This is it: >>>>> >>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/... >>>>> >>>>> And a bit above the snippet below, there is: >>>>> >>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.upgrade.asynctasks plugin.execute:926 execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr: >>>>> >>>>> /usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory >>>>> >>>>>>> >>>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception >>>>>>> Traceback (most recent call last): >>>>>>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >>>>>>> method['method']() >>>>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies >>>>>>> self._clearZombies() >>>>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies >>>>>>> 'Failed to clear zombie commands. ' >>>>>>> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>>>>> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>>> >>>>> >>>>> With [1], taskcleaner.sh sources generate-pgpass.sh . >>>>> >>>>> generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, is not >>>>> yet upgraded (at the point of above failure). >>>>> >>>>> generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a path >>>>> relative to "$0". In master it does not, but we now upgrade and it does. >>>>> >>>>> This, in principle, is the core of the bug: >>>>> >>>>> A file, such as generate-pgpass.sh, that's supposed to be sourced >>>>> from some other files, should not by itself source other files >>>>> that are relative to "$0", because it can't know what "$0" is - it's >>>>> the path of the script sourcing it, not of itself. >>>>> >>>>> It seems like luckily we were not affected by this in 4.1, because >>>>> all of the files that sourced generate-pgpass.sh were together with >>>>> it in the same directory. But with [1], taskcleaner does too now, >>>>> and is in a different directory. >>>>> >>>>> Not sure what's the best solution: >>>>> >>>>> - revert [1] (and introduce it later on, in 4.3) >>>>> >>>>> - patch 4.1's generate-pgpass.sh and require the fixed 4.1 version >>>>> in 4.2 setup > > > E.g.: > > https://gerrit.ovirt.org/82800 > >>>>> >>>>> >>>>> - Somehow trick everything to work together? Not sure. Seems like >>>>> you can't set $0. >>>>> >>>>> [1] https://gerrit.ovirt.org/82511 >>>> >>>> >>>> The same bug exists with unlock_entity: >>>> >>>> https://gerrit.ovirt.org/82615 >>>> >>>> So we should probably revert both. >>> >>> >>> I tend to agree. >>> Doesn't mean we cannot fix this for 4.2, but let's revert for the meantime. >>> Y. >>> >>>> >>>> >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> DANIEL BELENKY >>>>>>> >>>>>>> RHV DEVOPS >>>>>>> >>>>>>> EMEA VIRTUALIZATION R&D >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> DANIEL BELENKY >>>>>> >>>>>> RHV DEVOPS >>>>>> >>>>>> EMEA VIRTUALIZATION R&D >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Devel mailing list >>>>>> Devel@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Didi >>>> >>>> >>>> >>>> >>>> -- >>>> Didi >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>> >>> >>> >>> _______________________________________________ >>> Devel mailing list >>> Devel@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >> >> > > > > -- > Didi > > _______________________________________________ > Devel mailing list > Devel@ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel
-- Didi
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA
TRIED. TESTED. TRUSTED.
--
DANIEL BELENKY
RHV DEVOPS
EMEA VIRTUALIZATION R&D
-- Didi
participants (6)
-
Daniel Belenky -
Martin Perina -
Miroslava Voglova -
Sandro Bonazzola -
Yaniv Kaul -
Yedidyah Bar David