[CQ]: 88046, 11 (ovirt-hosted-engine-setup) failed "ovirt-master" system tests
Change 88046,11 (ovirt-hosted-engine-setup) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/88046/11 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5866/
2018-02-26 13:39 GMT+01:00 oVirt Jenkins <jenkins@ovirt.org>:
Change 88046,11 (ovirt-hosted-engine-setup) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed.
I don't think so. Failure is: Error: Fault reason is "Operation Failed". Fault detail is "[Cannot deactivate Storage while there are running tasks on this Storage. -Please wait until tasks will finish and try again.]". HTTP response code is 409. in 007_sd_reattach.deactivate_storage_domain (from nosetests) and sounds like a bug in the test suite not waiting for running task completion before deactivating the storage.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/88046/11
For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5866/ _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
it is a bug in the test suite which is reported on OST bugzilla. On Tue, Feb 27, 2018 at 10:52 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2018-02-26 13:39 GMT+01:00 oVirt Jenkins <jenkins@ovirt.org>:
Change 88046,11 (ovirt-hosted-engine-setup) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed.
I don't think so. Failure is:
Error: Fault reason is "Operation Failed". Fault detail is "[Cannot deactivate Storage while there are running tasks on this Storage. -Please wait until tasks will finish and try again.]". HTTP response code is 409.
in 007_sd_reattach.deactivate_storage_domain (from nosetests)
and sounds like a bug in the test suite not waiting for running task completion before deactivating the storage.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/88046/11
For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5866/ _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
On Tue, Feb 27, 2018 at 1:32 PM, Dafna Ron <dron@redhat.com> wrote:
it is a bug in the test suite which is reported on OST bugzilla.
Is it possible to disable the test until we get a fix for it?
On Tue, Feb 27, 2018 at 10:52 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2018-02-26 13:39 GMT+01:00 oVirt Jenkins <jenkins@ovirt.org>:
Change 88046,11 (ovirt-hosted-engine-setup) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed.
I don't think so. Failure is:
Error: Fault reason is "Operation Failed". Fault detail is "[Cannot deactivate Storage while there are running tasks on this Storage. -Please wait until tasks will finish and try again.]". HTTP response code is 409.
in 007_sd_reattach.deactivate_storage_domain (from nosetests)
and sounds like a bug in the test suite not waiting for running task completion before deactivating the storage.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/88046/11
For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5866/ _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
this happens randomly when there is a delay of some sorts in the OST run. I did not skip since its not failing often. On Tue, Feb 27, 2018 at 11:40 AM, Eyal Edri <eedri@redhat.com> wrote:
On Tue, Feb 27, 2018 at 1:32 PM, Dafna Ron <dron@redhat.com> wrote:
it is a bug in the test suite which is reported on OST bugzilla.
Is it possible to disable the test until we get a fix for it?
On Tue, Feb 27, 2018 at 10:52 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2018-02-26 13:39 GMT+01:00 oVirt Jenkins <jenkins@ovirt.org>:
Change 88046,11 (ovirt-hosted-engine-setup) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed.
I don't think so. Failure is:
Error: Fault reason is "Operation Failed". Fault detail is "[Cannot deactivate Storage while there are running tasks on this Storage. -Please wait until tasks will finish and try again.]". HTTP response code is 409.
in 007_sd_reattach.deactivate_storage_domain (from nosetests)
and sounds like a bug in the test suite not waiting for running task completion before deactivating the storage.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/88046/11
For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5866/ _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Tue, Feb 27, 2018 at 2:08 PM, Dafna Ron <dron@redhat.com> wrote:
this happens randomly when there is a delay of some sorts in the OST run.
We have to understand why it happens and if we can't, the test should be disabled, we can't allow known issues to be failing every once in a while. If there is an effort to debug it and understand root cause then it might be worth to let it fail a few more times, otherwise I suggest to see how we can disable it until the bug will be fixed.
I did not skip since its not failing often.
On Tue, Feb 27, 2018 at 11:40 AM, Eyal Edri <eedri@redhat.com> wrote:
On Tue, Feb 27, 2018 at 1:32 PM, Dafna Ron <dron@redhat.com> wrote:
it is a bug in the test suite which is reported on OST bugzilla.
Is it possible to disable the test until we get a fix for it?
On Tue, Feb 27, 2018 at 10:52 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2018-02-26 13:39 GMT+01:00 oVirt Jenkins <jenkins@ovirt.org>:
Change 88046,11 (ovirt-hosted-engine-setup) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed.
I don't think so. Failure is:
Error: Fault reason is "Operation Failed". Fault detail is "[Cannot deactivate Storage while there are running tasks on this Storage. -Please wait until tasks will finish and try again.]". HTTP response code is 409.
in 007_sd_reattach.deactivate_storage_domain (from nosetests)
and sounds like a bug in the test suite not waiting for running task completion before deactivating the storage.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/88046/11
For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5866/ _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
I left the test so that I can understand why this happens :) I suspected it was latency but wanted to see if indeed it is. the failure is a storage behaviour and they do not have a way to query tasks statuses before re-trying the same action. I can disable the test until the bug is fixed, however, I am not sure if that would happen and the question is if the test provides more benefits running and failing in case there is latency (which can actually help debug other issues) or is the noise of this failing every month or so is worse? On Tue, Feb 27, 2018 at 12:13 PM, Eyal Edri <eedri@redhat.com> wrote:
On Tue, Feb 27, 2018 at 2:08 PM, Dafna Ron <dron@redhat.com> wrote:
this happens randomly when there is a delay of some sorts in the OST run.
We have to understand why it happens and if we can't, the test should be disabled, we can't allow known issues to be failing every once in a while. If there is an effort to debug it and understand root cause then it might be worth to let it fail a few more times, otherwise I suggest to see how we can disable it until the bug will be fixed.
I did not skip since its not failing often.
On Tue, Feb 27, 2018 at 11:40 AM, Eyal Edri <eedri@redhat.com> wrote:
On Tue, Feb 27, 2018 at 1:32 PM, Dafna Ron <dron@redhat.com> wrote:
it is a bug in the test suite which is reported on OST bugzilla.
Is it possible to disable the test until we get a fix for it?
On Tue, Feb 27, 2018 at 10:52 AM, Sandro Bonazzola <sbonazzo@redhat.com
wrote:
2018-02-26 13:39 GMT+01:00 oVirt Jenkins <jenkins@ovirt.org>:
Change 88046,11 (ovirt-hosted-engine-setup) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed.
I don't think so. Failure is:
Error: Fault reason is "Operation Failed". Fault detail is "[Cannot deactivate Storage while there are running tasks on this Storage. -Please wait until tasks will finish and try again.]". HTTP response code is 409.
in 007_sd_reattach.deactivate_storage_domain (from nosetests)
and sounds like a bug in the test suite not waiting for running task completion before deactivating the storage.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/88046/11
For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5866/ _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Tue, Feb 27, 2018 at 2:31 PM, Dafna Ron <dron@redhat.com> wrote:
I left the test so that I can understand why this happens :) I suspected it was latency but wanted to see if indeed it is. the failure is a storage behaviour and they do not have a way to query tasks statuses before re-trying the same action. I can disable the test until the bug is fixed, however, I am not sure if that would happen and the question is if the test provides more benefits running and failing in case there is latency (which can actually help debug other issues) or is the noise of this failing every month or so is worse?
Let's monitor it next week and if it continues to fail, we'll disable it. But in any case, we need to make sure that the bug is prioritized and targeted for a fix during the next oVirt milestone.
On Tue, Feb 27, 2018 at 12:13 PM, Eyal Edri <eedri@redhat.com> wrote:
On Tue, Feb 27, 2018 at 2:08 PM, Dafna Ron <dron@redhat.com> wrote:
this happens randomly when there is a delay of some sorts in the OST run.
We have to understand why it happens and if we can't, the test should be disabled, we can't allow known issues to be failing every once in a while. If there is an effort to debug it and understand root cause then it might be worth to let it fail a few more times, otherwise I suggest to see how we can disable it until the bug will be fixed.
I did not skip since its not failing often.
On Tue, Feb 27, 2018 at 11:40 AM, Eyal Edri <eedri@redhat.com> wrote:
On Tue, Feb 27, 2018 at 1:32 PM, Dafna Ron <dron@redhat.com> wrote:
it is a bug in the test suite which is reported on OST bugzilla.
Is it possible to disable the test until we get a fix for it?
On Tue, Feb 27, 2018 at 10:52 AM, Sandro Bonazzola < sbonazzo@redhat.com> wrote:
2018-02-26 13:39 GMT+01:00 oVirt Jenkins <jenkins@ovirt.org>:
> Change 88046,11 (ovirt-hosted-engine-setup) is probably the reason > behind > recent system test failures in the "ovirt-master" change queue and > needs to be > fixed. >
I don't think so. Failure is:
Error: Fault reason is "Operation Failed". Fault detail is "[Cannot deactivate Storage while there are running tasks on this Storage. -Please wait until tasks will finish and try again.]". HTTP response code is 409.
in 007_sd_reattach.deactivate_storage_domain (from nosetests)
and sounds like a bug in the test suite not waiting for running task completion before deactivating the storage.
> > This change had been removed from the testing queue. Artifacts build > from this > change will not be released until it is fixed. > > For further details about the change see: > https://gerrit.ovirt.org/#/c/88046/11 > > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5866/ > _______________________________________________ > Infra mailing list > Infra@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra >
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
its not failing any more. this was a one time failure that happens every once in a while when the network is loaded. On Thu, Mar 1, 2018 at 9:17 AM, Eyal Edri <eedri@redhat.com> wrote:
On Tue, Feb 27, 2018 at 2:31 PM, Dafna Ron <dron@redhat.com> wrote:
I left the test so that I can understand why this happens :) I suspected it was latency but wanted to see if indeed it is. the failure is a storage behaviour and they do not have a way to query tasks statuses before re-trying the same action. I can disable the test until the bug is fixed, however, I am not sure if that would happen and the question is if the test provides more benefits running and failing in case there is latency (which can actually help debug other issues) or is the noise of this failing every month or so is worse?
Let's monitor it next week and if it continues to fail, we'll disable it. But in any case, we need to make sure that the bug is prioritized and targeted for a fix during the next oVirt milestone.
On Tue, Feb 27, 2018 at 12:13 PM, Eyal Edri <eedri@redhat.com> wrote:
On Tue, Feb 27, 2018 at 2:08 PM, Dafna Ron <dron@redhat.com> wrote:
this happens randomly when there is a delay of some sorts in the OST run.
We have to understand why it happens and if we can't, the test should be disabled, we can't allow known issues to be failing every once in a while. If there is an effort to debug it and understand root cause then it might be worth to let it fail a few more times, otherwise I suggest to see how we can disable it until the bug will be fixed.
I did not skip since its not failing often.
On Tue, Feb 27, 2018 at 11:40 AM, Eyal Edri <eedri@redhat.com> wrote:
On Tue, Feb 27, 2018 at 1:32 PM, Dafna Ron <dron@redhat.com> wrote:
it is a bug in the test suite which is reported on OST bugzilla.
Is it possible to disable the test until we get a fix for it?
On Tue, Feb 27, 2018 at 10:52 AM, Sandro Bonazzola < sbonazzo@redhat.com> wrote:
> > > 2018-02-26 13:39 GMT+01:00 oVirt Jenkins <jenkins@ovirt.org>: > >> Change 88046,11 (ovirt-hosted-engine-setup) is probably the reason >> behind >> recent system test failures in the "ovirt-master" change queue and >> needs to be >> fixed. >> > > I don't think so. Failure is: > > Error: Fault reason is "Operation Failed". Fault detail is "[Cannot deactivate Storage while there are running tasks on this Storage. > -Please wait until tasks will finish and try again.]". HTTP response code is 409. > > > in 007_sd_reattach.deactivate_storage_domain (from nosetests) > > and sounds like a bug in the test suite not waiting for running task > completion before deactivating the storage. > > > >> >> This change had been removed from the testing queue. Artifacts >> build from this >> change will not be released until it is fixed. >> >> For further details about the change see: >> https://gerrit.ovirt.org/#/c/88046/11 >> >> For failed test results see: >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5866/ >> _______________________________________________ >> Infra mailing list >> Infra@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/infra >> > > > > -- > > SANDRO BONAZZOLA > > ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D > > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > > > _______________________________________________ > Infra mailing list > Infra@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > >
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
participants (4)
-
Dafna Ron -
Eyal Edri -
oVirt Jenkins -
Sandro Bonazzola