Re: Change in ovirt-system-tests[master]: Test hosted-engine cleanup
On Wed, Mar 21, 2018 at 11:48 AM, Code Review <gerrit@ovirt.org> wrote:
From Jenkins CI:
Jenkins CI has posted comments on this change.
Change subject: Test hosted-engine cleanup ......................................................................
Patch Set 19:
Build Failed
http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_6... : FAILURE
This is a retrigger of my previous mail to infra. Failed with: 09:48:27 CLEANUP: Stop all running containers and remove unwanted images ... 09:48:37 Error response from daemon: conflict: unable to delete 782fe38c8050 (cannot be forced) - image has dependent child images 09:48:37 Error response from daemon: conflict: unable to delete bcec0ae8107e (cannot be forced) - image has dependent child images 09:48:37 ERROR: Failed to clean docker images I have already seen this before. Perhaps we should automatically reprovision all slaves periodically? They should be cattle, not pets, right? :-) I already suggested this in the past and there was some objection...
-- To view, visit https://gerrit.ovirt.org/88331 To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-Project: ovirt-system-tests Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5876a419dc2437de51149b1fe02ffd900e6ab29e Gerrit-Change-Number: 88331 Gerrit-PatchSet: 19 Gerrit-Owner: Yedidyah Bar David <didi@redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Sandro Bonazzola <sbonazzo@redhat.com> Gerrit-Reviewer: Yedidyah Bar David <didi@redhat.com> Gerrit-Comment-Date: Wed, 21 Mar 2018 09:48:41 +0000 Gerrit-HasComments: No
-- Didi
On Wed, Mar 21, 2018 at 11:56 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Mar 21, 2018 at 11:48 AM, Code Review <gerrit@ovirt.org> wrote:
From Jenkins CI:
Jenkins CI has posted comments on this change.
Change subject: Test hosted-engine cleanup ......................................................................
Patch Set 19:
Build Failed
http://jenkins.ovirt.org/job/ovirt-system-tests_master_ check-patch-el7-x86_64/4616/ : FAILURE
This is a retrigger of my previous mail to infra. Failed with:
09:48:27 CLEANUP: Stop all running containers and remove unwanted images ...
09:48:37 Error response from daemon: conflict: unable to delete 782fe38c8050 (cannot be forced) - image has dependent child images 09:48:37 Error response from daemon: conflict: unable to delete bcec0ae8107e (cannot be forced) - image has dependent child images 09:48:37 ERROR: Failed to clean docker images
I have already seen this before.
Perhaps we should automatically reprovision all slaves periodically? They should be cattle, not pets, right? :-) I already suggested this in the past and there was some objection...
Unfourtunately oVIrt knows more how to handle pets than cattle :) This topic has indeed been discussed numerous times, since we use mock, we don't need to reprovision slaves, but there are some cleanups being done that might fail occasionally. Our plan is to move away from mock to containers, so hopefully jobs will run on clean containers running on OpenShift, so I don't think its worth investing too much in the current solution over other priorities, Especially since it doesn't fail often, and each time it fails we improve the cleanup to handle the issue.
-- To view, visit https://gerrit.ovirt.org/88331 To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-Project: ovirt-system-tests Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5876a419dc2437de51149b1fe02ffd900e6ab29e Gerrit-Change-Number: 88331 Gerrit-PatchSet: 19 Gerrit-Owner: Yedidyah Bar David <didi@redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Sandro Bonazzola <sbonazzo@redhat.com> Gerrit-Reviewer: Yedidyah Bar David <didi@redhat.com> Gerrit-Comment-Date: Wed, 21 Mar 2018 09:48:41 +0000 Gerrit-HasComments: No
-- Didi _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
2018-03-21 11:01 GMT+01:00 Eyal Edri <eedri@redhat.com>:
On Wed, Mar 21, 2018 at 11:56 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Mar 21, 2018 at 11:48 AM, Code Review <gerrit@ovirt.org> wrote:
From Jenkins CI:
Jenkins CI has posted comments on this change.
Change subject: Test hosted-engine cleanup ......................................................................
Patch Set 19:
Build Failed
http://jenkins.ovirt.org/job/ovirt-system-tests_master_check -patch-el7-x86_64/4616/ : FAILURE
This is a retrigger of my previous mail to infra. Failed with:
09:48:27 CLEANUP: Stop all running containers and remove unwanted images ...
09:48:37 Error response from daemon: conflict: unable to delete 782fe38c8050 (cannot be forced) - image has dependent child images 09:48:37 Error response from daemon: conflict: unable to delete bcec0ae8107e (cannot be forced) - image has dependent child images 09:48:37 ERROR: Failed to clean docker images
I have already seen this before.
Perhaps we should automatically reprovision all slaves periodically? They should be cattle, not pets, right? :-) I already suggested this in the past and there was some objection...
Unfourtunately oVIrt knows more how to handle pets than cattle :)
Can't we use a stateless pool for this scope?
This topic has indeed been discussed numerous times, since we use mock, we don't need to reprovision slaves, but there are some cleanups being done that might fail occasionally. Our plan is to move away from mock to containers, so hopefully jobs will run on clean containers running on OpenShift, so I don't think its worth investing too much in the current solution over other priorities,
I don't recall this plan has been discussed, can you link to the thread about this?
Especially since it doesn't fail often, and each time it fails we improve the cleanup to handle the issue.
-- To view, visit https://gerrit.ovirt.org/88331 To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-Project: ovirt-system-tests Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5876a419dc2437de51149b1fe02ffd900e6ab29e Gerrit-Change-Number: 88331 Gerrit-PatchSet: 19 Gerrit-Owner: Yedidyah Bar David <didi@redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Sandro Bonazzola <sbonazzo@redhat.com> Gerrit-Reviewer: Yedidyah Bar David <didi@redhat.com> Gerrit-Comment-Date: Wed, 21 Mar 2018 09:48:41 +0000 Gerrit-HasComments: No
-- Didi _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://red.ht/sig> <https://redhat.com/summit>
On Wed, Apr 4, 2018 at 12:10 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2018-03-21 11:01 GMT+01:00 Eyal Edri <eedri@redhat.com>:
On Wed, Mar 21, 2018 at 11:56 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Mar 21, 2018 at 11:48 AM, Code Review <gerrit@ovirt.org> wrote:
From Jenkins CI:
Jenkins CI has posted comments on this change.
Change subject: Test hosted-engine cleanup ......................................................................
Patch Set 19:
Build Failed
http://jenkins.ovirt.org/job/ovirt-system-tests_master_check -patch-el7-x86_64/4616/ : FAILURE
This is a retrigger of my previous mail to infra. Failed with:
09:48:27 CLEANUP: Stop all running containers and remove unwanted images ...
09:48:37 Error response from daemon: conflict: unable to delete 782fe38c8050 (cannot be forced) - image has dependent child images 09:48:37 Error response from daemon: conflict: unable to delete bcec0ae8107e (cannot be forced) - image has dependent child images 09:48:37 ERROR: Failed to clean docker images
I have already seen this before.
Perhaps we should automatically reprovision all slaves periodically? They should be cattle, not pets, right? :-) I already suggested this in the past and there was some objection...
Unfourtunately oVIrt knows more how to handle pets than cattle :)
Can't we use a stateless pool for this scope?
We tried it in downstream for some time, it didn't work well and we no longer use it. Barak wrote the flow, he can share more info if you want.
This topic has indeed been discussed numerous times, since we use mock, we don't need to reprovision slaves, but there are some cleanups being done that might fail occasionally. Our plan is to move away from mock to containers, so hopefully jobs will run on clean containers running on OpenShift, so I don't think its worth investing too much in the current solution over other priorities,
I don't recall this plan has been discussed, can you link to the thread about this?
I'm not sure if there was a thread on it, but there is a long-standing feature in JIRA [1], there are not immediate resources or plan for starting it though, we were hoping to get cooperation also from other CI teams on that front, but doesn't seem the case so far. [1] https://ovirt-jira.atlassian.net/browse/OVIRT-873
Especially since it doesn't fail often, and each time it fails we improve the cleanup to handle the issue.
-- To view, visit https://gerrit.ovirt.org/88331 To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-Project: ovirt-system-tests Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5876a419dc2437de51149b1fe02ffd900e6ab29e Gerrit-Change-Number: 88331 Gerrit-PatchSet: 19 Gerrit-Owner: Yedidyah Bar David <didi@redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Sandro Bonazzola <sbonazzo@redhat.com> Gerrit-Reviewer: Yedidyah Bar David <didi@redhat.com> Gerrit-Comment-Date: Wed, 21 Mar 2018 09:48:41 +0000 Gerrit-HasComments: No
-- Didi _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <https://red.ht/sig> <https://redhat.com/summit>
-- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
participants (3)
-
Eyal Edri -
Sandro Bonazzola -
Yedidyah Bar David