September 2018 - Infra - oVirt List Archives

[JIRA] (OVIRT-2496) master ovirt-engine is failing on 002_bootstrap.add_hosts in upgrade suite
by Dafna Ron (oVirt JIRA) 17 Sep '18

17 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2496?page=com.atlassian.jira.… ] Dafna Ron updated OVIRT-2496: ----------------------------- Resolution: Fixed Status: Done (was: To Do) > master ovirt-engine is failing on 002_bootstrap.add_hosts in upgrade suite > --------------------------------------------------------------------------- > > Key: OVIRT-2496 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2496 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_code_regression, ost_failures > > we have a new regression after the previous one was fixed. > the issue is in the upgrade suite with test 002_bootstrap.add_hosts. > full logs: > https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chang… > error: https://pastebin.com/B5wxKTJy -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2496) master ovirt-engine is failing on 002_bootstrap.add_hosts in upgrade suite
by Dafna Ron (oVirt JIRA) 17 Sep '18

17 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2496?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2496: ---------------------------------- fix: https://gerrit.ovirt.org/94357 we have a passing CQ and a new package in tested. > master ovirt-engine is failing on 002_bootstrap.add_hosts in upgrade suite > --------------------------------------------------------------------------- > > Key: OVIRT-2496 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2496 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_code_regression, ost_failures > > we have a new regression after the previous one was fixed. > the issue is in the upgrade suite with test 002_bootstrap.add_hosts. > full logs: > https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chang… > error: https://pastebin.com/B5wxKTJy -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

PLEASE READ: Stop merging to ovirt-engine master until further notice
by Dafna Ron 17 Sep '18

17 Sep '18

Hello to all, I asked to stop merging last week but it seemed to have been ignored so writing again. *please stop merging to ovirt-engine master until further notice.* the project has been failing for the last two weeks on various different regressions. You continue to merge changes which means that you are risking further regressions and making it hard to stabilize the master branch for ovirt-engine. to make it clear: 1. all changes that were merged since sept. 4th to the master branch exist only in the ovirt-engine master branch (i.e they are not included in your project package). 2. any code merged to master in the past two weeks was most likely not tested by CQ since we exit on first test failure (if your code effects anything after the reported issues ran, they would not be tested). 3. you are putting other projects at risk for regressions since they are tested with a 2 weeks old ovirt-engine packages. Thanks you. Dafna

1 1

[JIRA] (OVIRT-2498) Failing KubeVirt CI
by Petr Kotas (oVirt JIRA) 17 Sep '18

17 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2498?page=com.atlassian.jira.… ] Petr Kotas commented on OVIRT-2498: ----------------------------------- I have created the issue. The failures in the stdci are only partially causing our failures. Most of the failures are due to unknown timeouts. For this I would like to see live load on the test machine. If this can be done. Thanks in advance. Best, Petr On Mon, Sep 17, 2018 at 4:18 PM Barak Korren (oVirt JIRA) < > Failing KubeVirt CI > ------------------- > > Key: OVIRT-2498 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2498 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Petr Kotas > Assignee: infra > > Hi, > I am working on fixing the issues on the KubeVirt e2e test suites. This > task is directly related to unstable CI, due to unknown errors. > The progress is reported in the CNV trello: > https://trello.com/c/HNXcMEQu/161-epic-improve-ci > I am creating this issue since the KubeVirt experience random timeouts on > random tests most of the times when test suites run. > The issue from outside is showing as timeouts on difference part of tests. > Sometimes the tests fails in set up phase, again due to random timeout. > The example in the link bellow timed out for network connection on > localhost. > [check-patch.k8s-1.11.0-dev.el7.x86_64] > requests.exceptions.ReadTimeout: > UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. > (read timeout=60) > Example of failing test suites is here > https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/1916/cons… > The list of errors related to the failing CI can be found in my notes > https://docs.google.com/document/d/1_ll1DOMHgCRHn_Df9i4uvtRFyMK-bDCHEeGfJFT… > I am not sure whether KubeVirt already shared the resource requirements, so > I provide short summary: > *Resources for KubeVirt e2e tests:* > - at least 12GB of RAM - we start 3 nodes (3 docker images) each require > 4GB of RAM > - exposed /dev/kvm to enable native virtualization > - cached images, since these are used to build the test cluster: > - kubevirtci/os-3.10.0-crio:latest > - kubevirtci/os-3.10.0-multus:latest > - kubevirtci/os-3.10.0:latest > - kubevirtci/k8s-1.10.4:latest > - kubevirtci/k8s-multus-1.11.1:latest > - kubevirtci/k8s-1.11.0:latest > How can we overcome this? Can we work together to build a suitable > requirements for running the tests so it passes each time? > Kind regards, > Petr Kotas -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2504) standard CI fails in project_setup phase
by Petr Kotas (oVirt JIRA) 17 Sep '18

17 Sep '18

Petr Kotas created OVIRT-2504: --------------------------------- Summary: standard CI fails in project_setup phase Key: OVIRT-2504 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2504 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Petr Kotas Assignee: infra Hi, the standard CI is failing in project_setup phase. Can you please debug that? The issue is somehow related to docker clean up phase. Example of failing CI is : - https://jenkins.ovirt.org/blue/organizations/jenkins/kubevirt_kubevirt_stan… - https://jenkins.ovirt.org/blue/organizations/jenkins/kubevirt_kubevirt_stan… - https://jenkins.ovirt.org/blue/organizations/jenkins/kubevirt_kubevirt_stan… Unfortunately, the logs are the only thing I can see. From our side, there is nothing special. Just normal test run. We did not tune anything. Thank you, Best, Petr Kotas -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2503) Testing automatic Jira ticket logging for CQ monitoring
by Dafna Ron (oVirt JIRA) 17 Sep '18

17 Sep '18

Dafna Ron created OVIRT-2503: -------------------------------- Summary: Testing automatic Jira ticket logging for CQ monitoring Key: OVIRT-2503 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2503 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Dafna Ron Assignee: infra As part of monitoring improvement and shifting partial responsibility for CQ monitoring to the developers I would like to do some experiments on connecting CQ alerts to create Jira tickets automatically. As we have the ost monitoring project we start by doing some tests on that project so that we do not overflow our jira with alerts. For now I would like to start with Jira's opening with the following specifications: 1. Subject: [ CQ ] [$patch number] [ oVirt $VER ($project) ] [ TEST NAME ] 2. at the beginning, email sent to: dron(a)redhat.com 3. jira description [cq message] 4. all jira's has to have label=ost_failure and infra-owner as default. I would like to change the project type to allow: 1. easy closing of Jira's (one/two clicks if we can) 2. view of Jira's like service tickets (rather then bugs) There is a plugin called zapier that allows to easily connect a jira from an email and also allow to add some rules to the Jira which may make this easier for us. can you also install it and link it to the ost jira? I have an email account that we can use for that. cq.ovirt(a)gmail.com https://zapier.com/apps/jira/integrations [~bkorren(a)redhat.com] once I do some tests on my own on this, I wanted to try and collaborate with one of the projects (maybe networking or one of sandro's teams) where CQ failures would automatically open a ticket to their team and they can handle the monitoring and escalate issues to us if needed. Any advice on configurations we should be thinking of for that? -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2498) Failing KubeVirt CI
by Barak Korren (oVirt JIRA) 17 Sep '18

17 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2498?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2498: ------------------------------------- [~pkotas] ok, I see its failing in the docketr-cleaup script... hmm.. we'll need to debug that... Can you please open a specific ticket on that and include logs and any other specific information that can help us figure out why it may be failing there... (What containers might be on the machine that its failing to remove...) But that is not what is causing all the failures right? We already fixed a couple of issues with that script... > Failing KubeVirt CI > ------------------- > > Key: OVIRT-2498 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2498 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Petr Kotas > Assignee: infra > > Hi, > I am working on fixing the issues on the KubeVirt e2e test suites. This > task is directly related to unstable CI, due to unknown errors. > The progress is reported in the CNV trello: > https://trello.com/c/HNXcMEQu/161-epic-improve-ci > I am creating this issue since the KubeVirt experience random timeouts on > random tests most of the times when test suites run. > The issue from outside is showing as timeouts on difference part of tests. > Sometimes the tests fails in set up phase, again due to random timeout. > The example in the link bellow timed out for network connection on > localhost. > [check-patch.k8s-1.11.0-dev.el7.x86_64] > requests.exceptions.ReadTimeout: > UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. > (read timeout=60) > Example of failing test suites is here > https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/1916/cons… > The list of errors related to the failing CI can be found in my notes > https://docs.google.com/document/d/1_ll1DOMHgCRHn_Df9i4uvtRFyMK-bDCHEeGfJFT… > I am not sure whether KubeVirt already shared the resource requirements, so > I provide short summary: > *Resources for KubeVirt e2e tests:* > - at least 12GB of RAM - we start 3 nodes (3 docker images) each require > 4GB of RAM > - exposed /dev/kvm to enable native virtualization > - cached images, since these are used to build the test cluster: > - kubevirtci/os-3.10.0-crio:latest > - kubevirtci/os-3.10.0-multus:latest > - kubevirtci/os-3.10.0:latest > - kubevirtci/k8s-1.10.4:latest > - kubevirtci/k8s-multus-1.11.1:latest > - kubevirtci/k8s-1.11.0:latest > How can we overcome this? Can we work together to build a suitable > requirements for running the tests so it passes each time? > Kind regards, > Petr Kotas -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[JIRA] (OVIRT-2498) Failing KubeVirt CI
by Petr Kotas (oVirt JIRA) 17 Sep '18

17 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2498?page=com.atlassian.jira.… ] Petr Kotas commented on OVIRT-2498: ----------------------------------- [~bkorren(a)redhat.com] I would like to see the machine resources. The use of CPU, the use of RAM to understand how the tests behave live. I am not sure whether this is working on Blue Ocean. WRT the docker, as I already pointer in the logs. The issue is way before our setup even kicks in. [Here|https://jenkins.ovirt.org/blue/organizations/jenkins/kubevirt_kubevirt…] is the direct link for that. It seems that the jenkins project_setup.sh fails somehow. Again this is not our code, it is part of standard ci located [here|https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=jobs/confs/shel…] It seems that the project setup, was doing its job and than randomly failed due to networking issue. I have no idea why. Also I do not thing the issue is due to proxy as the failures are totally random on random tests. So I am guessing something more hidden fails. > Failing KubeVirt CI > ------------------- > > Key: OVIRT-2498 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2498 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Petr Kotas > Assignee: infra > > Hi, > I am working on fixing the issues on the KubeVirt e2e test suites. This > task is directly related to unstable CI, due to unknown errors. > The progress is reported in the CNV trello: > https://trello.com/c/HNXcMEQu/161-epic-improve-ci > I am creating this issue since the KubeVirt experience random timeouts on > random tests most of the times when test suites run. > The issue from outside is showing as timeouts on difference part of tests. > Sometimes the tests fails in set up phase, again due to random timeout. > The example in the link bellow timed out for network connection on > localhost. > [check-patch.k8s-1.11.0-dev.el7.x86_64] > requests.exceptions.ReadTimeout: > UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. > (read timeout=60) > Example of failing test suites is here > https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/1916/cons… > The list of errors related to the failing CI can be found in my notes > https://docs.google.com/document/d/1_ll1DOMHgCRHn_Df9i4uvtRFyMK-bDCHEeGfJFT… > I am not sure whether KubeVirt already shared the resource requirements, so > I provide short summary: > *Resources for KubeVirt e2e tests:* > - at least 12GB of RAM - we start 3 nodes (3 docker images) each require > 4GB of RAM > - exposed /dev/kvm to enable native virtualization > - cached images, since these are used to build the test cluster: > - kubevirtci/os-3.10.0-crio:latest > - kubevirtci/os-3.10.0-multus:latest > - kubevirtci/os-3.10.0:latest > - kubevirtci/k8s-1.10.4:latest > - kubevirtci/k8s-multus-1.11.1:latest > - kubevirtci/k8s-1.11.0:latest > How can we overcome this? Can we work together to build a suitable > requirements for running the tests so it passes each time? > Kind regards, > Petr Kotas -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[CQ]: 94357, 1 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 17 Sep '18

17 Sep '18

A system test invoked by the "ovirt-master" change queue including change 94357,1 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94081,3 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94081,3 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94357/1 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94081/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10338/

1 0

[JIRA] (OVIRT-2502) official releases not being tested by change queue
by sbonazzo (oVirt JIRA) 17 Sep '18

17 Sep '18

sbonazzo created OVIRT-2502: ------------------------------- Summary: official releases not being tested by change queue Key: OVIRT-2502 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2502 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: sbonazzo Assignee: infra Hi, just noticed in https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-q… that both the patch including the official release patch and the post-release patch has been added to the same change queue job, causing the official release to be skipped in favor of the post release: - Started by upstream project ovirt-4.2_change-queue <https://jenkins.ovirt.org/job/ovirt-4.2_change-queue/> build number 8642 <https://jenkins.ovirt.org/job/ovirt-4.2_change-queue/8642/> originally caused by: - Started by upstream project standard-enqueue <https://jenkins.ovirt.org/job/standard-enqueue/> build number 16477 <https://jenkins.ovirt.org/job/standard-enqueue/16477/> originally caused by: - Triggered by Gerrit: https://gerrit.ovirt.org/94367 Other triggered builds for this event. ovirt-engine_4.2_build-artifacts-el7-x86_64 <https://jenkins.ovirt.org/job/ovirt-engine_4.2_build-artifacts-el7-x86_64/> #828 <https://jenkins.ovirt.org/job/ovirt-engine_4.2_build-artifacts-el7-x86_64/8…> ovirt-engine_4.2_check-merged-el7-x86_64 <https://jenkins.ovirt.org/job/ovirt-engine_4.2_check-merged-el7-x86_64/> #800 <https://jenkins.ovirt.org/job/ovirt-engine_4.2_check-merged-el7-x86_64/800/> - Started by upstream project ovirt-4.2_change-queue <https://jenkins.ovirt.org/job/ovirt-4.2_change-queue/> build number 8643 <https://jenkins.ovirt.org/job/ovirt-4.2_change-queue/8643/> originally caused by: - Started by upstream project standard-enqueue <https://jenkins.ovirt.org/job/standard-enqueue/> build number 16478 <https://jenkins.ovirt.org/job/standard-enqueue/16478/> originally caused by: - Triggered by Gerrit: https://gerrit.ovirt.org/94368 Other triggered builds for this event. ovirt-engine_4.2_build-artifacts-el7-x86_64 <https://jenkins.ovirt.org/job/ovirt-engine_4.2_build-artifacts-el7-x86_64/> #829 <https://jenkins.ovirt.org/job/ovirt-engine_4.2_build-artifacts-el7-x86_64/8…> ovirt-engine_4.2_check-merged-el7-x86_64 <https://jenkins.ovirt.org/job/ovirt-engine_4.2_check-merged-el7-x86_64/> #801 <https://jenkins.ovirt.org/job/ovirt-engine_4.2_check-merged-el7-x86_64/801/> This is really bad, meaning that a regression introduced in the release patch may be completely ignored provided that the post release patch fixes it. -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f…> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0