March 2019 - Infra - oVirt List Archives

[JIRA] (OVIRT-2695) [VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?)
by Eyal Edri (oVirt JIRA) 10 Mar '19

10 Mar '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2695?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2695: ---------------------------------- Barak, can you please add info how and where you cleared the cache for future reference so we can look up this ticket if needed in the future? thanks On Sun, Mar 10, 2019, 13:12 Barak Korren (oVirt JIRA) < > [VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?) > ------------------------------------------------------------------ > > Key: OVIRT-2695 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2695 > Project: oVirt - virtualization made easy > Issue Type: Outage > Components: CI client projects, Jenkins Slaves > Reporter: Nir Soffer > Assignee: Barak Korren > > build-artifacts on s390x fail now with: > [Errno 2] No such file or directory: > '/var/cache/dnf/updates-07b7057ee4fded96/packages/systemd-238-12.git07f8cd5.fc28.s390x.rpm' > Here are few example failures: > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… > This is going to block the change queue when we merge these patches, > and we have 4.3 build soon. > We had this issue few week ago, and Barak fixed this by deleting caches on > the slaves. I hope this can be fixed quickly > this time. > Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100099)

1 0

[JIRA] (OVIRT-2695) [VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?)
by Barak Korren (oVirt JIRA) 10 Mar '19

10 Mar '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2695?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-2695: -------------------------------- Blocked By: user response Status: Blocked (was: To Do) blocking ticket on user response > [VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?) > ------------------------------------------------------------------ > > Key: OVIRT-2695 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2695 > Project: oVirt - virtualization made easy > Issue Type: Outage > Components: CI client projects, Jenkins Slaves > Reporter: Nir Soffer > Assignee: Barak Korren > > build-artifacts on s390x fail now with: > [Errno 2] No such file or directory: > '/var/cache/dnf/updates-07b7057ee4fded96/packages/systemd-238-12.git07f8cd5.fc28.s390x.rpm' > Here are few example failures: > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… > This is going to block the change queue when we merge these patches, > and we have 4.3 build soon. > We had this issue few week ago, and Barak fixed this by deleting caches on > the slaves. I hope this can be fixed quickly > this time. > Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100099)

2 1

[JIRA] (OVIRT-2695) [VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?)
by Barak Korren (oVirt JIRA) 10 Mar '19

10 Mar '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2695?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2695: ------------------------------------- [~nsoffer(a)redhat.com] I cleared the caches, please check that your jobs pass now > [VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?) > ------------------------------------------------------------------ > > Key: OVIRT-2695 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2695 > Project: oVirt - virtualization made easy > Issue Type: Outage > Components: CI client projects, Jenkins Slaves > Reporter: Nir Soffer > Assignee: Barak Korren > > build-artifacts on s390x fail now with: > [Errno 2] No such file or directory: > '/var/cache/dnf/updates-07b7057ee4fded96/packages/systemd-238-12.git07f8cd5.fc28.s390x.rpm' > Here are few example failures: > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… > This is going to block the change queue when we merge these patches, > and we have 4.3 build soon. > We had this issue few week ago, and Barak fixed this by deleting caches on > the slaves. I hope this can be fixed quickly > this time. > Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100099)

1 0

[JIRA] (OVIRT-2695) [VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?)
by Barak Korren (oVirt JIRA) 10 Mar '19

10 Mar '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2695?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-2695: -------------------------------- Issue Type: Outage (was: By-EMAIL) > [VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?) > ------------------------------------------------------------------ > > Key: OVIRT-2695 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2695 > Project: oVirt - virtualization made easy > Issue Type: Outage > Components: CI client projects, Jenkins Slaves > Reporter: Nir Soffer > Assignee: Barak Korren > > build-artifacts on s390x fail now with: > [Errno 2] No such file or directory: > '/var/cache/dnf/updates-07b7057ee4fded96/packages/systemd-238-12.git07f8cd5.fc28.s390x.rpm' > Here are few example failures: > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… > This is going to block the change queue when we merge these patches, > and we have 4.3 build soon. > We had this issue few week ago, and Barak fixed this by deleting caches on > the slaves. I hope this can be fixed quickly > this time. > Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100099)

1 0

[JIRA] (OVIRT-2695) [VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?)
by Barak Korren (oVirt JIRA) 10 Mar '19

10 Mar '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2695?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-2695: -------------------------------- Component/s: CI client projects Jenkins Slaves > [VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?) > ------------------------------------------------------------------ > > Key: OVIRT-2695 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2695 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Components: CI client projects, Jenkins Slaves > Reporter: Nir Soffer > Assignee: Barak Korren > > build-artifacts on s390x fail now with: > [Errno 2] No such file or directory: > '/var/cache/dnf/updates-07b7057ee4fded96/packages/systemd-238-12.git07f8cd5.fc28.s390x.rpm' > Here are few example failures: > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… > This is going to block the change queue when we merge these patches, > and we have 4.3 build soon. > We had this issue few week ago, and Barak fixed this by deleting caches on > the slaves. I hope this can be fixed quickly > this time. > Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100099)

1 0

[CQ]: 97030, 14 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 10 Mar '19

10 Mar '19

A system test invoked by the "ovirt-master" change queue including change 97030,14 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 97381,22 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 97381,22 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/97030/14 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/97381/22 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13338/

1 0

[JIRA] (OVIRT-2695) [VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?)
by Barak Korren (oVirt JIRA) 10 Mar '19

10 Mar '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2695?page=com.atlassian.jira.… ] Barak Korren reassigned OVIRT-2695: ----------------------------------- Assignee: Barak Korren (was: infra) > [VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?) > ------------------------------------------------------------------ > > Key: OVIRT-2695 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2695 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: Barak Korren > > build-artifacts on s390x fail now with: > [Errno 2] No such file or directory: > '/var/cache/dnf/updates-07b7057ee4fded96/packages/systemd-238-12.git07f8cd5.fc28.s390x.rpm' > Here are few example failures: > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… > This is going to block the change queue when we merge these patches, > and we have 4.3 build soon. > We had this issue few week ago, and Barak fixed this by deleting caches on > the slaves. I hope this can be fixed quickly > this time. > Nir -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100099)

1 0

[JIRA] (OVIRT-2252) The s390x slave is used in parallel by both the staging and the production CI systems
by Barak Korren (oVirt JIRA) 10 Mar '19

10 Mar '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2252?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2252: ------------------------------------- we cannot, because of the way stdciv2 works, if we drop the slave, the jobs for the 'jenkins' repo will never finish. I wouldn't want to just drop the 'jenkins' repo jobs, because that is the only way we have ATM to pre-test some things. I think at this point the solution we should aim for is to containerize the s390x host, that would resolve this issue and would enable us to drop support for not-containerized hosts in the future. I actually had a chat with the host's maintainer a while ago, and he was willing to install docker there. The intiative died out because I did not have time to push it further. > The s390x slave is used in parallel by both the staging and the production CI systems > ------------------------------------------------------------------------------------- > > Key: OVIRT-2252 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2252 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: Jenkins Slaves > Reporter: Barak Korren > Assignee: infra > > Since we only have one s390x slave, it is currently attached to both the staging and the production CI systems, and while they use separate user accounts, it turns out this is not enough to isolate them from one another. > The are several issues that are caused by this configuration: > # Tests that allocate a fixed network port can fail if they are run by both systems at the same time - this happens in practice when sending Python patchs to the '{{jenkins}}' repo because the {{mirror_client.py}} tests start a web server on port 8675. > # The {{mock_cleanup.sh}} script that is being run by one system can time out trying to umount things from a mock environment that was created and is being used by the other system. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100099)

1 0

[CQ]: 98364, 5 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 10 Mar '19

10 Mar '19

A system test invoked by the "ovirt-master" change queue including change 98364,5 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 97381,22 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 97381,22 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/98364/5 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/97381/22 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13336/

1 0

oVirt infra daily report - unstable production jobs - 759
by jenkins＠jenkins.phx.ovirt.org 09 Mar '19

09 Mar '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/759//artifact/exported-a… Cheers, Jenkins

1 0