- Infra - oVirt List Archives

[JIRA] (OVIRT-3010) [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"
by Nir Soffer (oVirt JIRA) 10 Sep '20

10 Sep '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3010?page=com.atlassian.jira.… ] Nir Soffer commented on OVIRT-3010: ----------------------------------- On Tue, Sep 8, 2020 at 1:14 PM Ehud Yonasi <eyonasi(a)redhat.com> wrote: > I found out the root cause was because of space issues in the nightly > publisher run. > > The 3 ovirt publishers ran in together and the jenkins dir did not have > that capacity. > > Currently, I am restoring the ovirt-master snapshot and then 4.3 + 4.2, > Thanks! > plus sent a patch [1] to ensure they will not run in parallel. > > [1]: https://gerrit.ovirt.org/111160 > I probably don't understand the change, but it seems that all jobs will run now at 00:00? I think we need to work on a better mechanism for updating the repository. For example, we can have a symlink the to current repo: current -> 200908143754 200908143754/ repo files... 200908143830/ repo files... When creating a new repo, create a new directory. Update the "current" symlink only if creating the new repo succeeded. Then delete the previous respo. With this the cases when the entire repo or some files are missing are not possible. On Tue, Sep 8, 2020 at 12:13 PM Ehud Yonasi <eyonasi(a)redhat.com> wrote: > >> Hey, >> >> I am looking into this. >> Will update with the results I will find. >> >> On Tue, Sep 8, 2020 at 12:00 PM Lev Veyde <lveyde(a)redhat.com> wrote: >> >>> Hi Nir, >>> >>> It looks like the CI job broke it again, breaking 4.2, 4.3 and master >>> snapshot release RPM. >>> >>> I guess it's the same issue we had last time. >>> >>> Thanks in advance, >>> >>> On Tue, Sep 8, 2020 at 11:45 AM Nir Soffer <nsoffer(a)redhat.com> wrote: >>> >>>> On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote: >>>> > >>>> > This is the second time, last time there was an issue during a >>>> nightly job that >>>> > left the master snapshot repo broken. >>>> >>>> This is still broken today. >>>> >>>> In the past sending mail to infra-support created a bug, and someone >>>> was handling >>>> issue quickly. This seems to be broken now. >>>> >>>> We depend on >>>> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >>>> for development and CI. When the rpm is missing, CI jobs fail, and >>>> developers cannot >>>> update their setup. >>>> >>>> Please fix it as soon as possible. >>>> >>>> > I have 3 failure builds: >>>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ >>>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ >>>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ >>>> > >>>> > All seems to fail in: >>>> > >>>> > [2020-09-07T10:00:56.163Z] + dnf install -y >>>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >>>> > [2020-09-07T10:01:08.524Z] Status code: 403 for >>>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: >>>> > 66.187.230.40) >>>> > >>>> > Checking manually show: >>>> > >>>> > $ curl >>>> https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >>>> > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> >>>> > <html><head> >>>> > <title>403 Forbidden</title> >>>> > </head><body> >>>> > <h1>Forbidden</h1> >>>> > <p>You don't have permission to access >>>> /pub/yum-repo/ovirt-release-master.rpm >>>> > on this server.</p> >>>> > </body></html> >>>> > >>>> > Looking at: >>>> > https://resources.ovirt.org/pub/yum-repo/ >>>> > >>>> > ovirt-elease-master.rpm does not exist. >>>> > >>>> > So we have 2 issues: >>>> > - The file does not exist >>>> > - We return "403 Forbidden" instead of "404 Not Found" >>>> >>>> >>> >>> -- >>> >>> Lev Veyde >>> >>> Senior Software Engineer, RHCE | RHCVA | MCITP >>> >>> Red Hat Israel >>> >>> <https://www.redhat.com> >>> >>> lev(a)redhat.com | lveyde(a)redhat.com >>> <https://red.ht/sig> >>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>> _______________________________________________ >>> Devel mailing list -- devel(a)ovirt.org >>> To unsubscribe send an email to devel-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/B7VMLC7JKYSU2… >>> >> > [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm" > ------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-3010 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3010 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > This is the second time, last time there was an issue during a nightly job that > left the master snapshot repo broken. > I have 3 failure builds: > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ > All seems to fail in: > [2020-09-07T10:00:56.163Z] + dnf install -y > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > [2020-09-07T10:01:08.524Z] Status code: 403 for > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: > 66.187.230.40) > Checking manually show: > $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > <html><head> > <title>403 Forbidden</title> > </head><body> > <h1>Forbidden</h1> > <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm > on this server.</p> > </body></html> > Looking at: > https://resources.ovirt.org/pub/yum-repo/ > ovirt-elease-master.rpm does not exist. > So we have 2 issues: > - The file does not exist > - We return "403 Forbidden" instead of "404 Not Found" -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3010) [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"
by Ehud Yonasi (oVirt JIRA) 10 Sep '20

10 Sep '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3010?page=com.atlassian.jira.… ] Ehud Yonasi commented on OVIRT-3010: ------------------------------------ I found out the root cause was because of space issues in the nightly publisher run. The 3 ovirt publishers ran in together and the jenkins dir did not have that capacity. Currently, I am restoring the ovirt-master snapshot and then 4.3 + 4.2, plus sent a patch [1] to ensure they will not run in parallel. [1]: https://gerrit.ovirt.org/111160 On Tue, Sep 8, 2020 at 12:13 PM Ehud Yonasi <eyonasi(a)redhat.com> wrote: > Hey, > > I am looking into this. > Will update with the results I will find. > > On Tue, Sep 8, 2020 at 12:00 PM Lev Veyde <lveyde(a)redhat.com> wrote: > >> Hi Nir, >> >> It looks like the CI job broke it again, breaking 4.2, 4.3 and master >> snapshot release RPM. >> >> I guess it's the same issue we had last time. >> >> Thanks in advance, >> >> On Tue, Sep 8, 2020 at 11:45 AM Nir Soffer <nsoffer(a)redhat.com> wrote: >> >>> On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote: >>> > >>> > This is the second time, last time there was an issue during a nightly >>> job that >>> > left the master snapshot repo broken. >>> >>> This is still broken today. >>> >>> In the past sending mail to infra-support created a bug, and someone >>> was handling >>> issue quickly. This seems to be broken now. >>> >>> We depend on >>> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >>> for development and CI. When the rpm is missing, CI jobs fail, and >>> developers cannot >>> update their setup. >>> >>> Please fix it as soon as possible. >>> >>> > I have 3 failure builds: >>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ >>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ >>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ >>> > >>> > All seems to fail in: >>> > >>> > [2020-09-07T10:00:56.163Z] + dnf install -y >>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >>> > [2020-09-07T10:01:08.524Z] Status code: 403 for >>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: >>> > 66.187.230.40) >>> > >>> > Checking manually show: >>> > >>> > $ curl >>> https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >>> > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> >>> > <html><head> >>> > <title>403 Forbidden</title> >>> > </head><body> >>> > <h1>Forbidden</h1> >>> > <p>You don't have permission to access >>> /pub/yum-repo/ovirt-release-master.rpm >>> > on this server.</p> >>> > </body></html> >>> > >>> > Looking at: >>> > https://resources.ovirt.org/pub/yum-repo/ >>> > >>> > ovirt-elease-master.rpm does not exist. >>> > >>> > So we have 2 issues: >>> > - The file does not exist >>> > - We return "403 Forbidden" instead of "404 Not Found" >>> >>> >> >> -- >> >> Lev Veyde >> >> Senior Software Engineer, RHCE | RHCVA | MCITP >> >> Red Hat Israel >> >> <https://www.redhat.com> >> >> lev(a)redhat.com | lveyde(a)redhat.com >> <https://red.ht/sig> >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/B7VMLC7JKYSU2… >> > > [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm" > ------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-3010 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3010 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > This is the second time, last time there was an issue during a nightly job that > left the master snapshot repo broken. > I have 3 failure builds: > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ > All seems to fail in: > [2020-09-07T10:00:56.163Z] + dnf install -y > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > [2020-09-07T10:01:08.524Z] Status code: 403 for > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: > 66.187.230.40) > Checking manually show: > $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > <html><head> > <title>403 Forbidden</title> > </head><body> > <h1>Forbidden</h1> > <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm > on this server.</p> > </body></html> > Looking at: > https://resources.ovirt.org/pub/yum-repo/ > ovirt-elease-master.rpm does not exist. > So we have 2 issues: > - The file does not exist > - We return "403 Forbidden" instead of "404 Not Found" -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3010) [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"
by Ehud Yonasi (oVirt JIRA) 10 Sep '20

10 Sep '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3010?page=com.atlassian.jira.… ] Ehud Yonasi commented on OVIRT-3010: ------------------------------------ Hey, I am looking into this. Will update with the results I will find. On Tue, Sep 8, 2020 at 12:00 PM Lev Veyde <lveyde(a)redhat.com> wrote: > Hi Nir, > > It looks like the CI job broke it again, breaking 4.2, 4.3 and master > snapshot release RPM. > > I guess it's the same issue we had last time. > > Thanks in advance, > > On Tue, Sep 8, 2020 at 11:45 AM Nir Soffer <nsoffer(a)redhat.com> wrote: > >> On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote: >> > >> > This is the second time, last time there was an issue during a nightly >> job that >> > left the master snapshot repo broken. >> >> This is still broken today. >> >> In the past sending mail to infra-support created a bug, and someone >> was handling >> issue quickly. This seems to be broken now. >> >> We depend on >> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >> for development and CI. When the rpm is missing, CI jobs fail, and >> developers cannot >> update their setup. >> >> Please fix it as soon as possible. >> >> > I have 3 failure builds: >> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ >> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ >> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ >> > >> > All seems to fail in: >> > >> > [2020-09-07T10:00:56.163Z] + dnf install -y >> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >> > [2020-09-07T10:01:08.524Z] Status code: 403 for >> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: >> > 66.187.230.40) >> > >> > Checking manually show: >> > >> > $ curl >> https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >> > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> >> > <html><head> >> > <title>403 Forbidden</title> >> > </head><body> >> > <h1>Forbidden</h1> >> > <p>You don't have permission to access >> /pub/yum-repo/ovirt-release-master.rpm >> > on this server.</p> >> > </body></html> >> > >> > Looking at: >> > https://resources.ovirt.org/pub/yum-repo/ >> > >> > ovirt-elease-master.rpm does not exist. >> > >> > So we have 2 issues: >> > - The file does not exist >> > - We return "403 Forbidden" instead of "404 Not Found" >> >> > > -- > > Lev Veyde > > Senior Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/B7VMLC7JKYSU2… > > [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm" > ------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-3010 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3010 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > This is the second time, last time there was an issue during a nightly job that > left the master snapshot repo broken. > I have 3 failure builds: > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ > All seems to fail in: > [2020-09-07T10:00:56.163Z] + dnf install -y > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > [2020-09-07T10:01:08.524Z] Status code: 403 for > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: > 66.187.230.40) > Checking manually show: > $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > <html><head> > <title>403 Forbidden</title> > </head><body> > <h1>Forbidden</h1> > <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm > on this server.</p> > </body></html> > Looking at: > https://resources.ovirt.org/pub/yum-repo/ > ovirt-elease-master.rpm does not exist. > So we have 2 issues: > - The file does not exist > - We return "403 Forbidden" instead of "404 Not Found" -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3010) [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"
by Lev Veyde (oVirt JIRA) 10 Sep '20

10 Sep '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3010?page=com.atlassian.jira.… ] Lev Veyde commented on OVIRT-3010: ---------------------------------- Hi Nir, It looks like the CI job broke it again, breaking 4.2, 4.3 and master snapshot release RPM. I guess it's the same issue we had last time. Thanks in advance, On Tue, Sep 8, 2020 at 11:45 AM Nir Soffer <nsoffer(a)redhat.com> wrote: > On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > > > > This is the second time, last time there was an issue during a nightly > job that > > left the master snapshot repo broken. > > This is still broken today. > > In the past sending mail to infra-support created a bug, and someone > was handling > issue quickly. This seems to be broken now. > > We depend on > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > for development and CI. When the rpm is missing, CI jobs fail, and > developers cannot > update their setup. > > Please fix it as soon as possible. > > > I have 3 failure builds: > > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ > > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ > > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ > > > > All seems to fail in: > > > > [2020-09-07T10:00:56.163Z] + dnf install -y > > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > > [2020-09-07T10:01:08.524Z] Status code: 403 for > > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: > > 66.187.230.40) > > > > Checking manually show: > > > > $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > > <html><head> > > <title>403 Forbidden</title> > > </head><body> > > <h1>Forbidden</h1> > > <p>You don't have permission to access > /pub/yum-repo/ovirt-release-master.rpm > > on this server.</p> > > </body></html> > > > > Looking at: > > https://resources.ovirt.org/pub/yum-repo/ > > > > ovirt-elease-master.rpm does not exist. > > > > So we have 2 issues: > > - The file does not exist > > - We return "403 Forbidden" instead of "404 Not Found" > > -- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev(a)redhat.com | lveyde(a)redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm" > ------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-3010 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3010 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > This is the second time, last time there was an issue during a nightly job that > left the master snapshot repo broken. > I have 3 failure builds: > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ > All seems to fail in: > [2020-09-07T10:00:56.163Z] + dnf install -y > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > [2020-09-07T10:01:08.524Z] Status code: 403 for > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: > 66.187.230.40) > Checking manually show: > $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > <html><head> > <title>403 Forbidden</title> > </head><body> > <h1>Forbidden</h1> > <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm > on this server.</p> > </body></html> > Looking at: > https://resources.ovirt.org/pub/yum-repo/ > ovirt-elease-master.rpm does not exist. > So we have 2 issues: > - The file does not exist > - We return "403 Forbidden" instead of "404 Not Found" -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3010) [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"
by Anton Marchukov (oVirt JIRA) 10 Sep '20

10 Sep '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3010?page=com.atlassian.jira.… ] Anton Marchukov commented on OVIRT-3010: ---------------------------------------- > Begin forwarded message: > > From: Nir Soffer <nsoffer(a)redhat.com> > Subject: Re: [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm" > Date: 8 September 2020 at 10:44:44 CEST > To: infra-support(a)ovirt.org > Cc: devel <devel(a)ovirt.org>, Lev Veyde <lveyde(a)redhat.com>, Anton Marchukov <amarchuk(a)redhat.com>, "Nisan, Tal" <tnisan(a)redhat.com>, Galit Rosenthal <grosenth(a)redhat.com>, Michal Skrivanek <mskrivan(a)redhat.com>, infra <infra(a)ovirt.org> > > On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote: >> >> This is the second time, last time there was an issue during a nightly job that >> left the master snapshot repo broken. > > This is still broken today. > > In the past sending mail to infra-support created a bug, and someone > was handling > issue quickly. This seems to be broken now. > > We depend on http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > for development and CI. When the rpm is missing, CI jobs fail, and > developers cannot > update their setup. > > Please fix it as soon as possible. > >> I have 3 failure builds: >> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ >> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ >> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ >> >> All seems to fail in: >> >> [2020-09-07T10:00:56.163Z] + dnf install -y >> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >> [2020-09-07T10:01:08.524Z] Status code: 403 for >> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: >> 66.187.230.40) >> >> Checking manually show: >> >> $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> >> <html><head> >> <title>403 Forbidden</title> >> </head><body> >> <h1>Forbidden</h1> >> <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm >> on this server.</p> >> </body></html> >> >> Looking at: >> https://resources.ovirt.org/pub/yum-repo/ >> >> ovirt-elease-master.rpm does not exist. >> >> So we have 2 issues: >> - The file does not exist >> - We return "403 Forbidden" instead of "404 Not Found" > -- Anton Marchukov Associate Manager - RHV DevOps - Red Hat > [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm" > ------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-3010 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3010 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > This is the second time, last time there was an issue during a nightly job that > left the master snapshot repo broken. > I have 3 failure builds: > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ > All seems to fail in: > [2020-09-07T10:00:56.163Z] + dnf install -y > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > [2020-09-07T10:01:08.524Z] Status code: 403 for > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: > 66.187.230.40) > Checking manually show: > $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > <html><head> > <title>403 Forbidden</title> > </head><body> > <h1>Forbidden</h1> > <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm > on this server.</p> > </body></html> > Looking at: > https://resources.ovirt.org/pub/yum-repo/ > ovirt-elease-master.rpm does not exist. > So we have 2 issues: > - The file does not exist > - We return "403 Forbidden" instead of "404 Not Found" -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3010) [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"
by Nir Soffer (oVirt JIRA) 10 Sep '20

10 Sep '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3010?page=com.atlassian.jira.… ] Nir Soffer commented on OVIRT-3010: ----------------------------------- On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > > This is the second time, last time there was an issue during a nightly job that > left the master snapshot repo broken. This is still broken today. In the past sending mail to infra-support created a bug, and someone was handling issue quickly. This seems to be broken now. We depend on http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm for development and CI. When the rpm is missing, CI jobs fail, and developers cannot update their setup. Please fix it as soon as possible. > I have 3 failure builds: > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ > > All seems to fail in: > > [2020-09-07T10:00:56.163Z] + dnf install -y > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > [2020-09-07T10:01:08.524Z] Status code: 403 for > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: > 66.187.230.40) > > Checking manually show: > > $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > <html><head> > <title>403 Forbidden</title> > </head><body> > <h1>Forbidden</h1> > <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm > on this server.</p> > </body></html> > > Looking at: > https://resources.ovirt.org/pub/yum-repo/ > > ovirt-elease-master.rpm does not exist. > > So we have 2 issues: > - The file does not exist > - We return "403 Forbidden" instead of "404 Not Found" > [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm" > ------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-3010 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3010 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > This is the second time, last time there was an issue during a nightly job that > left the master snapshot repo broken. > I have 3 failure builds: > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ > All seems to fail in: > [2020-09-07T10:00:56.163Z] + dnf install -y > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > [2020-09-07T10:01:08.524Z] Status code: 403 for > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: > 66.187.230.40) > Checking manually show: > $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > <html><head> > <title>403 Forbidden</title> > </head><body> > <h1>Forbidden</h1> > <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm > on this server.</p> > </body></html> > Looking at: > https://resources.ovirt.org/pub/yum-repo/ > ovirt-elease-master.rpm does not exist. > So we have 2 issues: > - The file does not exist > - We return "403 Forbidden" instead of "404 Not Found" -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3010) [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"
by Nir Soffer (oVirt JIRA) 10 Sep '20

10 Sep '20

Nir Soffer created OVIRT-3010: --------------------------------- Summary: [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm" Key: OVIRT-3010 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3010 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra This is the second time, last time there was an issue during a nightly job that left the master snapshot repo broken. I have 3 failure builds: https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ All seems to fail in: [2020-09-07T10:00:56.163Z] + dnf install -y http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm [2020-09-07T10:01:08.524Z] Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: 66.187.230.40) Checking manually show: $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm on this server.</p> </body></html> Looking at: https://resources.ovirt.org/pub/yum-repo/ ovirt-elease-master.rpm does not exist. So we have 2 issues: - The file does not exist - We return "403 Forbidden" instead of "404 Not Found" -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3009) errors on infra
by Yedidyah Bar David (oVirt JIRA) 10 Sep '20

10 Sep '20

Yedidyah Bar David created OVIRT-3009: ----------------------------------------- Summary: errors on infra Key: OVIRT-3009 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3009 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Yedidyah Bar David Assignee: infra Hi, 1. Please see thread "[ovirt-devel] gerrit pull/clone timeout". 2. Please see [1] - it failed check-merged, I tried again, and it failed again. In both cases, e.g. [2]: Error: Error downloading packages: Cannot download x86_64/ovirt-engine-wildfly-19.1.0-2.el8.x86_64.rpm: All mirrors were tried Of course, I can't be sure these are due to a single root cause. Please fix :-) Thanks, [1] https://gerrit.ovirt.org/111098 [2] https://jenkins.ovirt.org/job/ovirt-dwh_standard-on-merge/124/artifact/buil… -- Didi -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3008) IMAPS certificate expired on lists.ovirt.org
by Evgheni Dereveanchin (oVirt JIRA) 10 Sep '20

10 Sep '20

Evgheni Dereveanchin created OVIRT-3008: ------------------------------------------- Summary: IMAPS certificate expired on lists.ovirt.org Key: OVIRT-3008 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3008 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Evgheni Dereveanchin Assignee: infra We've got reports about ticket creation issues from email. Looking at dovecot logs the email handler tries to connect but disconnects immediately. Connecting to port 993 shows an expired cert: openssl s_client lists.ovirt.org:993 CONNECTED(00000003) depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 verify return:1 depth=0 CN = lists.ovirt.org verify error:num=10:certificate has expired notAfter=Jul 1 09:47:58 2020 GMT verify return:1 depth=0 CN = lists.ovirt.org notAfter=Jul 1 09:47:58 2020 GMT verify return:1 --- Certificate chain 0 s:CN = lists.ovirt.org i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 i:O = Digital Signature Trust Co., CN = DST Root CA X3 Looks like dovecot is not getting restarted after a new cert is issued. Leaving as-is for now for further review. A service restart should theoretically be enough to make use of the current cert. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 363 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Sep '20

10 Sep '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 363 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #363 [Martin Necas] Ansible: remove nic_on_boot from cloud-init ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.100_basic_ui_sanity_pytest.test_login[firefox] Error Message: test setup failure Stack Trace: engine_fqdn = 'engine', engine_ip = '192.168.201.4' @pytest.fixture(scope="session") def hub_url(engine_fqdn, engine_ip): env_url = _env_hub_url() if env_url is not None: yield env_url else: backend = _grid_backend() if backend == "podman": if os_utils.on_centos(8) and os_utils.inside_mock(): pytest.skip("podman doesn't work in mock") hub_port = network_utils.find_free_port(4444, 4544) with podman.grid(engine_fqdn, engine_ip, hub_port=hub_port) as hub_url: yield hub_url elif backend == "docker": > with docker.grid(engine_fqdn, engine_ip) as hub_url: ../ost_utils/ost_utils/pytest/fixtures/selenium/__init__.py:76: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python2.7/contextlib.py:17: in __enter__ return self.gen.next() ../ost_utils/ost_utils/selenium/docker.py:118: in grid with _network(network_name): /usr/lib64/python2.7/contextlib.py:17: in __enter__ return self.gen.next() ../ost_utils/ost_utils/selenium/docker.py:61: in _network shell(["docker", "network", "create", network_name]) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'network', 'create', 'grid'], bytes_output = False kwargs = {}, process = <subprocess.Popen object at 0x7f48c8c633d0>, out = '' err = 'Error response from daemon: network with name grid already exists ' def shell(args, bytes_output=False, **kwargs): process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = process.communicate() if not bytes_output: out = out.decode("utf-8") err = err.decode("utf-8") if process.returncode: > raise ShellError(process.returncode, out, err) E ShellError: Command failed with rc=1. Stdout: E E Stderr: E Error response from daemon: network with name grid already exists ../ost_utils/ost_utils/shell.py:51: ShellError

1 1