[CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"
6:30 a.m.
This is the second time, last time there was an issue during a nightly job that
left the master snapshot repo broken.
I have 3 failure builds:
https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
All seems to fail in:
[2020-09-07T10:00:56.163Z] + dnf install -y
http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
[2020-09-07T10:01:08.524Z] Status code: 403 for
http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
66.187.230.40)
Checking manually show:
$ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm
on this server.</p>
</body></html>
Looking at:
https://resources.ovirt.org/pub/yum-repo/
ovirt-elease-master.rpm does not exist.
So we have 2 issues:
- The file does not exist
- We return "403 Forbidden" instead of "404 Not Found"
3:44 a.m.
On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
This is the second time, last time there was an issue during a nightly job that
left the master snapshot repo broken.
This is still broken today.
In the past sending mail to infra-support created a bug, and someone
was handling
issue quickly. This seems to be broken now.
We depend on http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
for development and CI. When the rpm is missing, CI jobs fail, and
developers cannot
update their setup.
Please fix it as soon as possible.
I have 3 failure builds:
https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
All seems to fail in:
[2020-09-07T10:00:56.163Z] + dnf install -y
http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
[2020-09-07T10:01:08.524Z] Status code: 403 for
http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
66.187.230.40)
Checking manually show:
$ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm
on this server.</p>
</body></html>
Looking at:
https://resources.ovirt.org/pub/yum-repo/
ovirt-elease-master.rpm does not exist.
So we have 2 issues:
- The file does not exist
- We return "403 Forbidden" instead of "404 Not Found"
3:53 a.m.
Hi Nir,
It looks like the CI job broke it again, breaking 4.2, 4.3 and master
snapshot release RPM.
I guess it's the same issue we had last time.
Thanks in advance,
On Tue, Sep 8, 2020 at 11:45 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com>
wrote:
>
> This is the second time, last time there was an issue during a nightly
job that
> left the master snapshot repo broken.
This is still broken today.
In the past sending mail to infra-support created a bug, and someone
was handling
issue quickly. This seems to be broken now.
We depend on
http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
for development and CI. When the rpm is missing, CI jobs fail, and
developers cannot
update their setup.
Please fix it as soon as possible.
> I have 3 failure builds:
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
>
> All seems to fail in:
>
> [2020-09-07T10:00:56.163Z] + dnf install -y
> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> [2020-09-07T10:01:08.524Z] Status code: 403 for
> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
> 66.187.230.40)
>
> Checking manually show:
>
> $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>403 Forbidden</title>
> </head><body>
> <h1>Forbidden</h1>
> <p>You don't have permission to access
/pub/yum-repo/ovirt-release-master.rpm
> on this server.</p>
> </body></html>
>
> Looking at:
> https://resources.ovirt.org/pub/yum-repo/
>
> ovirt-elease-master.rpm does not exist.
>
> So we have 2 issues:
> - The file does not exist
> - We return "403 Forbidden" instead of "404 Not Found"
--
Lev Veyde
Senior Software Engineer, RHCE | RHCVA | MCITP
Red Hat Israel
<https://www.redhat.com>
lev(a)redhat.com | lveyde(a)redhat.com
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
4:13 a.m.
Hey,
I am looking into this.
Will update with the results I will find.
On Tue, Sep 8, 2020 at 12:00 PM Lev Veyde <lveyde(a)redhat.com> wrote:
Hi Nir,
It looks like the CI job broke it again, breaking 4.2, 4.3 and master
snapshot release RPM.
I guess it's the same issue we had last time.
Thanks in advance,
On Tue, Sep 8, 2020 at 11:45 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
> On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
> >
> > This is the second time, last time there was an issue during a nightly
> job that
> > left the master snapshot repo broken.
>
> This is still broken today.
>
> In the past sending mail to infra-support created a bug, and someone
> was handling
> issue quickly. This seems to be broken now.
>
> We depend on
> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> for development and CI. When the rpm is missing, CI jobs fail, and
> developers cannot
> update their setup.
>
> Please fix it as soon as possible.
>
> > I have 3 failure builds:
> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
> >
> > All seems to fail in:
> >
> > [2020-09-07T10:00:56.163Z] + dnf install -y
> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> > [2020-09-07T10:01:08.524Z] Status code: 403 for
> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
> > 66.187.230.40)
> >
> > Checking manually show:
> >
> > $ curl
> https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> > <html><head>
> > <title>403 Forbidden</title>
> > </head><body>
> > <h1>Forbidden</h1>
> > <p>You don't have permission to access
> /pub/yum-repo/ovirt-release-master.rpm
> > on this server.</p>
> > </body></html>
> >
> > Looking at:
> > https://resources.ovirt.org/pub/yum-repo/
> >
> > ovirt-elease-master.rpm does not exist.
> >
> > So we have 2 issues:
> > - The file does not exist
> > - We return "403 Forbidden" instead of "404 Not Found"
>
>
--
Lev Veyde
Senior Software Engineer, RHCE | RHCVA | MCITP
Red Hat Israel
<https://www.redhat.com>
lev(a)redhat.com | lveyde(a)redhat.com
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
_______________________________________________
Devel mailing list -- devel(a)ovirt.org
To unsubscribe send an email to devel-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/B7VMLC7JKYS...
5:14 a.m.
I found out the root cause was because of space issues in the nightly
publisher run.
The 3 ovirt publishers ran in together and the jenkins dir did not have
that capacity.
Currently, I am restoring the ovirt-master snapshot and then 4.3 + 4.2,
plus sent a patch [1] to ensure they will not run in parallel.
[1]: https://gerrit.ovirt.org/111160
On Tue, Sep 8, 2020 at 12:13 PM Ehud Yonasi <eyonasi(a)redhat.com> wrote:
Hey,
I am looking into this.
Will update with the results I will find.
On Tue, Sep 8, 2020 at 12:00 PM Lev Veyde <lveyde(a)redhat.com> wrote:
> Hi Nir,
>
> It looks like the CI job broke it again, breaking 4.2, 4.3 and master
> snapshot release RPM.
>
> I guess it's the same issue we had last time.
>
> Thanks in advance,
>
> On Tue, Sep 8, 2020 at 11:45 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
>
>> On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
>> >
>> > This is the second time, last time there was an issue during a nightly
>> job that
>> > left the master snapshot repo broken.
>>
>> This is still broken today.
>>
>> In the past sending mail to infra-support created a bug, and someone
>> was handling
>> issue quickly. This seems to be broken now.
>>
>> We depend on
>> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>> for development and CI. When the rpm is missing, CI jobs fail, and
>> developers cannot
>> update their setup.
>>
>> Please fix it as soon as possible.
>>
>> > I have 3 failure builds:
>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
>> >
>> > All seems to fail in:
>> >
>> > [2020-09-07T10:00:56.163Z] + dnf install -y
>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>> > [2020-09-07T10:01:08.524Z] Status code: 403 for
>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
>> > 66.187.230.40)
>> >
>> > Checking manually show:
>> >
>> > $ curl
>> https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>> > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
>> > <html><head>
>> > <title>403 Forbidden</title>
>> > </head><body>
>> > <h1>Forbidden</h1>
>> > <p>You don't have permission to access
>> /pub/yum-repo/ovirt-release-master.rpm
>> > on this server.</p>
>> > </body></html>
>> >
>> > Looking at:
>> > https://resources.ovirt.org/pub/yum-repo/
>> >
>> > ovirt-elease-master.rpm does not exist.
>> >
>> > So we have 2 issues:
>> > - The file does not exist
>> > - We return "403 Forbidden" instead of "404 Not Found"
>>
>>
>
> --
>
> Lev Veyde
>
> Senior Software Engineer, RHCE | RHCVA | MCITP
>
> Red Hat Israel
>
> <https://www.redhat.com>
>
> lev(a)redhat.com | lveyde(a)redhat.com
> <https://red.ht/sig>
> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org
> To unsubscribe send an email to devel-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/B7VMLC7JKYS...
>
6:42 a.m.
On Tue, Sep 8, 2020 at 1:14 PM Ehud Yonasi <eyonasi(a)redhat.com> wrote:
I found out the root cause was because of space issues in the
nightly
publisher run.
The 3 ovirt publishers ran in together and the jenkins dir did not have
that capacity.
Currently, I am restoring the ovirt-master snapshot and then 4.3 + 4.2,
Thanks!
plus sent a patch [1] to ensure they will not run in parallel.
[1]: https://gerrit.ovirt.org/111160
I probably don't understand the change, but it seems that all jobs
will run now at 00:00?
I think we need to work on a better mechanism for updating the repository.
For example, we can have a symlink the to current repo:
current -> 200908143754
200908143754/
repo files...
200908143830/
repo files...
When creating a new repo, create a new directory. Update the "current"
symlink only
if creating the new repo succeeded. Then delete the previous respo.
With this the cases when the entire repo or some files are missing are not
possible.
On Tue, Sep 8, 2020 at 12:13 PM Ehud Yonasi <eyonasi(a)redhat.com> wrote:
> Hey,
>
> I am looking into this.
> Will update with the results I will find.
>
> On Tue, Sep 8, 2020 at 12:00 PM Lev Veyde <lveyde(a)redhat.com> wrote:
>
>> Hi Nir,
>>
>> It looks like the CI job broke it again, breaking 4.2, 4.3 and master
>> snapshot release RPM.
>>
>> I guess it's the same issue we had last time.
>>
>> Thanks in advance,
>>
>> On Tue, Sep 8, 2020 at 11:45 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
>>
>>> On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
>>> >
>>> > This is the second time, last time there was an issue during a
>>> nightly job that
>>> > left the master snapshot repo broken.
>>>
>>> This is still broken today.
>>>
>>> In the past sending mail to infra-support created a bug, and someone
>>> was handling
>>> issue quickly. This seems to be broken now.
>>>
>>> We depend on
>>> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>> for development and CI. When the rpm is missing, CI jobs fail, and
>>> developers cannot
>>> update their setup.
>>>
>>> Please fix it as soon as possible.
>>>
>>> > I have 3 failure builds:
>>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
>>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
>>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
>>> >
>>> > All seems to fail in:
>>> >
>>> > [2020-09-07T10:00:56.163Z] + dnf install -y
>>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>> > [2020-09-07T10:01:08.524Z] Status code: 403 for
>>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
>>> > 66.187.230.40)
>>> >
>>> > Checking manually show:
>>> >
>>> > $ curl
>>> https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>> > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
>>> > <html><head>
>>> > <title>403 Forbidden</title>
>>> > </head><body>
>>> > <h1>Forbidden</h1>
>>> > <p>You don't have permission to access
>>> /pub/yum-repo/ovirt-release-master.rpm
>>> > on this server.</p>
>>> > </body></html>
>>> >
>>> > Looking at:
>>> > https://resources.ovirt.org/pub/yum-repo/
>>> >
>>> > ovirt-elease-master.rpm does not exist.
>>> >
>>> > So we have 2 issues:
>>> > - The file does not exist
>>> > - We return "403 Forbidden" instead of "404 Not
Found"
>>>
>>>
>>
>> --
>>
>> Lev Veyde
>>
>> Senior Software Engineer, RHCE | RHCVA | MCITP
>>
>> Red Hat Israel
>>
>> <https://www.redhat.com>
>>
>> lev(a)redhat.com | lveyde(a)redhat.com
>> <https://red.ht/sig>
>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>> _______________________________________________
>> Devel mailing list -- devel(a)ovirt.org
>> To unsubscribe send an email to devel-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/B7VMLC7JKYS...
>>
>
8:02 a.m.
Hey Nir,
Yeah, we need to work on a better mechanism. I will create a ticket and
discuss it in the infra meeting.
Currently, the ovirt release master rpm is back, and soon the 4.3 + 4.2
will be restored.
Thanks for the report!
On Tue, Sep 8, 2020 at 2:42 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Tue, Sep 8, 2020 at 1:14 PM Ehud Yonasi <eyonasi(a)redhat.com>
wrote:
> I found out the root cause was because of space issues in the nightly
> publisher run.
>
> The 3 ovirt publishers ran in together and the jenkins dir did not have
> that capacity.
>
> Currently, I am restoring the ovirt-master snapshot and then 4.3 + 4.2,
>
Thanks!
> plus sent a patch [1] to ensure they will not run in parallel.
>
> [1]: https://gerrit.ovirt.org/111160
>
I probably don't understand the change, but it seems that all jobs
will run now at 00:00?
I think we need to work on a better mechanism for updating the repository.
For example, we can have a symlink the to current repo:
current -> 200908143754
200908143754/
repo files...
200908143830/
repo files...
When creating a new repo, create a new directory. Update the "current"
symlink only
if creating the new repo succeeded. Then delete the previous respo.
With this the cases when the entire repo or some files are missing are not
possible.
On Tue, Sep 8, 2020 at 12:13 PM Ehud Yonasi <eyonasi(a)redhat.com> wrote:
>
>> Hey,
>>
>> I am looking into this.
>> Will update with the results I will find.
>>
>> On Tue, Sep 8, 2020 at 12:00 PM Lev Veyde <lveyde(a)redhat.com> wrote:
>>
>>> Hi Nir,
>>>
>>> It looks like the CI job broke it again, breaking 4.2, 4.3 and master
>>> snapshot release RPM.
>>>
>>> I guess it's the same issue we had last time.
>>>
>>> Thanks in advance,
>>>
>>> On Tue, Sep 8, 2020 at 11:45 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
>>>
>>>> On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com>
wrote:
>>>> >
>>>> > This is the second time, last time there was an issue during a
>>>> nightly job that
>>>> > left the master snapshot repo broken.
>>>>
>>>> This is still broken today.
>>>>
>>>> In the past sending mail to infra-support created a bug, and someone
>>>> was handling
>>>> issue quickly. This seems to be broken now.
>>>>
>>>> We depend on
>>>> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>>> for development and CI. When the rpm is missing, CI jobs fail, and
>>>> developers cannot
>>>> update their setup.
>>>>
>>>> Please fix it as soon as possible.
>>>>
>>>> > I have 3 failure builds:
>>>> >
>>>> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
>>>> >
>>>> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
>>>> >
>>>> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
>>>> >
>>>> > All seems to fail in:
>>>> >
>>>> > [2020-09-07T10:00:56.163Z] + dnf install -y
>>>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>>> > [2020-09-07T10:01:08.524Z] Status code: 403 for
>>>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>>> (IP:
>>>> > 66.187.230.40)
>>>> >
>>>> > Checking manually show:
>>>> >
>>>> > $ curl
>>>> https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>>> > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
>>>> > <html><head>
>>>> > <title>403 Forbidden</title>
>>>> > </head><body>
>>>> > <h1>Forbidden</h1>
>>>> > <p>You don't have permission to access
>>>> /pub/yum-repo/ovirt-release-master.rpm
>>>> > on this server.</p>
>>>> > </body></html>
>>>> >
>>>> > Looking at:
>>>> > https://resources.ovirt.org/pub/yum-repo/
>>>> >
>>>> > ovirt-elease-master.rpm does not exist.
>>>> >
>>>> > So we have 2 issues:
>>>> > - The file does not exist
>>>> > - We return "403 Forbidden" instead of "404 Not
Found"
>>>>
>>>>
>>>
>>> --
>>>
>>> Lev Veyde
>>>
>>> Senior Software Engineer, RHCE | RHCVA | MCITP
>>>
>>> Red Hat Israel
>>>
>>> <https://www.redhat.com>
>>>
>>> lev(a)redhat.com | lveyde(a)redhat.com
>>> <https://red.ht/sig>
>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>>> _______________________________________________
>>> Devel mailing list -- devel(a)ovirt.org
>>> To unsubscribe send an email to devel-leave(a)ovirt.org
>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/B7VMLC7JKYS...
>>>
>>
1325
days inactive
1326
days old
6 comments
3 participants
participants (3)
-
Ehud Yonasi -
Lev Veyde -
Nir Soffer