I found out the root cause was because of space issues in the nightly publisher run.

The 3 ovirt publishers ran in together and the jenkins dir did not have that capacity.

Currently, I am restoring the ovirt-master snapshot and then 4.3 + 4.2, plus sent a patch [1] to ensure they will not run in parallel.

[1]: https://gerrit.ovirt.org/111160


On Tue, Sep 8, 2020 at 12:13 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
Hey,

I am looking into this.
Will update with the results I will find.

On Tue, Sep 8, 2020 at 12:00 PM Lev Veyde <lveyde@redhat.com> wrote:
Hi Nir,

It looks like the CI job broke it again, breaking 4.2, 4.3 and master snapshot release RPM.

I guess it's the same issue we had last time.

Thanks in advance,

On Tue, Sep 8, 2020 at 11:45 AM Nir Soffer <nsoffer@redhat.com> wrote:
On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer@redhat.com> wrote:
>
> This is the second time, last time there was an issue during a nightly job that
> left the master snapshot repo broken.

This is still broken today.

In the past sending mail to infra-support created a bug, and someone
was handling
issue quickly. This seems to be broken now.

We depend on http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
for development and CI. When the rpm is missing, CI jobs fail, and
developers cannot
update their setup.

Please fix it as soon as possible.

> I have 3 failure builds:
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
>
> All seems to fail in:
>
> [2020-09-07T10:00:56.163Z] + dnf install -y
> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> [2020-09-07T10:01:08.524Z] Status code: 403 for
> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
> 66.187.230.40)
>
> Checking manually show:
>
> $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>403 Forbidden</title>
> </head><body>
> <h1>Forbidden</h1>
> <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm
> on this server.</p>
> </body></html>
>
> Looking at:
> https://resources.ovirt.org/pub/yum-repo/
>
> ovirt-elease-master.rpm does not exist.
>
> So we have 2 issues:
> - The file does not exist
> - We return "403 Forbidden" instead of "404 Not Found"



--

Lev Veyde

Senior Software Engineer, RHCE | RHCVA | MCITP

Red Hat Israel

lev@redhat.com | lveyde@redhat.com

_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/B7VMLC7JKYSU2LI6TRMEAJCXXM3NLSSG/