Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]
6:36 a.m.
Thank you.
I was not sure if this is repo related since I could not see a specific
package that it failed on.
Sandro believes this might have been a repo outage so I am opening a
ticket to try and investigate this issue and find the root cause.
https://ovirt-jira.atlassian.net/browse/OVIRT-1693
Thanks,
Dafna
On 10/12/2017 11:37 AM, Viktor Mihajlovski wrote:
On 12.10.2017 12:27, Yaniv Kaul wrote:
> Repo issues (again?)
> See log[1].
> Y.
>
> [1]
>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3104/artifa...
>
> On Thu, Oct 12, 2017 at 12:26 PM, Dafna Ron <dron(a)redhat.com> wrote:
>
I agree, seems to be unrelated to my patch.
12:51 p.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]
On Thu, Oct 12, 2017 at 2:36 PM, Dafna Ron <dron(a)redhat.com> wrote:
Thank you.
I was not sure if this is repo related since I could not see a specific
package that it failed on.
Sandro believes this might have been a repo outage so I am opening a
ticket to try and investigate this issue and find the root cause.
https://ovirt-jira.atlassian.net/browse/OVIRT-1693
We have far too many repo outages.
I believe it could be partially solved by properly and consistently keeping
the reposync up-to-date.
It's far from bullet-proof, and is annoying work, but we need to once every
other week or so to just do it, to ensure we can perform offline
installation (I don't believe it's a complete repo outage, but partial,
which is why I think it'll help).
Y.
Thanks,
Dafna
On 10/12/2017 11:37 AM, Viktor Mihajlovski wrote:
> On 12.10.2017 12:27, Yaniv Kaul wrote:
>> Repo issues (again?)
>> See log[1].
>> Y.
>>
>> [1]
>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-
tester/3104/artifact/exported-artifacts/basic-suit-master-
el7/003_00_metrics_bootstrap.py.junit.xml
>>
>> On Thu, Oct 12, 2017 at 12:26 PM, Dafna Ron <dron(a)redhat.com> wrote:
>>
> I agree, seems to be unrelated to my patch.
>
1:02 p.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]
On Thu, Oct 12, 2017 at 8:51 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
On Thu, Oct 12, 2017 at 2:36 PM, Dafna Ron <dron(a)redhat.com> wrote:
> Thank you.
> I was not sure if this is repo related since I could not see a specific
> package that it failed on.
> Sandro believes this might have been a repo outage so I am opening a
> ticket to try and investigate this issue and find the root cause.
>
> https://ovirt-jira.atlassian.net/browse/OVIRT-1693
We have far too many repo outages.
I believe it could be partially solved by properly and consistently
keeping the reposync up-to-date.
It's far from bullet-proof, and is annoying work, but we need to once
every other week or so to just do it, to ensure we can perform offline
installation (I don't believe it's a complete repo outage, but partial,
which is why I think it'll help).
Spoke too soon:
yum.Errors.NoMoreMirrorsRepoError: failure: repodata/repomd.xml from
centos-ovirt-4.2-el7: [Errno 256] No more mirrors to try.
http://cbs.centos.org/repos/virt7-ovirt-42-testing/x86_64/os/repodata/rep...:
[Errno 14] HTTP Error 404 - Not Found
Perhaps we are not using mirror links properly.
Y.
Y.
>
>
> Thanks,
> Dafna
>
>
> On 10/12/2017 11:37 AM, Viktor Mihajlovski wrote:
> > On 12.10.2017 12:27, Yaniv Kaul wrote:
> >> Repo issues (again?)
> >> See log[1].
> >> Y.
> >>
> >> [1]
> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
> r/3104/artifact/exported-artifacts/basic-suit-master-el7/
> 003_00_metrics_bootstrap.py.junit.xml
> >>
> >> On Thu, Oct 12, 2017 at 12:26 PM, Dafna Ron <dron(a)redhat.com> wrote:
> >>
> > I agree, seems to be unrelated to my patch.
> >
>
>
4:29 a.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]
This is a multi-part message in MIME format.
--------------A751BD54E68444A2C28A68D7
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Adding Eyal, Barak and Sandro and removing Victor.
Personally, I do not mind taking on this task, but I think that I would
need help in creating such a task.
On 10/12/2017 06:51 PM, Yaniv Kaul wrote:
On Thu, Oct 12, 2017 at 2:36 PM, Dafna Ron <dron(a)redhat.com
<mailto:dron@redhat.com>> wrote:
Thank you.
I was not sure if this is repo related since I could not see a
specific
package that it failed on.
Sandro believes this might have been a repo outage so I am opening a
ticket to try and investigate this issue and find the root cause.
https://ovirt-jira.atlassian.net/browse/OVIRT-1693
<https://ovirt-jira.atlassian.net/browse/OVIRT-1693>
We have far too many repo outages.
I believe it could be partially solved by properly and consistently
keeping the reposync up-to-date.
It's far from bullet-proof, and is annoying work, but we need to once
every other week or so to just do it, to ensure we can perform offline
installation (I don't believe it's a complete repo outage, but
partial, which is why I think it'll help).
Y.
Thanks,
Dafna
On 10/12/2017 11:37 AM, Viktor Mihajlovski wrote:
> On 12.10.2017 12:27, Yaniv Kaul wrote:
>> Repo issues (again?)
>> See log[1].
>> Y.
>>
>> [1]
>>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3104/artifa...
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3104/artifa...
>>
>> On Thu, Oct 12, 2017 at 12:26 PM, Dafna Ron <dron(a)redhat.com
<mailto:dron@redhat.com>> wrote:
>>
> I agree, seems to be unrelated to my patch.
>
--------------A751BD54E68444A2C28A68D7
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Adding Eyal, Barak and Sandro and
removing Victor. <br>
<br>
Personally, I do not mind taking on this task, but I think that I
would need help in creating such a task. <br>
<br>
On 10/12/2017 06:51 PM, Yaniv Kaul wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJgorsZVc1vQRHeA2nVJVAwk5Kx-iQjQkESiSix7z5K4Ayy5Cw@mail.gmail.com">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Oct 12, 2017 at 2:36 PM,
Dafna Ron <span dir="ltr"><<a
href="mailto:dron@redhat.com" target="_blank"
moz-do-not-send="true">dron(a)redhat.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Thank
you.<br>
I was not sure if this is repo related since I could not
see a specific<br>
package that it failed on.<br>
Sandro believes this might have been a repo outage so I am
opening a<br>
ticket to try and investigate this issue and find the root
cause.<br>
<br>
<a
href="https://ovirt-jira.atlassian.net/browse/OVIRT-1693"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://ovirt-jira.atlassian.<wbr>net/browse/OVIRT-1693</a></blockquote>
<div><br>
</div>
<div>We have far too many repo outages.</div>
<div>I believe it could be partially solved by properly and
consistently keeping the reposync up-to-date.</div>
<div>It's far from bullet-proof, and is annoying work, but
we need to once every other week or so to just do it, to
ensure we can perform offline installation (I don't
believe it's a complete repo outage, but partial, which is
why I think it'll help).</div>
<div>Y.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
Thanks,<br>
Dafna<br>
<div class="HOEnZb">
<div class="h5"><br>
<br>
On 10/12/2017 11:37 AM, Viktor Mihajlovski wrote:<br>
> On 12.10.2017 12:27, Yaniv Kaul wrote:<br>
>> Repo issues (again?)<br>
>> See log[1].<br>
>> Y.<br>
>><br>
>> [1]<br>
>> <a
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://jenkins.ovirt.org/job/<wbr&...
>><br>
>> On Thu, Oct 12, 2017 at 12:26 PM, Dafna Ron
<<a href="mailto:dron@redhat.com"
moz-do-not-send="true">dron(a)redhat.com</a>&gt;
wrote:<br>
>><br>
> I agree, seems to be unrelated to my patch.<br>
><br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>
--------------A751BD54E68444A2C28A68D7--
6:31 a.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]
2017-10-13 11:29 GMT+02:00 Dafna Ron <dron(a)redhat.com>:
Adding Eyal, Barak and Sandro and removing Victor.
Personally, I do not mind taking on this task, but I think that I would
need help in creating such a task.
I think we are already mirroring everything and refreshing the mirror every
few hours.
Issue looks like we are not using them in some jobs.
On 10/12/2017 06:51 PM, Yaniv Kaul wrote:
On Thu, Oct 12, 2017 at 2:36 PM, Dafna Ron <dron(a)redhat.com> wrote:
> Thank you.
> I was not sure if this is repo related since I could not see a specific
> package that it failed on.
> Sandro believes this might have been a repo outage so I am opening a
> ticket to try and investigate this issue and find the root cause.
>
> https://ovirt-jira.atlassian.net/browse/OVIRT-1693
We have far too many repo outages.
I believe it could be partially solved by properly and consistently
keeping the reposync up-to-date.
It's far from bullet-proof, and is annoying work, but we need to once
every other week or so to just do it, to ensure we can perform offline
installation (I don't believe it's a complete repo outage, but partial,
which is why I think it'll help).
Y.
>
>
> Thanks,
> Dafna
>
>
> On 10/12/2017 11:37 AM, Viktor Mihajlovski wrote:
> > On 12.10.2017 12:27, Yaniv Kaul wrote:
> >> Repo issues (again?)
> >> See log[1].
> >> Y.
> >>
> >> [1]
> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
> r/3104/artifact/exported-artifacts/basic-suit-master-el7/
> 003_00_metrics_bootstrap.py.junit.xml
> >>
> >> On Thu, Oct 12, 2017 at 12:26 PM, Dafna Ron <dron(a)redhat.com> wrote:
> >>
> > I agree, seems to be unrelated to my patch.
> >
>
>
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
<http://www.teraplan.it/redhat-osd-2017/>
2:15 a.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]
On 13 October 2017 at 14:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
2017-10-13 11:29 GMT+02:00 Dafna Ron <dron(a)redhat.com>:
> Adding Eyal, Barak and Sandro and removing Victor.
>
> Personally, I do not mind taking on this task, but I think that I would
> need help in creating such a task.
>
I think we are already mirroring everything and refreshing the mirror
every few hours.
Issue looks like we are not using them in some jobs.
I did not look deeply into this particular issue, but just to get everyone
on the same page.
* We have mirror system that is getting synced every 8 hours and has no
known issues
ATM
* All repo issues you're seeing in OST are due to one of the following two
reasons:
1. We are white-listing packages into the OST environment and the list
needs to
be maintained as package dependencies change
2. The OST VMs are not blocked from using the upstream CentOS
repos/mirrors. And
the upstream repos are not updated in an atomic fashion
We have ongoing work [1] to fix issue #2 above, it takes time because it
requires meticulous work to get all the required things into the whitelist.
BTW when you see these issues in OST that are doe to upstream CentOS repos
not being updated atomically, it usually correlates with a similar failure
in the mirror sync job.
[1]: https://ovirt-jira.atlassian.net/browse/OVIRT-1280
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
2:57 a.m.
New subject: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]
On Sun, Oct 15, 2017 at 10:15 AM, Barak Korren <bkorren(a)redhat.com> wrote:
On 13 October 2017 at 14:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
>
> 2017-10-13 11:29 GMT+02:00 Dafna Ron <dron(a)redhat.com>:
>
>> Adding Eyal, Barak and Sandro and removing Victor.
>>
>> Personally, I do not mind taking on this task, but I think that I would
>> need help in creating such a task.
>>
>
> I think we are already mirroring everything and refreshing the mirror
> every few hours.
> Issue looks like we are not using them in some jobs.
>
I did not look deeply into this particular issue, but just to get everyone
on the same page.
* We have mirror system that is getting synced every 8 hours and has no
known issues
ATM
* All repo issues you're seeing in OST are due to one of the following two
reasons:
1. We are white-listing packages into the OST environment and the list
needs to
be maintained as package dependencies change
2. The OST VMs are not blocked from using the upstream CentOS
repos/mirrors. And
the upstream repos are not updated in an atomic fashion
We have ongoing work [1] to fix issue #2 above, it takes time because it
requires meticulous work to get all the required things into the whitelist.
Since I send here and there patches to do this meticulous work, I know it's
not such a big deal.
Yes, it's annoying and I have not yet come up with an automated way to do
it (I'm sure there is!), but it takes few hours and we can do it once every
2 weeks or so.
It also has the nice benefit of reducing run time, sometimes dramatically.
Y.
BTW when you see these issues in OST that are doe to upstream CentOS repos
not being updated atomically, it usually correlates with a similar failure
in the mirror sync job.
[1]: https://ovirt-jira.atlassian.net/browse/OVIRT-1280
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
2551
days inactive
2554
days old
6 comments
4 participants
participants (4)
-
Barak Korren -
Dafna Ron -
Sandro Bonazzola -
Yaniv Kaul