<div dir="ltr">There is another issue, which is currently failing all CQ, and its related to the new IBRS CPU model.<div>It looks like all of the lago slaves were upgraded to new Libvirt and kernel on Friday, while we still don't have a fix on lago-ost-plugin for that. </div><div><br></div><div>I think there was a misunderstanding about what to upgrade, and it might have been understood that only the bios upgrade breaks it and not the kernel one.</div><div><br></div><div>In any case, we're currently fixing the issue, either by downgrading the relevant pkgs on lago slaves or adding the mapping to new CPU types from OST.</div><div><br></div><div>For future, I suggest a few updates to maintenance work on Jenkins slaves ( VMs or BM ):</div><div><br></div><div>1. Let's avoid doing an upgrade close to a weekend ( i.e not on Thu-Sun ), so all the team can be around to help if needed or if something unexpected happens.</div><div>2. When we have a system-wide upgrade scheduled, like all BM slaves or VMs for a specific OS, let's adopt a gradual upgrade with a few days window in between, </div><div> e.g, if we need to upgrade all Lago slaves, let's upgrade 1-2 and wait to see if nothing breaks and continue after we verify OST runs ( either seeing on CQ or running manually ) </div><div><br></div><div><br></div><div>Thoughts? </div><div><br></div><div><br><div class="gmail_extra"><div class="gmail_quote">On Sun, Jan 21, 2018 at 10:42 AM, Barak Korren <span dir="ltr"><<a href="mailto:bkorren@redhat.com" target="_blank">bkorren@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
We seen a great deal of noise coming from the change queue this<br>
weekend. While a part of it is due to actual code regressions, some of<br>
that was actually due to two separate infra issues.<br>
<br>
One issue we had was with building FC26 packages - it turns out that a<br>
yum-incompatible update if the 'cmake' package was introduced to the<br>
FC26 updates repo. Since for the time being we use 'yum' to setup the<br>
mock environments, the build jobs for FC26 started failing.<br>
<br>
This issue was actually reported to us [1].<br>
<br>
To resolve this - we rolled back the FC26 mirror to a time before the<br>
breaking change was introduced, and then re-triggered the merge events<br>
for all the patches that failed building to introduce passing build to<br>
the change queue.<br>
<br>
The second issue had to do with the introduction of FC27 slaves - it<br>
seems that slaves were misconfigured [2] and did not include vary<br>
basic packages like 'git' - this caused the CQ master job to simply<br>
crash and stop queue processing.<br>
<br>
To resolve this issue we disabled the FC27 slaves, resumed CQ<br>
operation and then re-sent all changes that failed to be added into<br>
the queue.<br>
<br>
We are in the final phases of integrating a new oVirt release, so<br>
proper CQ operation is crucial at this time. Additionally, due to a<br>
substantial amount of regressions introduced last week, the CQ<br>
currently has a huge backlog of ~180 changes to work through, this<br>
means that every bisection takes 8 OST runs, so we have no CQ minutes<br>
to spare.<br>
<br>
The FC27 slaves issue cost us 11 hours in which the CQ was not<br>
running. It also manifested itself in failures of the<br>
'standard-enqueue' job. These kinds of failures need to be handled<br>
promptly or be avoided altogether.<br>
<br>
Build failures can make the CQ waste time too, as it runs bisections<br>
to detect and remove changes that fail to build. At this time, a<br>
single failed build can waste up to 8 hours!<br>
<br>
Lets try to be more careful about introducing infrastructure changes<br>
to the system at sensitive times, and be more vigilant about failure<br>
reports from jenkins.<br>
<br>
[1]: <a href="https://ovirt-jira.atlassian.net/browse/OVIRT-1854" rel="noreferrer" target="_blank">https://ovirt-jira.atlassian.<wbr>net/browse/OVIRT-1854</a><br>
[2]: <a href="https://ovirt-jira.atlassian.net/browse/OVIRT-1855" rel="noreferrer" target="_blank">https://ovirt-jira.atlassian.<wbr>net/browse/OVIRT-1855</a><br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Barak Korren<br>
RHV DevOps team , RHCE, RHCi<br>
Red Hat EMEA<br>
<a href="http://redhat.com" rel="noreferrer" target="_blank">redhat.com</a> | TRIED. TESTED. TRUSTED. | <a href="http://redhat.com/trusted" rel="noreferrer" target="_blank">redhat.com/trusted</a><br>
______________________________<wbr>_________________<br>
Infra mailing list<br>
<a href="mailto:Infra@ovirt.org">Infra@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/infra" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/infra</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><p style="font-family:overpass,sans-serif;margin:0px;padding:0px;font-size:14px;text-transform:uppercase;font-weight:bold"><font color="#cc0000">Eyal edri</font></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><br></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">MANAGER</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">RHV DevOps</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">EMEA VIRTUALIZATION R&D</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><br></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat EMEA</a></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" style="color:rgb(17,85,204)" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></td></tr></tbody></table></div><div>phone: +972-9-7692018<br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div></div></div></div></div>
</div></div></div>