infra security update

[Ewoud Kohl van Wijngaarden]

On Fri, Jun 06, 2014 at 01:29:44PM +0200, Michael Scherer wrote:
> Due to CVE on openssl and on kernel, I did upgrade various piece of the
> infrastructure ( foreman, lists, stats, monitoring ), which implied a
> few reboots ( due to kernel lagging behind, which is not that great with
> local root exploit ). As this is friday and I assumed most of the Tel
> Aviv office was not working, i hope this kept the disruption to a
> minimum. However, if something is broken, please tell it so we can fix.

Nice work.

> This also got me thinking. In order to bring a bit more order, what
> about having a fixed schedule for upgrade ?
> 
> In my previous position, we were doing that once per month ( except
> during end of quarter freeze ), with mandatory reboot ( cause if
> something do not boot, you want to know it when you have a planned
> outage, not when everyone is running around updating stuff ). Fedora has
> a rather complex procedure to decide what to upgrade, hilighted on
> http://infrastructure.fedoraproject.org/infra/docs/massupgrade.txt

At my previous employer we had something similar. I also wrote a puppet
custom fact reboot_needed which checks if the running kernel matches the
default kernel that would be booted.

> So we could adopt a schedule ( once per month, unless there is something
> critical, in which case we do it ASAP, with warning on the list and irc
> ). 

+1

> The schedule should of course take in account "business need", which is
> "release schedule of ovirt".
> 
> So what about "first friday of the month, unless exception" ?

We should also make sure that we don't reboot ALL servers at once. So if
we have multiple centos 6 jenkins slaves, try to just reboot one at a
time. Also would be nice if the slave did proper scheduling in jenkins
so no jobs are running.

> And by update, i mean "yum upgrade -y". Cleaning the list of repo on
> various servers is also IMHO another task to discuss, to make sure the
> task can be safely executed. ( having something like
> mcollective/ansible/func is also needed, but that's more a convenience
> than a requirement at this stage ).

We sometimes have pinned versions on jenkins build slaves. That means we
should either do a proper yum versionlock or find something else. Note
that I'm all in favor of being able to to a blind yum upgrades.