infra security update

[Michael Scherer]

Le vendredi 06 juin 2014 à 14:36 +0200, Ewoud Kohl van Wijngaarden a
écrit :
> On Fri, Jun 06, 2014 at 01:29:44PM +0200, Michael Scherer wrote:
> > Due to CVE on openssl and on kernel, I did upgrade various piece of the
> > infrastructure ( foreman, lists, stats, monitoring ), which implied a
> > few reboots ( due to kernel lagging behind, which is not that great with
> > local root exploit ). As this is friday and I assumed most of the Tel
> > Aviv office was not working, i hope this kept the disruption to a
> > minimum. However, if something is broken, please tell it so we can fix.
> 
> Nice work.


> > This also got me thinking. In order to bring a bit more order, what
> > about having a fixed schedule for upgrade ?
> > 
> > In my previous position, we were doing that once per month ( except
> > during end of quarter freeze ), with mandatory reboot ( cause if
> > something do not boot, you want to know it when you have a planned
> > outage, not when everyone is running around updating stuff ). Fedora has
> > a rather complex procedure to decide what to upgrade, hilighted on
> > http://infrastructure.fedoraproject.org/infra/docs/massupgrade.txt
> 
> At my previous employer we had something similar. I also wrote a puppet
> custom fact reboot_needed which checks if the running kernel matches the
> default kernel that would be booted.

We also want to restart services that are affected. Hence a reboot would
be simpler from a engineering point of view.

> > So we could adopt a schedule ( once per month, unless there is something
> > critical, in which case we do it ASAP, with warning on the list and irc
> > ). 
> 
> +1
> 
> > The schedule should of course take in account "business need", which is
> > "release schedule of ovirt".
> > 
> > So what about "first friday of the month, unless exception" ?
> 
> We should also make sure that we don't reboot ALL servers at once. So if
> we have multiple centos 6 jenkins slaves, try to just reboot one at a
> time. Also would be nice if the slave did proper scheduling in jenkins
> so no jobs are running.

Yep, proper orchestration would be nice. Now, if jenkins is resilient
enough ( ie, if it can survive a 5 minutes downtime ), it may not be
that business critical to have it running all the time.

I am not a jenkins expert, not even a user, so I defer to David for
that. 

> > And by update, i mean "yum upgrade -y". Cleaning the list of repo on
> > various servers is also IMHO another task to discuss, to make sure the
> > task can be safely executed. ( having something like
> > mcollective/ansible/func is also needed, but that's more a convenience
> > than a requirement at this stage ).
> 
> We sometimes have pinned versions on jenkins build slaves. That means we
> should either do a proper yum versionlock or find something else. Note
> that I'm all in favor of being able to to a blind yum upgrades.

-- 
Michael Scherer
Open Source and Standards, Sysadmin



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20140606/b66ba6a0/attachment.sig>