infra security update

[Michael Scherer]

Le dimanche 08 juin 2014 à 02:55 -0400, Eyal Edri a écrit :
> 
> ----- Original Message -----
> > From: "Michael Scherer" <mscherer at redhat.com>
> > To: infra at ovirt.org
> > Sent: Friday, June 6, 2014 2:29:44 PM
> > Subject: infra security update
> > 
> > Hi,
> > 
> > Due to CVE on openssl and on kernel, I did upgrade various piece of the
> > infrastructure ( foreman, lists, stats, monitoring ), which implied a
> > few reboots ( due to kernel lagging behind, which is not that great with
> > local root exploit ). As this is friday and I assumed most of the Tel
> > Aviv office was not working, i hope this kept the disruption to a
> > minimum. However, if something is broken, please tell it so we can fix.
> > 
> > This also got me thinking. In order to bring a bit more order, what
> > about having a fixed schedule for upgrade ?
> 
> 
> 
> > 
> > In my previous position, we were doing that once per month ( except
> > during end of quarter freeze ), with mandatory reboot ( cause if
> > something do not boot, you want to know it when you have a planned
> > outage, not when everyone is running around updating stuff ). Fedora has
> > a rather complex procedure to decide what to upgrade, hilighted on
> > http://infrastructure.fedoraproject.org/infra/docs/massupgrade.txt
> > 
> > So we could adopt a schedule ( once per month, unless there is something
> > critical, in which case we do it ASAP, with warning on the list and irc
> > ).
> 
> 
> 
> > 
> > The schedule should of course take in account "business need", which is
> > "release schedule of ovirt".
> > 
> > So what about "first friday of the month, unless exception" ?
> > 
> > And by update, i mean "yum upgrade -y". Cleaning the list of repo on
> > various servers is also IMHO another task to discuss, to make sure the
> > task can be safely executed. ( having something like
> > mcollective/ansible/func is also needed, but that's more a convenience
> > than a requirement at this stage ).
> 
> we use 'fabric' for these kind of stuff in redhat, so we might be able to 
> use that for oVirt as well.

wel, IT use taboot/func, and Fedora use ansible ( some part of IT also
do look at ansible ). Openshift use mcollective.

basically, what we need is a tool to enforce configuration ( which is
puppet ), and one to execute remote command with a specific schedule and
orchestration. 

For a simple task, they would likely all work fine, so maybe we should
start to list what we want, and then see which one fullfill the need and
some constraints.

based on the issue I faced in the past, and my understanding of the
sitaution, I would propose the following list of criteria:

- be packaged for Fedora/Centos ( agent and server )
 - packaged in epel/main repo (so we can be sure there is a proper
update policy)
 - optionally, for Debian/Ubuntu (cf the need of having jenkins builder
on others platform)
- the simple, the better (I am not fond of having to deploy amqp for
mcollective)
- know by at least a few people in the current team (the more, the
better)
- able to orchestrate, ie, run tasks on X server on after the other
- still maintained (ie, unlike func)
- not moving too fast

I guess we for now want to have it run the tasks of upgrades, but in the
event of needing more ( regenerate ssl certificates, restart some
specific service ), I guess we may need more.

We should also take a look at the infra deployment, ie do we have stuff
over ssh ( ansible, fabric ), which mean giving a lot of access
(basically, root), with likely a need for using something like a
restricted user and a bastion. Or there is mcollective, salt, func,
where you have a 2nd communication channel and fine grained ACL



> +1 for monthly maintenance window,Friday sounds good, since most of the users are from tlv office.
> we can keep sunday as optional also, if a critical server should be up on a certain friday.
> so either tlv office of non-tlv can performan the outages.
> 
> also, worth adding it to the calendar, as a monthly maintenance outage,
> where we can update servers like jenkins/gerrit/formean etc...
> we can use either ovirt cal [1] or open a new infra cal for that. 

Also, what happen if we forget. And how do we make sure that more than 1
people can do the job.

> thought, we should map pkg we want to keep latest, and ensure them via puppet,
> while the maintenance windows will be used for reboots and downtimes.
> 
> we also should update that info on the wiki once ready [2]
> 
> [1] https://www.google.com/calendar/ical/ppqtk46u9cglj7l987ruo2l0f8%40group.calendar.google.com/public/basic.ics
> [2] http://www.ovirt.org/Infra

/me still have to enable https on the wiki
-- 
Michael Scherer
Open Source and Standards, Sysadmin



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20140609/051ce1db/attachment.sig>