Wiki and Mailing Lists Outage -- 2012-11-14

[Mike Burns]

On Wed, 2012-11-14 at 08:45 -0500, Doron Fediuck wrote:
> Thanks Mike!
> I suggest to have a cron alerting for no-space issues.

We run logwatch which is supposed to highlight these issues, but I
suspect that no one is actually reading the logwatch report.  A separate
cron job or monitoring service is also a possibility.

Mike
> 
> ----- Original Message -----
> > From: "Mike Burns" <mburns at redhat.com>
> > To: "board" <board at ovirt.org>, "infra" <infra at ovirt.org>, "users" <users at ovirt.org>
> > Sent: Wednesday, November 14, 2012 3:31:11 PM
> > Subject: Wiki and Mailing Lists Outage -- 2012-11-14
> > 
> > We experienced an outage today in both the wiki and the mailing
> > lists.
> > 
> > * Wiki content was available throughout the outage, but attempts to
> > login or edit received an error message about requiring cookies to be
> > enabled.
> > * All mails to the mailing  list failed to show up on the lists, but
> > also did not return rejection messages.
> > 
> > Cause:
> > 
> > This was caused by an "Out of Space" error on the host running both
> > of
> > these services.  A temporary workaround was put in place to get both
> > services up and running again.
> > 
> > 
> > Action Taken:
> > 
> > Remove the oldest gerrit backup (600MB)
> > Remove some older non-functional ovirt-node-iso images and rpms from
> > the
> > releases (source remains there)
> > 
> > Long term solution:
> > 
> > Migrating these services away from a single host onto hosted
> > solutions
> > (OpenShift, AlterWay).
> > 
> > Current Situation:
> > 
> > Wiki is back up and running, login works as expected
> > Lists are processing the backlog of emails since the outage began.
> > At this time, it does not appear that any mail was lost due to the
> > outage.
> > 
> > 
> > Thanks for the patience and understanding
> > 
> > Mike
> > 
> > _______________________________________________
> > Infra mailing list
> > Infra at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> > 
> _______________________________________________
> Board mailing list
> Board at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/board