[OUTAGE] ovirt.org wiki outage report [07/12/14]

[Michael Scherer]

Le mercredi 10 décembre 2014 à 10:41 +0100, Michael Scherer a écrit :
> Le mardi 09 décembre 2014 à 18:00 -0500, Brian Proffitt a écrit :
> > 
> > ----- Original Message -----
> > > From: "Michael Scherer" <mscherer at redhat.com>
> > > To: infra at ovirt.org
> > > Sent: Monday, December 8, 2014 8:07:33 AM
> > > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
> > > 
> > > Le lundi 08 décembre 2014 à 13:28 +0100, Michael Scherer a écrit :
> > > > Le lundi 08 décembre 2014 à 06:23 -0500, Eyal Edri a écrit :
> > > > > 
> > > > > ----- Original Message -----
> > > > > > From: "Michael Scherer" <mscherer at redhat.com>
> > > > > > To: infra at ovirt.org
> > > > > > Sent: Monday, December 8, 2014 1:03:01 PM
> > > > > > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
> > > > > > 
> > > > > > Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit :
> > > > > > > fyi,
> > > > > > > 
> > > > > > > ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST.
> > > > > > > 
> > > > > > > REASON:
> > > > > > > =======
> > > > > > > after logging into the openshift gear, seems like there was 99.9%
> > > > > > > space
> > > > > > > taken.
> > > > > > > 
> > > > > > > 
> > > > > > > RESOLUTION:
> > > > > > > ===========
> > > > > > > deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted
> > > > > > > mysql
> > > > > > > using 'ctl_app restart'.
> > > > > > > space is still not great - at 84%, so we'll need to think soon how to
> > > > > > > resolve this.
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > wiki is up now and seems to working well.
> > > > > > > infra - we should look into migrating to a new gear maybe? or extend
> > > > > > > again
> > > > > > > the existing one.
> > > > > > 
> > > > > > Happened again today.
> > > > > > 
> > > > > > I still maintain that we should move to a dedicated hosting on the phx2
> > > > > > cluster :)
> > > > > > ( or request quota extensions ).
> > > > > 
> > > > > whether if we'll migrate or not, that will take time and won't happen
> > > > > until the ILO issue will be resolved.
> > > > > for now we need to extend our volume there ASAP, cause it will happen
> > > > > again soon, not too many log files
> > > > > are there to delete/gzip.
> > > > 
> > > > Agreed. We did request already a quota extension, IIRC, so I guess that
> > > > was not sufficient ?
> > > 
> > > Filled the form to get more, will tell you when I have a response.
> > > --
> > > Michael Scherer
> > > Open Source and Standards, Sysadmin
> > > 
> > > _______________________________________________
> > > Infra mailing list
> > > Infra at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/infra
> > > 
> > 
> > 
> > Went out again this afternoon (12/9), but managed to squeeze out a little more disk space running rhc app tidy. (Thanks to Jason and Justin for the assist!)
> > 
> > If there is any way I can help expedite more disk quota, please let me know!
> 
> It take a few days. Last time we did ask, it was request => go to
> marketing for approval => go to the sysadmins team for action. 
> 
> As long as this in the marketing side, I have no visibility ( but can
> ask ). On the openshift sysadmin side, I know where to ping :)

So, today it was full again, and openshift people did contact me. We
increased the quota, and while on it, I did the dump/restore needed to
clean the ever increasing ibdata1 file
( http://www.percona.com/blog/2013/08/20/why-is-the-ibdata1-file-continuously-growing-in-mysql/ ).

It took a few hours to process, and since things would not be fun
without it, my internet connexion started to behave badly.

But now, we are down to 6G out of 10G, so wait and see.
-- 
Michael Scherer
Open Source and Standards, Sysadmin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20141211/52f6c988/attachment.sig>