documenting ovirt.org outages downtime

[Eyal Edri]

----- Original Message -----
> From: "Ewoud Kohl van Wijngaarden" <ewoud+ovirt at kohlvanwijngaarden.nl>
> To: infra at ovirt.org
> Sent: Tuesday, October 22, 2013 1:03:07 PM
> Subject: Re: documenting ovirt.org outages downtime
> 
> On Tue, Oct 22, 2013 at 05:02:40AM -0400, Eyal Edri wrote:
> > Dave Neary wrote:
> > > The access I have is an SSH access (I shared my SSH key with Karsten,
> > > and could then git clone, push, and ssh to the application node). As of
> > > this week, OpenShift apps support teams, so Karsten will be able to add
> > > a number of us to be maintainers of the app, and we will also have usage
> > > of the rhc command line tools.
> >
> > Can you please add current infra team members to the group?
> > or do you want each to send his pub key?
> 
> IIRC Rydekull and I already have our keys there. Last time I used it,
> there were some actions that need the password though (such as using the
> RHC tool to easily restart).
> 
> >
> > >
> > > The contacts on the OpenShift team are #openshift on freenode, or the
> > > OpenShift forum.
> > >
> > > Karsten, does that look complete?
> >
> > I'm still missing real commands/actions on what to do in an outage.
> > examples:
> >  1. how to login and server name
> 
> It's easiest to just use www.ovirt.org, which is a CNAME to
> wiki-ovirt.rhcloud.com. I'm sure the username is now in the usual place.
> 
> >  2. where is our app is located (to those who are not familiar with
> >  openshift)
> 
> Not sure what you exactly mean. Physically? Which directory?
> 
> You should know openshift does a lot through environment variables. For
> example, there's $OPENSHIFT_PHP_LOG_DIR which points to the apache logs,
> $OPENSHIFT_DATA_DIR which points to the persistent data, such as our
> mediawiki image uploads or $OPENSHIFT_MYSQL_DB_LOG_DIR for the mysql
> logs. env is a very useful command on openshift.
> 
> >  3. how to check if disk is full, and ideas on fixing it - (e.g which logs
> >  to delete)
> 
> Disk is limited by standard quotas:
> 
> [wiki-ovirt.rhcloud.com 847edb45aea84198838f915be6faa066]\> quota -s
> Disk quotas for user 847edb45aea84198838f915be6faa066 (uid 3689):
>      Filesystem  blocks   quota   limit   grace   files   quota   limit
>      grace
> /dev/mapper/EBSStore01-user_home01
>                   4911M       0   5120M            9571       0    200k
> 
> It seems currently most space is used by our database (3.5G) and as you
> might know, mysql innodb storage is never returned. Even if you purge
> all tables, the ibdata file can't shrink. That doesn't have to be bad,
> but something to be aware of.
> 
> >  4. email/irc contact for support from openshift
> 
> On what kind of plan are we? If free, I think the support is mostly best
> effort on #openshift and https://www.openshift.com/forums/openshift.

i think that info is great for starts, i added it as 'troubleshooting_infra_issues.txt',
under HOWTOS dir in the normal place on linode.

this is important, cause once the wiki is down, we need an alternate place for DRP procedures. 
so if you think on other issues we might need to put there, please update the file.

Eyal.

> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>