documenting ovirt.org outages downtime

[Ewoud Kohl van Wijngaarden]

On Tue, Oct 22, 2013 at 05:02:40AM -0400, Eyal Edri wrote:
> Dave Neary wrote:
> > The access I have is an SSH access (I shared my SSH key with Karsten,
> > and could then git clone, push, and ssh to the application node). As of
> > this week, OpenShift apps support teams, so Karsten will be able to add
> > a number of us to be maintainers of the app, and we will also have usage
> > of the rhc command line tools.
>
> Can you please add current infra team members to the group?
> or do you want each to send his pub key?

IIRC Rydekull and I already have our keys there. Last time I used it,
there were some actions that need the password though (such as using the
RHC tool to easily restart).

>
> >
> > The contacts on the OpenShift team are #openshift on freenode, or the
> > OpenShift forum.
> >
> > Karsten, does that look complete?
>
> I'm still missing real commands/actions on what to do in an outage.
> examples:
>  1. how to login and server name

It's easiest to just use www.ovirt.org, which is a CNAME to
wiki-ovirt.rhcloud.com. I'm sure the username is now in the usual place.

>  2. where is our app is located (to those who are not familiar with openshift)

Not sure what you exactly mean. Physically? Which directory?

You should know openshift does a lot through environment variables. For
example, there's $OPENSHIFT_PHP_LOG_DIR which points to the apache logs,
$OPENSHIFT_DATA_DIR which points to the persistent data, such as our
mediawiki image uploads or $OPENSHIFT_MYSQL_DB_LOG_DIR for the mysql
logs. env is a very useful command on openshift.

>  3. how to check if disk is full, and ideas on fixing it - (e.g which logs to delete)

Disk is limited by standard quotas:

[wiki-ovirt.rhcloud.com 847edb45aea84198838f915be6faa066]\> quota -s
Disk quotas for user 847edb45aea84198838f915be6faa066 (uid 3689):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/mapper/EBSStore01-user_home01
                  4911M       0   5120M            9571       0    200k

It seems currently most space is used by our database (3.5G) and as you
might know, mysql innodb storage is never returned. Even if you purge
all tables, the ibdata file can't shrink. That doesn't have to be bad,
but something to be aware of.

>  4. email/irc contact for support from openshift

On what kind of plan are we? If free, I think the support is mostly best
effort on #openshift and https://www.openshift.com/forums/openshift.