2:10 p.m.
This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enigAC7E381E3EB9CC3AB6A2AEE6
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Hi:
I didn't really participate in this outage, so I thought others could
help us draft up notes about it. I put some barebones below.
One outcome we need to look at is, what do people do when they perceive
services are out?
Of course, if the service is the wiki, they can't check that for what to
do ...
How do we communicate when major communication services of ovirt.org are
down? IRC is great but not enough ... If we can arrange for a reliable
third-party mail relay to alias a page to the Infra team, great, but how
do we keep it from getting spam?
Another angle to resolve is service monitoring so we know when things go
out rather than waiting for service users to tell us. I got some direct
emails from people (since the infra@ list wasn't working), but I was
unavailable and unaware of the problem until Robert called me when he
was working on fixing it. I don't mind getting pager alerts, as long as
we can tune things so they are not crazy often. :)
=3D=3D What occurred =3D=3D
Even the doubled disk space on linode01.ovirt.ort (to 25 GB) wasn't
enough to last long.
=3D=3D When =3D=3D
XXXX?
date -d "2012-08-30 XXXX UTC"
=3D=3D Affected services =3D=3D
lists.ovirt.org
wiki.ovirt.org
ovirt.org/.*
ovirtbot
Gerritt backup
Jenkins backup
[[What else?]]
=3D=3D Responses to take =3D=3D
* Get new hosting solution in place.
* Double current disk space before new hosting move, to give us room to
breath.
* Work up a response place that is posted in the IRC topic or somewhere
good so people know how to contact all of the Infra team when something
is happening.
* New service need: monitoring server
--=20
Karsten 'quaid' Wade, Sr. Analyst - Community Growth
http://TheOpenSourceWay.org .^\ http://community.redhat.com
@quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41
--------------enigAC7E381E3EB9CC3AB6A2AEE6
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iD8DBQFQQSg82ZIOBq0ODEERAiERAJkB6BhMquAAw1afh4vAsvguFAW78wCfQVR7
rkZrO1s+Ym9Qi3ge4x6qj4c=
=ZjRV
-----END PGP SIGNATURE-----
--------------enigAC7E381E3EB9CC3AB6A2AEE6--
1:43 p.m.
----- Original Message -----
Hi:
I didn't really participate in this outage, so I thought others could
help us draft up notes about it. I put some barebones below.
One outcome we need to look at is, what do people do when they
perceive
services are out?
Of course, if the service is the wiki, they can't check that for what
to
do ...
How do we communicate when major communication services of ovirt.org
are
down? IRC is great but not enough ... If we can arrange for a
reliable
third-party mail relay to alias a page to the Infra team, great, but
how
do we keep it from getting spam?
Another angle to resolve is service monitoring so we know when things
go
out rather than waiting for service users to tell us. I got some
direct
emails from people (since the infra@ list wasn't working), but I was
unavailable and unaware of the problem until Robert called me when he
was working on fixing it. I don't mind getting pager alerts, as long
as
we can tune things so they are not crazy often. :)
I think we should have multiple places that we notify.
1. IRC
2. wiki page
3. infra list
4. someplace on wordpress (preferably the main page).
This should be sufficient long term (i.e. once we have a better hosting solution than just
the kitchen sink box.
I agree with getting service monitoring set up as well. We can even accomplish this to a
certain extent with jenkins (and a separate non-jenkins cron job to monitor jenkins).
Mike
== What occurred ==
Even the doubled disk space on linode01.ovirt.ort (to 25 GB) wasn't
enough to last long.
== When ==
XXXX?
date -d "2012-08-30 XXXX UTC"
== Affected services ==
lists.ovirt.org
wiki.ovirt.org
ovirt.org/.*
ovirtbot
Gerritt backup
Jenkins backup
[[What else?]]
== Responses to take ==
* Get new hosting solution in place.
* Double current disk space before new hosting move, to give us room
to
breath.
* Work up a response place that is posted in the IRC topic or
somewhere
good so people know how to contact all of the Infra team when
something
is happening.
* New service need: monitoring server
--
Karsten 'quaid' Wade, Sr. Analyst - Community Growth
http://TheOpenSourceWay.org .^\ http://community.redhat.com
@quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
7:26 p.m.
On 09/01/2012 04:43 PM, Mike Burns wrote:
----- Original Message -----
> Hi:
>
> I didn't really participate in this outage, so I thought others could
> help us draft up notes about it. I put some barebones below.
>
> One outcome we need to look at is, what do people do when they
> perceive
> services are out?
>
> Of course, if the service is the wiki, they can't check that for what
> to
> do ...
>
> How do we communicate when major communication services of ovirt.org
> are
> down? IRC is great but not enough ... If we can arrange for a
> reliable
> third-party mail relay to alias a page to the Infra team, great, but
> how
> do we keep it from getting spam?
>
> Another angle to resolve is service monitoring so we know when things
> go
> out rather than waiting for service users to tell us. I got some
> direct
> emails from people (since the infra@ list wasn't working), but I was
> unavailable and unaware of the problem until Robert called me when he
> was working on fixing it. I don't mind getting pager alerts, as long
> as
> we can tune things so they are not crazy often. :)
I think we should have multiple places that we notify.
1. IRC
2. wiki page
3. infra list
4. someplace on wordpress (preferably the main page).
I agree that is a good list.
We need to make sure that we have each
other cell phone info in the event we need to get a hold of someone
although that might be hard for the people in diff county's al
This should be sufficient long term (i.e. once we have a better
hosting solution than just the kitchen sink box.
I agree with getting service monitoring set up as well. We can even accomplish this to a
certain extent with jenkins (and a separate non-jenkins cron job to monitor jenkins).
Mike
Monitor requires hardware unless someone is able to add it to there
existing system. I use a very basic monitor inside my Cerberus Email
Response / Help desk system. We can also add something to do basic jobs
inside Jenkins. I have a 5 Seat license for Cerberus that would allow
us to have a ticketing / monitoring system it is written in php and
pretty easy to customize. I am heavenly involved in the project and I
am already running Cerberus on my own VPS.
> == What occurred ==
>
> Even the doubled disk space on linode01.ovirt.ort (to 25 GB) wasn't
> enough to last long.
I made a mistake and the ripper script wasn't purging
old ovirt-node.iso
using up a lot of space.
>
> == When ==
>
> XXXX?
>
> date -d "2012-08-30 XXXX UTC"
>
> == Affected services ==
>
> lists.ovirt.org
> wiki.ovirt.org
> ovirt.org/.*
> ovirtbot
> Gerritt backup
> Jenkins backup
> [[What else?]]
>
> == Responses to take ==
>
> * Get new hosting solution in place.
> * Double current disk space before new hosting move, to give us room
> to
> breath.
> * Work up a response place that is posted in the IRC topic or
> somewhere
> good so people know how to contact all of the Infra team when
> something
> is happening.
> * New service need: monitoring server
Fix the reaper script to properly purge
the nightly files.
Thanks
Robert
>
> --
> Karsten 'quaid' Wade, Sr. Analyst - Community Growth
> http://TheOpenSourceWay.org .^\ http://community.redhat.com
> @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41
>
>
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
Thanks
Robert Middleswarth
@rmiddle (twitter/Freenode IRC)
@RobertM (OFTC IRC)
4254
days inactive
4256
days old
2 comments
3 participants
participants (3)
-
Karsten 'quaid' Wade -
Mike Burns -
Robert Middleswarth