Gerrit down

[Itamar Heim]

On 05/31/2013 12:16 PM, Dave Neary wrote:
> Hi guys,
>
> We really need to get a handle on this, having conversations like this
> is kind of ridiculous. First, by making it clear who has access to the
> various resources we depend on, second by setting up some kind of
> "weekend & TZ watch" system so that if something happens on a Friday or
> Saturday, or in the middle of our admins' night, that we know how to get
> services running again, and finally, by having some kind of monitoring
> solution up and running at last.
>
> I know that some of these things are on our agenda:
> https://fedorahosted.org/ovirt/ticket/48
> https://fedorahosted.org/ovirt/ticket/20

just for the immediate parts of this;
gerrit has monitoring - a jenkins job sending an email when it is down 
to infra mailing list.
basic "get it to work if hung/down" operator job is available in jenkins 
for a much wider distribution of users who can restart it.




>
> But this really feels like it should be a top priority for us.
>
>
> IRC extract, 10:20am CEST Friday May 31 2013:
>
> <kanagaraj> gerrit.ovirt.org is down. anyone facing this issue?
> <tdosek> kanagaraj: same here. it's down. :(
> <fabiand> yep, it's down ...
>   awh ...
> <fabiand> dneary, gerrit is down ... have you got control over it?
> <dneary> fabiand, I don't
>   dcaro, Ping?
>   fabiand, dcaro is the man
> <dcaro> dneary: me neither :S
> <dneary> Ah
>   Isn't Gerrit on Alterway servers now?
> <fabiand> ewoud, do you maybe know?
> <ewoud> dneary: not yet
> <dcaro> dneary: let me see if I can access...
> <mskrivanek> dcaro: use the force!
> <ewoud> itamar: can you check gerrit?
> <dcaro> dneary: it's in amazon :S
> <dcaro> no, no access :,(
> <dneary> The wiki page
> http://www.ovirt.org/Infrastructure_team_administrators lists itamar and
> Rydekull
> <dcaro> none of them are around...
> <lhornyak> gerrit is down?
> <lhornyak> ping is ok though
>   anyone working on the gerrit fubar? :)
> * dcaro looks the other way
>   dcaro starts whistling
>
>
> Cheers,
> Dave.
>
>