[ovirt-devel] PHX outage report 10.03.2017

[Evgheni Dereveanchin]

Hi everyone,

Tonight we experienced a hardware fault on one of our PHX storage servers.
The faulty server was used to provide storage for multiple production VMs.
Since automatic failover did not happen they became unavailable.
The outage lasted between 09.03.2017 20:36 UTC and 10.03.2017 09:15 UTC

Unavailable services included all of oVirt's CI infrastructure, mailing
lists
and package repositories. Services in other datecenters such as
gerrit.ovirt.org and www.ovirt.org were not affected.

We brought storage back up and this allowed for VMs to be restarted.
If you see tests that failed or didn't run during this period please
re-trigger them.
If there are still persisting issues please report them to the tracker
ticket
that has a more detailed root cause analysis:
https://ovirt-jira.atlassian.net/browse/OVIRT-1244

Sorry for the inconvenience caused. We are working on improving reliability
of the environment to avoid similar incidents from happening in the future.

-- 
Regards,
Evgheni Dereveanchin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170310/66bf2917/attachment.html>