PHX outage report 10.03.2017

Hi everyone, Tonight we experienced a hardware fault on one of our PHX storage servers. The faulty server was used to provide storage for multiple production VMs. Since automatic failover did not happen they became unavailable. The outage lasted between 09.03.2017 20:36 UTC and 10.03.2017 09:15 UTC Unavailable services included all of oVirt's CI infrastructure, mailing lists and package repositories. Services in other datecenters such as gerrit.ovirt.org and www.ovirt.org were not affected. We brought storage back up and this allowed for VMs to be restarted. If you see tests that failed or didn't run during this period please re-trigger them. If there are still persisting issues please report them to the tracker ticket that has a more detailed root cause analysis: https://ovirt-jira.atlassian.net/browse/OVIRT-1244 Sorry for the inconvenience caused. We are working on improving reliability of the environment to avoid similar incidents from happening in the future. -- Regards, Evgheni Dereveanchin