Tonight we experienced a hardware fault on one of our PHX storage servers.

The faulty server was used to provide storage for multiple production VMs. 

Since automatic failover did not happen they became unavailable.

The outage lasted between 09.03.2017 20:36 UTC and 10.03.2017 09:15 UTC

Unavailable services included all of oVirt's CI infrastructure, mailing lists

and package repositories. Services in other datecenters such as 

gerrit.ovirt.org and www.ovirt.org were not affected.

We brought storage back up and this allowed for VMs to be restarted.

If you see tests that failed or didn't run during this period please re-trigger them. 

If there are still persisting issues please report them to the tracker ticket 

that has a more detailed root cause analysis:

Sorry for the inconvenience caused. We are working on improving reliability

of the environment to avoid similar incidents from happening in the future.

--