<div dir="ltr">Hi everyone,<div><br></div><div>Tonight we experienced a hardware fault on one of our PHX storage servers.<br></div><div>The faulty server was used to provide storage for multiple production VMs. </div><div>Since automatic failover did not happen they became unavailable.</div><div>The outage lasted between 09.03.2017 20:36 UTC and 10.03.2017 09:15 UTC</div><div><br></div><div>Unavailable services included all of oVirt's CI infrastructure, mailing lists</div><div>and package repositories. Services in other datecenters such as </div><div><a href="http://gerrit.ovirt.org">gerrit.ovirt.org</a> and <a href="http://www.ovirt.org">www.ovirt.org</a> were not affected.</div><div><br></div><div>We brought storage back up and this allowed for VMs to be restarted.</div><div>If you see tests that failed or didn't run during this period please re-trigger them. </div><div>If there are still persisting issues please report them to the tracker ticket </div><div>that has a more detailed root cause analysis:</div><div><a href="https://ovirt-jira.atlassian.net/browse/OVIRT-1244">https://ovirt-jira.atlassian.net/browse/OVIRT-1244</a><br></div><div><br></div><div>Sorry for the inconvenience caused. We are working on improving reliability</div><div>of the environment to avoid similar incidents from happening in the future.</div><div><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div style="font-size:small">Regards,</div><div style="font-size:small">Evgheni Dereveanchin</div></div></div>
</div></div>