Re: [ovirt-devel] PHX storage outage - all jenkins slaves affected

--xHFwDpU9dbj6ez1V Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 02/08 20:28, David Caro wrote: panned age) an use have oest) nd and So the source of the issue has been sorted out, there was some uncoordinated effort that ended up changing the LACP settings in the switches for all the hosts, what caused a global network outage (all the hosts were affected) and that in turn caused the clustering to freeze as none of the nodes was able = to contact the network both went down. Then, once the network came up, the master of the cluster tried to remount = the drdb storage but was unable to due to some process keeping it busy, and did= not fully start up. That is a scenario that we did not test (we tested one node failure, not bo= th) so will have to investigate that failure case and find a solution for the clustering. We are also talking with the hosting to properly sync with us on that type = of interventions so this will not happen again. Thanks for your patience --=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --xHFwDpU9dbj6ez1V Content-Type: application/pgp-signature; name="signature.asc" --xHFwDpU9dbj6ez1V--