Content-Type: text/plain; charset=us-ascii
* Last week we finally retired all the rackspace hosts, there's a backup
available for a few weeks in case we need to restore anything.
* Major infra outage:
Starting on monday evening we had a series of issues that made the infra
unstable throughout all monday evening and tuesday.
The causes were an unexpected cluster degradation on the storage servers =
changing the netmask, and failures derived from the unfinished upgrade to=
on the ovirt setup.
All the servers are now back to stable, and we also took the opportunity =
upgrade all the hosts to fc20 and 3.5, so now we have a more robust setup
than we had before the outage.
We have now nested available for any VM on the jenkins or production
Also the ip range available for the jenkins slaves has been duplicated,
passing from /26 to /25
* A lot of jobs failed due to the outage, we are still stabilizing the jobs,
any new failure you see please ping me or send an email to infra, but if =
was during the outage there's no need to communicate it, we already know.
* Proxy setup in phoenix lab almost finished, doing the final tests
* There's a new team member apprentice, Vishnu, that has already taken one =
the tickets :), Welcome!!
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
RHT Global #: 82-62605
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
-----END PGP SIGNATURE-----