[SOLVED] Temporary Outage - ovirt.org Website
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --UjerjdpPI1W3Jwd0dWEnDKWKtGMXxXMg7 Content-Type: multipart/mixed; boundary="5rcCa7NeC9kJsmU0XEABtc3HH2nIqASqS" From: Mikey Ariel <mariel@redhat.com> To: infra@ovirt.org Message-ID: <56DD7E86.8070202@redhat.com> Subject: [SOLVED] Temporary Outage - ovirt.org Website --5rcCa7NeC9kJsmU0XEABtc3HH2nIqASqS Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi folks, It seems that our logs on OpenShift filled up and the ovirt.org website was unavailable for a few hours this morning. The issue was resolved and the site is back online. Cheers, Mikey --=20 Mikey Ariel Community Lead, oVirt www.ovirt.org "To be is to do" (Socrates) "To do is to be" (Jean-Paul Sartre) "Do be do be do" (Frank Sinatra) Mobile: +420-702-131-141 IRC: mariel / thatdocslady Twitter: @ThatDocsLady --5rcCa7NeC9kJsmU0XEABtc3HH2nIqASqS-- --UjerjdpPI1W3Jwd0dWEnDKWKtGMXxXMg7 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJW3X6GAAoJEHYPPTOszxHogaoH/iLn4bXy0uQDjusJidUwI1/l pVkGfPrgbgnwCDnVqNasipbRzCPHZ/xVEQV63xG/0KlFrm1yeiZ2HTrufkRYdR2d s6tRPcBCzGvPDygeM/pQySo/ozXMkccE4l1+U2vUjCV5N1K11B5E42AZ2SnVP0t7 CfRZGgxyYhjWcKJ6bWhn72UdiGmUc9x9LSsUDgAkez74Pk7++TUeQHt8Bv4tzOZq j9mpGis5e2uAMdwY6W+Xd4l0qSTrp3ammRI3QzApxN0ZdKTirgkdgtxaEXjtpq7N PX++sTZr6qm8I1xqnCDsNbxhlTZZe64l+FkzCd6DeXdI11vTyLyyIZ70WnqGeCY= =1ciD -----END PGP SIGNATURE----- --UjerjdpPI1W3Jwd0dWEnDKWKtGMXxXMg7--
--=-P+jMSFUV/o8IeH24nhkp Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Le lundi 07 mars 2016 =C3=A0 14:13 +0100, Mikey Ariel a =C3=A9crit :
Hi folks, =20 It seems that our logs on OpenShift filled up and the ovirt.org website was unavailable for a few hours this morning. =20 The issue was resolved and the site is back online.
So to complete what Mikey said: - we have increased the partition from 1G to 3G (this was done automatically with previous openshift version, so insert my usual rant about changes in *aaS), so we would have more room next time. - a RCA (root cause analysis) show that we got 250 000 requests today from 1 single IP at OVH, around 100 000 from bing bot reindexing the website (on 4 or 5 ip ), and 30 000 requests from a ip of bezeqint.net, with a nagios check_http signature.=20 While we can't do much for the first or the 2nd (besides blocking), I would suggest that the 3rd ip owner decide to reduce the frequency of check from 1 every 2 seconds to 1 every 5 minutes. --=20 Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS --=-P+jMSFUV/o8IeH24nhkp Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAABAgAGBQJW3YFzAAoJEE89Wa+PrSK92FcP/RU8Zdkkbi0qIvU9FodGP/dR 3yKtDHIp5Djxan1SWtppT+HlAPtp6q+YRA2rDovHKvfsrOjyzR9Ui4qd46S6yIv9 V2XV3Hw9vO50isV3vZEmEQ/8T/+vm/B/aZ1pi9c3ZvK7UlZdNw6qZzmYJLqxfpHh 8KbCa6iaCPe3hsrv9vZnCyu3bQo6zHNakb9DJx0KaScnDZcuqFDRa5eTk8JIBLh5 e+CkMJ0wrkLj8wg0o04Ybfc1/ptGE+MI0XXV7HNGIXULE7vRi8Q2GS8ZDhuRd0qf GLSncZ966owKkaSwRkftzxFbdjBLPHSn7tOXIcuJtacE81KDrs2JAG9ogVPwGM7Q XLuLIj1Ac6H/Zn6hPLSMXTf1xI3rpoojZ+OMTuXpFeoQJ2Jpt85gKntJGDSczitA ZuneSBmgKGd0blSj/Wc3JW5hxCownPBJJVrr4Rkf39izCT+HlsWkp4Q9bU8lr+SZ kXlnhEU42wQdy3aHrBrMp5RBUmsWNjqFiXB8LF0QX8T9OXK6RI3spGyeSIPHRtA6 ZfR5/SMYJS0U3xwTF4u3Y+lsFoCti16Hp/7Frr5e/Ro0kq1CJ9sl2iU2fswori6u nlJtZ4miVYVZLsTH8yhehm4HE8rK8q185UteK+WQBXwxSzFRawNZPVTOt3bC8Wrs jP+wKbaVubfNcwDA4Gxr =UhSQ -----END PGP SIGNATURE----- --=-P+jMSFUV/o8IeH24nhkp--
While we can't do much for the first or the 2nd (besides blocking), I would suggest that the 3rd ip owner decide to reduce the frequency of check from 1 every 2 seconds to 1 every 5 minutes.
Sure, that was us, I just removed the whole thing, I guess you have your own and better monitoring now. -- Barak Korren bkorren@redhat.com RHEV-CI Team
--=-gtIJFMhA5RKT2NfX6Qzy Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Le lundi 07 mars 2016 =C3=A0 15:50 +0200, Barak Korren a =C3=A9crit :
While we can't do much for the first or the 2nd (besides blocking), I would suggest that the 3rd ip owner decide to reduce the frequency of check from 1 every 2 seconds to 1 every 5 minutes.
=20 Sure, that was us, I just removed the whole thing, I guess you have your own and better monitoring now.
Nope, we don't. nagios is fine, but well, every second is excessive :/ --=20 Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS --=-gtIJFMhA5RKT2NfX6Qzy Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAABAgAGBQJW3Yd9AAoJEE89Wa+PrSK91OAP/31c3MqKucXvQHByhmeYDLUb Y89YUVfJmoqc9Ys8FqUZwT+0bDYFOOjTW9fySxDDEgjlrCIOXrFc847fl7OVtGS0 wIQ6ssY97pyf2DY4RxJizwoa8wZugDHlsO3meCbBmJPDg8jI59jwY2+RbvPpxuIC VQuCcrZdkFGbDaOx2Is1ZFb/GTj8eOM5v42b2U0P6/eV6+kq1KCP7njCsfzcEzYM SFPwgqvMS5/M4mqmr3fQAcvcZfWV1zd3j0nIi220wvbJe3txogTEHwSbuxHBLTCX maL7d3IVx1XIADcFR2JGqQZdtB8x/OgtVOYJquY9lgG8e7XEIoyoRb+IyGLiKfth x8HVEw8oFTxiE6y1K79PX36I74rq53pvWIswAHG1X3ELOqX7Rvw/f6LPNzvxGpTr fD9WYM31FE9ubNJ3q9d/oakdFooQO+m3BDMgy0DgHizE4HZpoRCTL+hWBB0x05wc mqMkslBQEAtIEutcwfXZLt8Nt6UFqhvwKxufx/A9sxZChmmyritggtpzqeJGsrdo fJl12oRiLj2mY4nWqe5tMO4g81UEityqM5ZF4ynNPgzK3p8ppgjpVOqoKf5UB151 mrpSMstyLNWIvjX8dSqsDrjYIY5KIQdn0hy0ll7iECvmEgBjFNUNd3LcqCmv6Rtg CVo3/F3p0KII3jnAX3gm =ynG3 -----END PGP SIGNATURE----- --=-gtIJFMhA5RKT2NfX6Qzy--
participants (3)
-
Barak Korren -
Michael Scherer -
Mikey Ariel