Followup on today lists.ovirt.org http outage
--=-lMFwBS4VzHn7ryn66xGn Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After scratching my head for a while ( since everything was running fine, despites regular Out of memory on the server ), it turned out to be a user trying to get the iso with a download accelerator. I first added more server, but without luck. So as I am more of the kind "shoot first, ask later", I did kill the connexion with iptables, then limit it with iptables ( but with some side effect ), then installed mod_limitipconn to limit to 10 tcp connexion per IP.=20 in short : - yum install mod_limitipconn - add=20 <IfModule mod_limitipconn.c> MaxConnPerIP 10 </IfModule> to /etc/httpd/conf.d/resources.ovirt.org.conf I guess we should add this in some puppet module somewhere ? --=20 Michael Scherer Open Source and Standards, Sysadmin --=-lMFwBS4VzHn7ryn66xGn Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAABAgAGBQJToEbiAAoJEE89Wa+PrSK9ynwQAIHWN8p/J+KMeIcHAGciasPC jOj1uBJVHUsRScgZ5SM705MIUZ+tJCuV28/3Wowky4LExhHi36IFIo4toJ7urGPb PekQ36JNsMF+8yUl/+pWsjZpBLDt4ZcUALpqqInp04n1qEXwt1HhVjhI+BEjTnV8 eAboFC5x3o7PxpSISY6nwA3sYiG4UDWp4XSHmM6oJIrr/uAjcAJm7lpIcsVC1HWV VujzBR3fLp9A9BzgHtlN8AaU8kLjCC0UzLXbY0Q8MPIzHIdyCMKNYKqQtRc+Es5/ mwXSjpV30323iidH3vGSXuJm8WfjT4He7nmY/XdMXB6Q/3ZAwS4ktJIb28Rwmc6v 9xBw7sE6eCZHyiLYJsoqUNr5z49uBZlVLYcZDntCCP+TjityFZeYV31WzTV1MutI qb1fiu0PtTqfOGoT0eBVmBBrFhHsmtgga/vjs7LyF7ZW5FWWKGUlYXVxQl4wenC1 eAIlk1e/CzWI/ybh0QnsZyEd+Li5uhcg0Na12T+6OS9+7o/zWsxtiaB91FG2DVtf 7x7ejv7jmE6IanWKsNTa+Lo0clUNm6ufMDOLIlrPP39kkbkVY/Cqh/9n6in4XezN 8NlrgjOjZHohvLEwtW+Qgo4XAjvfb2QEDOkI9RviNMEoT5ZmWVjLr4srrBcxohqV 6v/UMr3AyDfI3xppIsEs =OWTc -----END PGP SIGNATURE----- --=-lMFwBS4VzHn7ryn66xGn--
On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote:
Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After scratching my head for a while ( since everything was running fine, despites regular Out of memory on the server ), it turned out to be a user trying to get the iso with a download accelerator. I first added more server, but without luck.
So as I am more of the kind "shoot first, ask later", I did kill the connexion with iptables, then limit it with iptables ( but with some side effect ), then installed mod_limitipconn to limit to 10 tcp connexion per IP.
I'm all in favor of this. Maybe we should mention we have mirrors with MUCH more bandwith in our README.
in short : - yum install mod_limitipconn - add <IfModule mod_limitipconn.c> MaxConnPerIP 10 </IfModule> to /etc/httpd/conf.d/resources.ovirt.org.conf
I guess we should add this in some puppet module somewhere ?
We should, but the whole apache config isn't puppetized yet. I've been slacking on that because we want to move away from that server, but maybe we should bite the bullet and do it on the current server.
--=-eP3k5LV4Tp5Zr27AX3vY Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Le mardi 17 juin 2014 =C3=A0 15:55 +0200, Ewoud Kohl van Wijngaarden a =C3=A9crit :
On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote:
Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After scratching my head for a while ( since everything was running fine, despites regular Out of memory on the server ), it turned out to be a user trying to get the iso with a download accelerator. I first added more server, but without luck. =20 So as I am more of the kind "shoot first, ask later", I did kill the connexion with iptables, then limit it with iptables ( but with some side effect ), then installed mod_limitipconn to limit to 10 tcp connexion per IP.=20 =20 I'm all in favor of this. Maybe we should mention we have mirrors with MUCH more bandwith in our README.
Or maybe we do not need to tell that to people and use a redirector ? ( like mirrorbrain, etc ). Even if a solution that requires no maintainance is maybe a better solution for now.
in short : - yum install mod_limitipconn - add=20 <IfModule mod_limitipconn.c> MaxConnPerIP 10 </IfModule> to /etc/httpd/conf.d/resources.ovirt.org.conf =20 I guess we should add this in some puppet module somewhere ? =20 We should, but the whole apache config isn't puppetized yet. I've been slacking on that because we want to move away from that server, but maybe we should bite the bullet and do it on the current server.
Yep, and I think it would be easier to move away from the server if it is in puppet :) --=20 Michael Scherer Open Source and Standards, Sysadmin --=-eP3k5LV4Tp5Zr27AX3vY Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAABAgAGBQJToE0CAAoJEE89Wa+PrSK9C34P/0MxCNC5y/FEffxjMQcIrbaY pxOZv4PYYHvPds6ZZHIaAMCAUY08g0uN8zfbz3fhIW18SmXomcaKm3C5dUyPLRjR 1TsZL6jHN+cB0uVlBhrJ1J7BHxetd7HMCBh1ggPEs7+jOUE70zDGTJKhyinfZRnC HF89eXNfwBni9Dra1fMlj0QoktBttzbo0KWUgvVQULzWpabusV9QgEA9+1uGMNcU EaKaC51wdcloX+Oru8oY/wNd50dwwqOGuRUWrVQ0zO4HU8ykx59PYdt597rA31VZ FfXy2eGxsxgQJGV8fcz6g0abk3CI7wEn+q/3UJNGv5MD4lC/vR1owYgsNH/tQ/3L gKQxdsBKZuiS8IhCym6um4pug/NIgfdUbwMQTRZ83qCoMFYGC8GxQd+rutmk+UMe Bz6V83UrlEcjrCJv+iFbRuEyHxkxpfMFDpReKGfvQkQrzhiHUhwrsg79PXwp9MXj doS8mXznLYJlsBa6jnti1DUncMln8Vec9KR76MztWq3R10yYhOT7g6BWVu2d930E i8pCGcxII0YtvzVZmfoiVGqeAugGGTM27e0q5BByMtSdOpqFhmwf+lgL2vS91xxf CHE1VmbZ4kAjJBCYlKHy+onx74REVBvezUb8wYMN4Pv0/WH4M68DJ6/ETXlJuhYF dxtYtF9fC+52c14Xovhx =bhkS -----END PGP SIGNATURE----- --=-eP3k5LV4Tp5Zr27AX3vY--
Il 17/06/2014 16:13, Michael Scherer ha scritto:
Le mardi 17 juin 2014 à 15:55 +0200, Ewoud Kohl van Wijngaarden a écrit :
On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote:
Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After scratching my head for a while ( since everything was running fine, despites regular Out of memory on the server ), it turned out to be a user trying to get the iso with a download accelerator. I first added more server, but without luck.
So as I am more of the kind "shoot first, ask later", I did kill the connexion with iptables, then limit it with iptables ( but with some side effect ), then installed mod_limitipconn to limit to 10 tcp connexion per IP.
I'm all in favor of this. Maybe we should mention we have mirrors with MUCH more bandwith in our README.
Or maybe we do not need to tell that to people and use a redirector ? ( like mirrorbrain, etc ).
This won't allow to download packages until mirrors are synced. Now yum repo files have mirrorlist pointing to mirrors and baseurl pointing to ovirt.org. Introducing automated redirection won't allow this anymore.
Even if a solution that requires no maintainance is maybe a better solution for now.
in short : - yum install mod_limitipconn - add <IfModule mod_limitipconn.c> MaxConnPerIP 10 </IfModule> to /etc/httpd/conf.d/resources.ovirt.org.conf
I guess we should add this in some puppet module somewhere ?
We should, but the whole apache config isn't puppetized yet. I've been slacking on that because we want to move away from that server, but maybe we should bite the bullet and do it on the current server.
Yep, and I think it would be easier to move away from the server if it is in puppet :)
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
--=-z5XVsGi+AtxU3+t49xaR Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Le mardi 17 juin 2014 =C3=A0 16:17 +0200, Sandro Bonazzola a =C3=A9crit :
Il 17/06/2014 16:13, Michael Scherer ha scritto:
Le mardi 17 juin 2014 =C3=A0 15:55 +0200, Ewoud Kohl van Wijngaarden a =C3=A9crit :
On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote:
Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. Aft= er scratching my head for a while ( since everything was running fine, despites regular Out of memory on the server ), it turned out to be a user trying to get the iso with a download accelerator. I first added more server, but without luck.
So as I am more of the kind "shoot first, ask later", I did kill the connexion with iptables, then limit it with iptables ( but with some side effect ), then installed mod_limitipconn to limit to 10 tcp connexion per IP.=20
I'm all in favor of this. Maybe we should mention we have mirrors with MUCH more bandwith in our README. =20 Or maybe we do not need to tell that to people and use a redirector ? ( like mirrorbrain, etc ). =20 This won't allow to download packages until mirrors are synced. Now yum repo files have mirrorlist pointing to mirrors and baseurl pointi= ng to ovirt.org. Introducing automated redirection won't allow this anymore.
It work quite fine for fedora. But we indeed to make sure this doesn't break stuff for people, especially older setup. I guess we also want to still get some download stats. But we do not have much issues or risk with rpms, more on iso download, as they take more bandwidth and for longer time, and I think there is more risk of having a download accelerator for that. So what about just make a redirect for iso (if possible, something smart enough to not redirect if the mirror is not synced), and keep rpms as is ? --=20 Michael Scherer Open Source and Standards, Sysadmin --=-z5XVsGi+AtxU3+t49xaR Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAABAgAGBQJToFcEAAoJEE89Wa+PrSK9m64P/AmBi/NIQxbqbn/2NjtOrIqi ocMH0g/1CVkloUKxujFg1y+cbncf+kH7EHxy5Nb5umlPWxm1+3g3egLyVQc+OCAn oOAcwe7njrOi4AzP/kwedARDv77ttCTLc+B14PVtl/5d1tvWGWvsVeJVBzJ62NdC MtOpQpmjuuW1mFRtk6cfsgF90EpcJy6vX0zMdCVc0eXtt4uy2/lPsg3GJocrktTf sYEqWViaUsbvUl60NBJe1B9hNnpL4xvkN2kYtTtBMl0HYZY1vrgcdy2zIXo9GZFU dPR5BWTYkfIyIr+uSkGlhsO2acCM8x/vcAnIQVMK6qhzucsWM69UfAKkITFofX2z 1u+v+5at/zZOHYIBELPvt2cdu4zX/8Df32neQUPcFufq5m3BM5fZBFBzr/N9xfTu EPucKC1/pUlGB5McbNsJTyyNFwkC448b4Rd+bjEJ+t1yiQivfxXSM6fPHGvi/6Cw YdQk8no8DXXrzarbD/uEFQO2ljjgz/ELyLZhRP3JCaJB+G9vwlEll6OTsqJQZx1G fKIRt2qVVeM+wlfcdDpCW12K+m1chq4F7QebV+3WXQVlcGBIoNo5uwAB2c/VWLYp +sMlzsydvzESZ+ZC/rTHaukSmfXhAUO+7dvf2JZvBMjq4X6OOizof65uVn3YKpmP 9t07+CK5SWLcYeYhTaNa =pLch -----END PGP SIGNATURE----- --=-z5XVsGi+AtxU3+t49xaR--
Il 17/06/2014 16:56, Michael Scherer ha scritto:
Le mardi 17 juin 2014 à 16:17 +0200, Sandro Bonazzola a écrit :
Il 17/06/2014 16:13, Michael Scherer ha scritto:
Le mardi 17 juin 2014 à 15:55 +0200, Ewoud Kohl van Wijngaarden a écrit :
On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote:
Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After scratching my head for a while ( since everything was running fine, despites regular Out of memory on the server ), it turned out to be a user trying to get the iso with a download accelerator. I first added more server, but without luck.
So as I am more of the kind "shoot first, ask later", I did kill the connexion with iptables, then limit it with iptables ( but with some side effect ), then installed mod_limitipconn to limit to 10 tcp connexion per IP.
I'm all in favor of this. Maybe we should mention we have mirrors with MUCH more bandwith in our README.
Or maybe we do not need to tell that to people and use a redirector ? ( like mirrorbrain, etc ).
This won't allow to download packages until mirrors are synced. Now yum repo files have mirrorlist pointing to mirrors and baseurl pointing to ovirt.org. Introducing automated redirection won't allow this anymore.
It work quite fine for fedora. But we indeed to make sure this doesn't break stuff for people, especially older setup. I guess we also want to still get some download stats.
But we do not have much issues or risk with rpms, more on iso download, as they take more bandwidth and for longer time, and I think there is more risk of having a download accelerator for that.
So what about just make a redirect for iso (if possible, something smart enough to not redirect if the mirror is not synced), and keep rpms as is ?
If you have something smart enough to not redirect if the mirror is not synced, you can redirect rpms too. :-)
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
----- Original Message -----
From: "Michael Scherer" <mscherer@redhat.com> To: infra@ovirt.org Sent: Tuesday, June 17, 2014 5:56:04 PM Subject: Re: Followup on today lists.ovirt.org http outage
Le mardi 17 juin 2014 à 16:17 +0200, Sandro Bonazzola a écrit :
Il 17/06/2014 16:13, Michael Scherer ha scritto:
Le mardi 17 juin 2014 à 15:55 +0200, Ewoud Kohl van Wijngaarden a écrit :
On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote:
Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After scratching my head for a while ( since everything was running fine, despites regular Out of memory on the server ), it turned out to be a user trying to get the iso with a download accelerator. I first added more server, but without luck.
So as I am more of the kind "shoot first, ask later", I did kill the connexion with iptables, then limit it with iptables ( but with some side effect ), then installed mod_limitipconn to limit to 10 tcp connexion per IP.
I'm all in favor of this. Maybe we should mention we have mirrors with MUCH more bandwith in our README.
Or maybe we do not need to tell that to people and use a redirector ? ( like mirrorbrain, etc ).
This won't allow to download packages until mirrors are synced. Now yum repo files have mirrorlist pointing to mirrors and baseurl pointing to ovirt.org. Introducing automated redirection won't allow this anymore.
It work quite fine for fedora. But we indeed to make sure this doesn't break stuff for people, especially older setup. I guess we also want to still get some download stats.
But we do not have much issues or risk with rpms, more on iso download, as they take more bandwidth and for longer time, and I think there is more risk of having a download accelerator for that.
So what about just make a redirect for iso (if possible, something smart enough to not redirect if the mirror is not synced), and keep rpms as is ?
i wonder if we can setup some sort of a KB or wiki to include these outages info and resolution, for future reference, what do you think? e.
-- Michael Scherer Open Source and Standards, Sysadmin
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Il 17/06/2014 15:47, Michael Scherer ha scritto:
Hi,
Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After scratching my head for a while ( since everything was running fine, despites regular Out of memory on the server ), it turned out to be a user trying to get the iso with a download accelerator. I first added more server, but without luck.
So as I am more of the kind "shoot first, ask later", I did kill the connexion with iptables, then limit it with iptables ( but with some side effect ), then installed mod_limitipconn to limit to 10 tcp connexion per IP.
in short : - yum install mod_limitipconn - add <IfModule mod_limitipconn.c> MaxConnPerIP 10 </IfModule> to /etc/httpd/conf.d/resources.ovirt.org.conf
I guess we should add this in some puppet module somewhere ?
Maybe also limit bandwidth per IP?
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
--=-3ju7/Z5KhyO8SK89vDk9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Le mardi 17 juin 2014 =C3=A0 16:03 +0200, Sandro Bonazzola a =C3=A9crit :
Il 17/06/2014 15:47, Michael Scherer ha scritto:
Hi, =20 Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After scratching my head for a while ( since everything was running fine, despites regular Out of memory on the server ), it turned out to be a user trying to get the iso with a download accelerator. I first added more server, but without luck. =20 So as I am more of the kind "shoot first, ask later", I did kill the connexion with iptables, then limit it with iptables ( but with some side effect ), then installed mod_limitipconn to limit to 10 tcp connexion per IP.=20 =20 in short : - yum install mod_limitipconn - add=20 <IfModule mod_limitipconn.c> MaxConnPerIP 10 </IfModule> to /etc/httpd/conf.d/resources.ovirt.org.conf =20 I guess we should add this in some puppet module somewhere ? =20 Maybe also limit bandwidth per IP?
For now, the issue was more the ressources used serverside ( ie, 1 httpd slot per request ). Limit per IP could make sense, I didn't look at a supported apache module. Seems there is mod_bw for that. --=20 Michael Scherer Open Source and Standards, Sysadmin --=-3ju7/Z5KhyO8SK89vDk9 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAABAgAGBQJToFU4AAoJEE89Wa+PrSK9jyEP/2alxR0yC4F9Nl00PDdn+pAi f3RrxvaN1h6UAit4qsszfOhVmvcXs2ILbdfMO3Fuq04QiVCcptZZpo0YlmLWEgIW ZjI3lT9ejH6Z5GBTSvtftJygLicjnLcpofS/eHM/VlPhoYBUusBjuILWPKnbe5pi 5j/wJKHYteWbO1N6Gsd3AEGr1N/elaIoeN6x+ZGzSKRgXwz8WSBghkl1gdq1E2bi eQH/55z9ospIwfcJNUC6pwpF2+eOORyMSm68bHjTuznmu+M16EF0s/x2/l+8qHRw mqQRZtE/KPehw/ro+xXcoE7Ga5Ti9/Vn6S7+BgSBFpmDTqfVJOAyW7eNc2jEatpH 7boqajw7hYw61v/42k+huaiTW5yrhpIG5PiwGPEEb3Ph4FzArCMStqUXiO6xn65S lLTzZnnJX6lRVisu5Zrn4P1PpJoG5E2yH1oR/s7TUbjRRE4DkOhgUcXk3m4iOEck 82xE+JYxNOLXM1FuvqeoAosTkXfIqPEd2GjvC0b2AbFKNDmw4qcJvWmUBdk4tsrG 0TQ2rF71AZ2v9WsuAkD/Gd8VM5+wDN0rhNyAlbHXeiMJskO5e6RLM8yrgkoHXiXH +DKAyk7X9R5C/f5U9EhCzHV599vuXbQcJA/rqERAVvsaAilp/02Nqvp0IRhumIaN LeyC2+1gFva8aPFunSf5 =XCj1 -----END PGP SIGNATURE----- --=-3ju7/Z5KhyO8SK89vDk9--
participants (4)
-
Ewoud Kohl van Wijngaarden -
Eyal Edri -
Michael Scherer -
Sandro Bonazzola