4:47 p.m.
--=-lMFwBS4VzHn7ryn66xGn
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Hi,
Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After
scratching my head for a while ( since everything was running fine,
despites regular Out of memory on the server ), it turned out to be a
user trying to get the iso with a download accelerator. I first added
more server, but without luck.
So as I am more of the kind "shoot first, ask later", I did kill the
connexion with iptables, then limit it with iptables ( but with some
side effect ), then installed mod_limitipconn to limit to 10 tcp
connexion per IP.=20
in short :
- yum install mod_limitipconn
- add=20
<IfModule mod_limitipconn.c>
MaxConnPerIP 10
</IfModule>
to /etc/httpd/conf.d/resources.ovirt.org.conf
I guess we should add this in some puppet module somewhere ?
--=20
Michael Scherer
Open Source and Standards, Sysadmin
--=-lMFwBS4VzHn7ryn66xGn
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part
Content-Transfer-Encoding: 7bit
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAABAgAGBQJToEbiAAoJEE89Wa+PrSK9ynwQAIHWN8p/J+KMeIcHAGciasPC
jOj1uBJVHUsRScgZ5SM705MIUZ+tJCuV28/3Wowky4LExhHi36IFIo4toJ7urGPb
PekQ36JNsMF+8yUl/+pWsjZpBLDt4ZcUALpqqInp04n1qEXwt1HhVjhI+BEjTnV8
eAboFC5x3o7PxpSISY6nwA3sYiG4UDWp4XSHmM6oJIrr/uAjcAJm7lpIcsVC1HWV
VujzBR3fLp9A9BzgHtlN8AaU8kLjCC0UzLXbY0Q8MPIzHIdyCMKNYKqQtRc+Es5/
mwXSjpV30323iidH3vGSXuJm8WfjT4He7nmY/XdMXB6Q/3ZAwS4ktJIb28Rwmc6v
9xBw7sE6eCZHyiLYJsoqUNr5z49uBZlVLYcZDntCCP+TjityFZeYV31WzTV1MutI
qb1fiu0PtTqfOGoT0eBVmBBrFhHsmtgga/vjs7LyF7ZW5FWWKGUlYXVxQl4wenC1
eAIlk1e/CzWI/ybh0QnsZyEd+Li5uhcg0Na12T+6OS9+7o/zWsxtiaB91FG2DVtf
7x7ejv7jmE6IanWKsNTa+Lo0clUNm6ufMDOLIlrPP39kkbkVY/Cqh/9n6in4XezN
8NlrgjOjZHohvLEwtW+Qgo4XAjvfb2QEDOkI9RviNMEoT5ZmWVjLr4srrBcxohqV
6v/UMr3AyDfI3xppIsEs
=OWTc
-----END PGP SIGNATURE-----
--=-lMFwBS4VzHn7ryn66xGn--
4:55 p.m.
On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote:
Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC.
After
scratching my head for a while ( since everything was running fine,
despites regular Out of memory on the server ), it turned out to be a
user trying to get the iso with a download accelerator. I first added
more server, but without luck.
So as I am more of the kind "shoot first, ask later", I did kill the
connexion with iptables, then limit it with iptables ( but with some
side effect ), then installed mod_limitipconn to limit to 10 tcp
connexion per IP.
I'm all in favor of this. Maybe we should mention we have mirrors with
MUCH more bandwith in our README.
in short :
- yum install mod_limitipconn
- add
<IfModule mod_limitipconn.c>
MaxConnPerIP 10
</IfModule>
to /etc/httpd/conf.d/resources.ovirt.org.conf
I guess we should add this in some puppet module somewhere ?
We should, but the whole apache config isn't puppetized yet. I've been
slacking on that because we want to move away from that server, but
maybe we should bite the bullet and do it on the current server.
5:13 p.m.
--=-eP3k5LV4Tp5Zr27AX3vY
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Le mardi 17 juin 2014 =C3=A0 15:55 +0200, Ewoud Kohl van Wijngaarden a
=C3=A9crit :
On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote:
> Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After
> scratching my head for a while ( since everything was running fine,
> despites regular Out of memory on the server ), it turned out to be a
> user trying to get the iso with a download accelerator. I first added
> more server, but without luck.
>=20
> So as I am more of the kind "shoot first, ask later", I did kill the
> connexion with iptables, then limit it with iptables ( but with some
> side effect ), then installed mod_limitipconn to limit to 10 tcp
> connexion per IP.=20
=20
I'm all in favor of this. Maybe we should mention we have mirrors with
MUCH more bandwith in our README.
Or maybe we do not need to tell that to people and use a redirector ?
( like mirrorbrain, etc ).
Even if a solution that requires no maintainance is maybe a better
solution for now.
> in short :
> - yum install mod_limitipconn
> - add=20
> <IfModule mod_limitipconn.c>
> MaxConnPerIP 10
> </IfModule>
> to /etc/httpd/conf.d/resources.ovirt.org.conf
>=20
> I guess we should add this in some puppet module somewhere ?
=20
We should, but the whole apache config isn't puppetized yet. I've been
slacking on that because we want to move away from that server, but
maybe we should bite the bullet and do it on the current server.
Yep, and I think it would be easier to move away from the server if it
is in puppet :)
--=20
Michael Scherer
Open Source and Standards, Sysadmin
--=-eP3k5LV4Tp5Zr27AX3vY
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part
Content-Transfer-Encoding: 7bit
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAABAgAGBQJToE0CAAoJEE89Wa+PrSK9C34P/0MxCNC5y/FEffxjMQcIrbaY
pxOZv4PYYHvPds6ZZHIaAMCAUY08g0uN8zfbz3fhIW18SmXomcaKm3C5dUyPLRjR
1TsZL6jHN+cB0uVlBhrJ1J7BHxetd7HMCBh1ggPEs7+jOUE70zDGTJKhyinfZRnC
HF89eXNfwBni9Dra1fMlj0QoktBttzbo0KWUgvVQULzWpabusV9QgEA9+1uGMNcU
EaKaC51wdcloX+Oru8oY/wNd50dwwqOGuRUWrVQ0zO4HU8ykx59PYdt597rA31VZ
FfXy2eGxsxgQJGV8fcz6g0abk3CI7wEn+q/3UJNGv5MD4lC/vR1owYgsNH/tQ/3L
gKQxdsBKZuiS8IhCym6um4pug/NIgfdUbwMQTRZ83qCoMFYGC8GxQd+rutmk+UMe
Bz6V83UrlEcjrCJv+iFbRuEyHxkxpfMFDpReKGfvQkQrzhiHUhwrsg79PXwp9MXj
doS8mXznLYJlsBa6jnti1DUncMln8Vec9KR76MztWq3R10yYhOT7g6BWVu2d930E
i8pCGcxII0YtvzVZmfoiVGqeAugGGTM27e0q5BByMtSdOpqFhmwf+lgL2vS91xxf
CHE1VmbZ4kAjJBCYlKHy+onx74REVBvezUb8wYMN4Pv0/WH4M68DJ6/ETXlJuhYF
dxtYtF9fC+52c14Xovhx
=bhkS
-----END PGP SIGNATURE-----
--=-eP3k5LV4Tp5Zr27AX3vY--
5:17 p.m.
Il 17/06/2014 16:13, Michael Scherer ha scritto:
Le mardi 17 juin 2014 à 15:55 +0200, Ewoud Kohl van Wijngaarden a
écrit :
> On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote:
>> Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After
>> scratching my head for a while ( since everything was running fine,
>> despites regular Out of memory on the server ), it turned out to be a
>> user trying to get the iso with a download accelerator. I first added
>> more server, but without luck.
>>
>> So as I am more of the kind "shoot first, ask later", I did kill the
>> connexion with iptables, then limit it with iptables ( but with some
>> side effect ), then installed mod_limitipconn to limit to 10 tcp
>> connexion per IP.
>
> I'm all in favor of this. Maybe we should mention we have mirrors with
> MUCH more bandwith in our README.
Or maybe we do not need to tell that to people and use a redirector ?
( like mirrorbrain, etc ).
This won't allow to download packages until mirrors are synced.
Now yum repo files have mirrorlist pointing to mirrors and baseurl pointing to ovirt.org.
Introducing automated redirection won't allow this anymore.
Even if a solution that requires no maintainance is maybe a better
solution for now.
>> in short :
>> - yum install mod_limitipconn
>> - add
>> <IfModule mod_limitipconn.c>
>> MaxConnPerIP 10
>> </IfModule>
>> to /etc/httpd/conf.d/resources.ovirt.org.conf
>>
>> I guess we should add this in some puppet module somewhere ?
>
> We should, but the whole apache config isn't puppetized yet. I've been
> slacking on that because we want to move away from that server, but
> maybe we should bite the bullet and do it on the current server.
Yep, and I think it would be easier to move away from the server if it
is in puppet :)
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
5:56 p.m.
--=-z5XVsGi+AtxU3+t49xaR
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Le mardi 17 juin 2014 =C3=A0 16:17 +0200, Sandro Bonazzola a =C3=A9crit :
Il 17/06/2014 16:13, Michael Scherer ha scritto:
> Le mardi 17 juin 2014 =C3=A0 15:55 +0200, Ewoud Kohl van Wijngaarden a
> =C3=A9crit :
>> On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote:
>>> Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. Aft=
er
>>> scratching my head for a while ( since everything was
running fine,
>>> despites regular Out of memory on the server ), it turned out to be a
>>> user trying to get the iso with a download accelerator. I first added
>>> more server, but without luck.
>>>
>>> So as I am more of the kind "shoot first, ask later", I did kill
the
>>> connexion with iptables, then limit it with iptables ( but with some
>>> side effect ), then installed mod_limitipconn to limit to 10 tcp
>>> connexion per IP.=20
>>
>> I'm all in favor of this. Maybe we should mention we have mirrors with
>> MUCH more bandwith in our README.
>=20
> Or maybe we do not need to tell that to people and use a redirector ?
> ( like mirrorbrain, etc ).
=20
This won't allow to download packages until mirrors are synced.
Now yum repo files have mirrorlist pointing to mirrors and baseurl pointi=
ng to
ovirt.org.
Introducing automated redirection won't allow this anymore.
It work quite fine for fedora. But we indeed to make sure this doesn't
break stuff for people, especially older setup. I guess we also want to
still get some download stats.
But we do not have much issues or risk with rpms, more on iso download,
as they take more bandwidth and for longer time, and I think there is
more risk of having a download accelerator for that.
So what about just make a redirect for iso (if possible, something smart
enough to not redirect if the mirror is not synced), and keep rpms as
is ?
--=20
Michael Scherer
Open Source and Standards, Sysadmin
--=-z5XVsGi+AtxU3+t49xaR
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part
Content-Transfer-Encoding: 7bit
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAABAgAGBQJToFcEAAoJEE89Wa+PrSK9m64P/AmBi/NIQxbqbn/2NjtOrIqi
ocMH0g/1CVkloUKxujFg1y+cbncf+kH7EHxy5Nb5umlPWxm1+3g3egLyVQc+OCAn
oOAcwe7njrOi4AzP/kwedARDv77ttCTLc+B14PVtl/5d1tvWGWvsVeJVBzJ62NdC
MtOpQpmjuuW1mFRtk6cfsgF90EpcJy6vX0zMdCVc0eXtt4uy2/lPsg3GJocrktTf
sYEqWViaUsbvUl60NBJe1B9hNnpL4xvkN2kYtTtBMl0HYZY1vrgcdy2zIXo9GZFU
dPR5BWTYkfIyIr+uSkGlhsO2acCM8x/vcAnIQVMK6qhzucsWM69UfAKkITFofX2z
1u+v+5at/zZOHYIBELPvt2cdu4zX/8Df32neQUPcFufq5m3BM5fZBFBzr/N9xfTu
EPucKC1/pUlGB5McbNsJTyyNFwkC448b4Rd+bjEJ+t1yiQivfxXSM6fPHGvi/6Cw
YdQk8no8DXXrzarbD/uEFQO2ljjgz/ELyLZhRP3JCaJB+G9vwlEll6OTsqJQZx1G
fKIRt2qVVeM+wlfcdDpCW12K+m1chq4F7QebV+3WXQVlcGBIoNo5uwAB2c/VWLYp
+sMlzsydvzESZ+ZC/rTHaukSmfXhAUO+7dvf2JZvBMjq4X6OOizof65uVn3YKpmP
9t07+CK5SWLcYeYhTaNa
=pLch
-----END PGP SIGNATURE-----
--=-z5XVsGi+AtxU3+t49xaR--
5:59 p.m.
Il 17/06/2014 16:56, Michael Scherer ha scritto:
Le mardi 17 juin 2014 à 16:17 +0200, Sandro Bonazzola a écrit :
> Il 17/06/2014 16:13, Michael Scherer ha scritto:
>> Le mardi 17 juin 2014 à 15:55 +0200, Ewoud Kohl van Wijngaarden a
>> écrit :
>>> On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote:
>>>> Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After
>>>> scratching my head for a while ( since everything was running fine,
>>>> despites regular Out of memory on the server ), it turned out to be a
>>>> user trying to get the iso with a download accelerator. I first added
>>>> more server, but without luck.
>>>>
>>>> So as I am more of the kind "shoot first, ask later", I did
kill the
>>>> connexion with iptables, then limit it with iptables ( but with some
>>>> side effect ), then installed mod_limitipconn to limit to 10 tcp
>>>> connexion per IP.
>>>
>>> I'm all in favor of this. Maybe we should mention we have mirrors with
>>> MUCH more bandwith in our README.
>>
>> Or maybe we do not need to tell that to people and use a redirector ?
>> ( like mirrorbrain, etc ).
>
> This won't allow to download packages until mirrors are synced.
> Now yum repo files have mirrorlist pointing to mirrors and baseurl pointing to
ovirt.org.
> Introducing automated redirection won't allow this anymore.
It work quite fine for fedora. But we indeed to make sure this doesn't
break stuff for people, especially older setup. I guess we also want to
still get some download stats.
But we do not have much issues or risk with rpms, more on iso download,
as they take more bandwidth and for longer time, and I think there is
more risk of having a download accelerator for that.
So what about just make a redirect for iso (if possible, something smart
enough to not redirect if the mirror is not synced), and keep rpms as
is ?
If you have something smart enough to not redirect if the mirror is not synced, you can
redirect rpms too.
:-)
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
5:22 p.m.
----- Original Message -----
From: "Michael Scherer" <mscherer(a)redhat.com>
To: infra(a)ovirt.org
Sent: Tuesday, June 17, 2014 5:56:04 PM
Subject: Re: Followup on today lists.ovirt.org http outage
Le mardi 17 juin 2014 à 16:17 +0200, Sandro Bonazzola a écrit :
> Il 17/06/2014 16:13, Michael Scherer ha scritto:
> > Le mardi 17 juin 2014 à 15:55 +0200, Ewoud Kohl van Wijngaarden a
> > écrit :
> >> On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote:
> >>> Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC.
After
> >>> scratching my head for a while ( since everything was running fine,
> >>> despites regular Out of memory on the server ), it turned out to be a
> >>> user trying to get the iso with a download accelerator. I first added
> >>> more server, but without luck.
> >>>
> >>> So as I am more of the kind "shoot first, ask later", I did
kill the
> >>> connexion with iptables, then limit it with iptables ( but with some
> >>> side effect ), then installed mod_limitipconn to limit to 10 tcp
> >>> connexion per IP.
> >>
> >> I'm all in favor of this. Maybe we should mention we have mirrors with
> >> MUCH more bandwith in our README.
> >
> > Or maybe we do not need to tell that to people and use a redirector ?
> > ( like mirrorbrain, etc ).
>
> This won't allow to download packages until mirrors are synced.
> Now yum repo files have mirrorlist pointing to mirrors and baseurl pointing
> to ovirt.org.
> Introducing automated redirection won't allow this anymore.
It work quite fine for fedora. But we indeed to make sure this doesn't
break stuff for people, especially older setup. I guess we also want to
still get some download stats.
But we do not have much issues or risk with rpms, more on iso download,
as they take more bandwidth and for longer time, and I think there is
more risk of having a download accelerator for that.
So what about just make a redirect for iso (if possible, something smart
enough to not redirect if the mirror is not synced), and keep rpms as
is ?
i wonder if we can setup some sort of a KB or wiki to include these outages info
and resolution, for future reference, what do you think?
e.
--
Michael Scherer
Open Source and Standards, Sysadmin
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
5:03 p.m.
Il 17/06/2014 15:47, Michael Scherer ha scritto:
Hi,
Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After
scratching my head for a while ( since everything was running fine,
despites regular Out of memory on the server ), it turned out to be a
user trying to get the iso with a download accelerator. I first added
more server, but without luck.
So as I am more of the kind "shoot first, ask later", I did kill the
connexion with iptables, then limit it with iptables ( but with some
side effect ), then installed mod_limitipconn to limit to 10 tcp
connexion per IP.
in short :
- yum install mod_limitipconn
- add
<IfModule mod_limitipconn.c>
MaxConnPerIP 10
</IfModule>
to /etc/httpd/conf.d/resources.ovirt.org.conf
I guess we should add this in some puppet module somewhere ?
Maybe also limit bandwidth per IP?
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
5:48 p.m.
--=-3ju7/Z5KhyO8SK89vDk9
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Le mardi 17 juin 2014 =C3=A0 16:03 +0200, Sandro Bonazzola a =C3=A9crit :
Il 17/06/2014 15:47, Michael Scherer ha scritto:
> Hi,
>=20
> Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After
> scratching my head for a while ( since everything was running fine,
> despites regular Out of memory on the server ), it turned out to be a
> user trying to get the iso with a download accelerator. I first added
> more server, but without luck.
>=20
> So as I am more of the kind "shoot first, ask later", I did kill the
> connexion with iptables, then limit it with iptables ( but with some
> side effect ), then installed mod_limitipconn to limit to 10 tcp
> connexion per IP.=20
>=20
> in short :
> - yum install mod_limitipconn
> - add=20
> <IfModule mod_limitipconn.c>
> MaxConnPerIP 10
> </IfModule>
> to /etc/httpd/conf.d/resources.ovirt.org.conf
>=20
> I guess we should add this in some puppet module somewhere ?
=20
Maybe also limit bandwidth per IP?
For now, the issue was more the ressources used serverside ( ie, 1 httpd
slot per request ). Limit per IP could make sense, I didn't look at a
supported apache module. Seems there is mod_bw for that.
--=20
Michael Scherer
Open Source and Standards, Sysadmin
--=-3ju7/Z5KhyO8SK89vDk9
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part
Content-Transfer-Encoding: 7bit
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAABAgAGBQJToFU4AAoJEE89Wa+PrSK9jyEP/2alxR0yC4F9Nl00PDdn+pAi
f3RrxvaN1h6UAit4qsszfOhVmvcXs2ILbdfMO3Fuq04QiVCcptZZpo0YlmLWEgIW
ZjI3lT9ejH6Z5GBTSvtftJygLicjnLcpofS/eHM/VlPhoYBUusBjuILWPKnbe5pi
5j/wJKHYteWbO1N6Gsd3AEGr1N/elaIoeN6x+ZGzSKRgXwz8WSBghkl1gdq1E2bi
eQH/55z9ospIwfcJNUC6pwpF2+eOORyMSm68bHjTuznmu+M16EF0s/x2/l+8qHRw
mqQRZtE/KPehw/ro+xXcoE7Ga5Ti9/Vn6S7+BgSBFpmDTqfVJOAyW7eNc2jEatpH
7boqajw7hYw61v/42k+huaiTW5yrhpIG5PiwGPEEb3Ph4FzArCMStqUXiO6xn65S
lLTzZnnJX6lRVisu5Zrn4P1PpJoG5E2yH1oR/s7TUbjRRE4DkOhgUcXk3m4iOEck
82xE+JYxNOLXM1FuvqeoAosTkXfIqPEd2GjvC0b2AbFKNDmw4qcJvWmUBdk4tsrG
0TQ2rF71AZ2v9WsuAkD/Gd8VM5+wDN0rhNyAlbHXeiMJskO5e6RLM8yrgkoHXiXH
+DKAyk7X9R5C/f5U9EhCzHV599vuXbQcJA/rqERAVvsaAilp/02Nqvp0IRhumIaN
LeyC2+1gFva8aPFunSf5
=XCj1
-----END PGP SIGNATURE-----
--=-3ju7/Z5KhyO8SK89vDk9--
3625
days inactive
3626
days old
8 comments
4 participants
participants (4)
-
Ewoud Kohl van Wijngaarden -
Eyal Edri -
Michael Scherer -
Sandro Bonazzola