Op 26 apr. 2016 om 14:19 heeft Alexander Wels <awels(a)redhat.com&gt; het volgende geschreven: > On Tuesday, April 26, 2016 12:13:11 PM Joost(a)familiealbers.nl wrote: > Hi all, i have a somewhat awkward requirement to deploy datacenters to > around 500 satelite locations. Each datacenter will have 1 or 2 hypervisors > and if 2 a cluster with glusterfs for shared storage will be deployed. I am > expecting to run between 4 and 10 vms per data center. During operations > it is expected 20% to 40% of the satelite locations will be down or have a > very bad connection to the engine. I will still like to manage these > datacenters from ovirt engine. Does anyone have any figures on how many > datacenters i can add given what requirements for a single engine and > perhaps some best practices for dealing with 'bad' connections between the > hypervisors and the engine. Thanks a lot! Sounds like a good use case for manage IQ. If possible I would do hosted engines in each satellite location, and then use manage IQ to manage the engines. Alexander ps. Vrolijk Koning dag morgen > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users

I think that 1000 hosts per engine is a bit over what we recommend (and support). The fact that all of them are going to be remote might not be ideal either. The engine assumes the network connection to all hosts is almost flawless and the necessary routing and distance to your hosts might not play nice with (for example) the fencing logic.

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQGcBAEBCAAGBQJXH3VKAAoJEAq0kGAWDrqlrU4MALclfaT9+WJgapJdMglTHE6d G8DX2Ka0fAs3rphQDYBpZW73pvgfak0cZmD+MRyHKhn3leCxi0BAwnjlFJOXyt6c XbP/2FiCbq59+sLHT4JoYvgiZLbKGGuCmXNxAmrGHNmReWyXQutoviyK8wxKzg3Q YRsGoi8tHw6aLtAznjbqZ+3WZlACSMnF0gurQrdVfWhQYSHMLkxKMCyNWQECDUc2 yoW52/JS+ghwdTvnhq8PDLZsgUAhRruCZA77vCOa62T5OsIKKISsqOSq0FoBvAi2 WRofgaDNuFgqpXlHBtOswMCWQ2rxhpYzsJqKTsMUwxql2/N/bLe7+RLnMZPqEVt0 fIEllQIUAQLxMJZ1Yi0VDNrUwL/OoAWUS3yrreigERHg/DPY9aohVvZt5+tYYXlK yWowkqSBDQr7xoOpYnekeM5F+S1ivXBlLFpi5EwJDlvArip400r70Gq3Vouifx7P 0QPjnoORNfcjUHZQpeZgJZ6qx9O2GUDCTeCJlN98FA== =zIZZ -----END PGP SIGNATURE-----

On 26 Apr 2016, at 16:48, Martin Sivak <msivak(a)redhat.com&gt; wrote: > @awels: to add another layer of indirection via a dedicated > hosted-engine per outlet seems a little much. we are talking about 500 * > 4GB RAM at least in this example, so 2 TB RAM just for management > purposes, if you follow engine hardware recommendations? I would not go that far. Creating zones per continent (for example) might be enough. > At least RHEV states in the documentation you support up to 200 hosts > per cluster alone. The default configuration seems to only allow 250 hosts per datacenter. # engine-config -g MaxNumberOfHostsInStoragePool MaxNumberOfHostsInStoragePool: 250 version: general

-- Martin Sivak SLA / oVirt On Tue, Apr 26, 2016 at 4:03 PM, Sven Kieske <svenkieske(a)gmail.com&gt; wrote: > On 26.04.2016 14:46, Martin Sivak wrote: >> I think that 1000 hosts per engine is a bit over what we recommend >> (and support). The fact that all of them are going to be remote might >> not be ideal either. The engine assumes the network connection to all >> hosts is almost flawless and the necessary routing and distance to >> your hosts might not play nice with (for example) the fencing logic. > > Hi, > > this seems a little surprising. > > At least RHEV states in the documentation you support up to 200 hosts > per cluster alone. > > There are no documented maxima for clusters or datacenters though. > > @awels: to add another layer of indirection via a dedicated > hosted-engine per outlet seems a little much. we are talking about 500 * > 4GB RAM at least in this example, so 2 TB RAM just for management > purposes, if you follow engine hardware recommendations?

> > But I agree, ovirt does not handle unstable or remote connections that

> well, so you might be better of with hundredths of remote engines, but > it seems to be a nightmare to manage, even if you automate everything. > > My personal experience is, that ovirt does scale at least until about > 30-50 DCs managed by a single engine, but that setup was also on a LAN > (but I would say it could scale well beyond these numbers, at least on a > LAN). > > HTH > > Sven > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On 28 Apr 2016, at 14:39, Joost(a)familiealbers.nl wrote: Ok that sounds promising. Does it require 3.6 at minimum?

We already handle starting vms using vdsm in case the network from hypervisors to engine is down.

As well as disabling fencingmto avoid a sloppy network to try and make changes. I would like to reduce the comms between hosts and engine. Are there any ideas you would have about that.

Verstuurd vanaf mijn iPhone > Op 28 apr. 2016 om 13:55 heeft Michal Skrivanek <michal.skrivanek(a)redhat.com&gt; het volgende geschreven: > > >>> On 26 Apr 2016, at 16:48, Martin Sivak <msivak(a)redhat.com&gt; wrote: >>> >>> @awels: to add another layer of indirection via a dedicated >>> hosted-engine per outlet seems a little much. we are talking about 500 * >>> 4GB RAM at least in this example, so 2 TB RAM just for management >>> purposes, if you follow engine hardware recommendations? >> >> I would not go that far. Creating zones per continent (for example) >> might be enough. >> >>> At least RHEV states in the documentation you support up to 200 hosts >>> per cluster alone. >> >> The default configuration seems to only allow 250 hosts per datacenter. >> >> # engine-config -g MaxNumberOfHostsInStoragePool >> MaxNumberOfHostsInStoragePool: 250 version: general > > yep, but that liit is there because within a DC there is a lot of assumption for flawless fast enough communication, the most problematic is that all hosts need to access the same storage and the monitoring gets expensive then. > This is a different situation with separate DCs, there’s no cross-DC communication. > I would guess many DCs work great actually. > > Too many hosts and VMs in total might be an issue, but since the last official updates there were a lot of changes. E.g. in stable state due to VM status events introduced in 3.6 the traffic required between each host and engine is much lower. > I would not be so afraid of thousands anymore, but of course YMMV > >> >> -- >> Martin Sivak >> SLA / oVirt >> >>> On Tue, Apr 26, 2016 at 4:03 PM, Sven Kieske <svenkieske(a)gmail.com&gt; wrote: >>>> On 26.04.2016 14:46, Martin Sivak wrote: >>>> I think that 1000 hosts per engine is a bit over what we recommend >>>> (and support). The fact that all of them are going to be remote might >>>> not be ideal either. The engine assumes the network connection to all >>>> hosts is almost flawless and the necessary routing and distance to >>>> your hosts might not play nice with (for example) the fencing logic. >>> >>> Hi, >>> >>> this seems a little surprising. >>> >>> At least RHEV states in the documentation you support up to 200 hosts >>> per cluster alone. >>> >>> There are no documented maxima for clusters or datacenters though. >>> >>> @awels: to add another layer of indirection via a dedicated >>> hosted-engine per outlet seems a little much. we are talking about 500 * >>> 4GB RAM at least in this example, so 2 TB RAM just for management >>> purposes, if you follow engine hardware recommendations? > > yeah. currently the added layer of manageiq with HEs everywhere is not that helpful for this particular case. Still, a per-continent split or per-low-latency-area might not be a bad idea. > I can imagine with a bit more tolerant timeouts and refreshes it might work well, with incidents/disconnects being isolated within a DC > >>> >>> But I agree, ovirt does not handle unstable or remote connections that > > right. but most of that is again per-DC. You can’t do much cross-DC though (e.g. sharing a template is a pain) > > Thanks > michal > >>> well, so you might be better of with hundredths of remote engines, but >>> it seems to be a nightmare to manage, even if you automate everything. >>> >>> My personal experience is, that ovirt does scale at least until about >>> 30-50 DCs managed by a single engine, but that setup was also on a LAN >>> (but I would say it could scale well beyond these numbers, at least on a >>> LAN). >>> >>> HTH >>> >>> Sven >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >

Op 26 apr. 2016 om 14:56 heeft Alexander Wels <awels(a)redhat.com&gt; het volgende geschreven: > On Tuesday, April 26, 2016 02:29:22 PM Joost(a)familiealbers.nl wrote: > Thanks for the response. If possible i would like to reduce the amount of > tooling. We already have quite a lot. Do you think a single ovirt engine > could cope? The reason I mentioned manage IQ is that it would give you one tool to manage everything. You would not have to do much (perhaps the initial setup) in the oVirt manager. After that everything goes through manage IQ. Since it is designed to manage the managers it is much more fault tolerant which appears to be one of your requirements. As far as max number of data centers goes. I am not entirely sure to be honest. But I believe your proposed setup will push the engine beyond it limits. I would definitely go with multiple engine instances, maybe in zones where the connection to the hosts is better, and then something to control the engines from a central location (like manage IQ). > Verstuurd vanaf mijn iPhone > >> Op 26 apr. 2016 om 14:19 heeft Alexander Wels <awels(a)redhat.com&gt; het volgende geschreven: >>> On Tuesday, April 26, 2016 12:13:11 PM Joost(a)familiealbers.nl wrote: >>> Hi all, i have a somewhat awkward requirement to deploy datacenters to >>> around 500 satelite locations. Each datacenter will have 1 or 2 >>> hypervisors >>> and if 2 a cluster with glusterfs for shared storage will be deployed. I >>> am >>> expecting to run between 4 and 10 vms per data center. During operations >>> it is expected 20% to 40% of the satelite locations will be down or have >>> a >>> very bad connection to the engine. I will still like to manage these >>> datacenters from ovirt engine. Does anyone have any figures on how many >>> datacenters i can add given what requirements for a single engine and >>> perhaps some best practices for dealing with 'bad' connections between >>> the >>> hypervisors and the engine. Thanks a lot! >> >> Sounds like a good use case for manage IQ. If possible I would do hosted >> engines in each satellite location, and then use manage IQ to manage the >> engines. >> >> Alexander >> >> ps. >> Vrolijk Koning dag morgen >> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users