[ovirt-devel] Blog post: Monitoring Improvements in oVirt

[Arik Hadas]

----- Original Message -----
> Hi Arik,
> 
> Very nice output..

Thank you

> 
> see also our mojo page:
> 
> https://mojo.redhat.com/groups/rhev-m-scalability-performance-team
> 
> can you elaborate how did you rampup 6K vms?

1. Created a cluster with 1 real host, 1 fake-VDSM based host and 1 storage domain (so the storage pool will be up).
2. Created a pool with 6000 VMs that is based on template with one network interface and no disks.
3. Ran all the VMs on fake-VDSM (they were pinned to that host)

At that point I had a fake-VDSM host that runs 6K VMs.
Then, 1. I put all the things in my master-based setup to maintenance and removed all VMs.
      2. Configured 1 cluster with this host - all the 6K VMs were added as external VMs.
And on 3.6 I did the same - created a cluster only with that host (and the VMs were added as external VMs).

By this, I had the exact same setup on both 3.6 and 4.0 and I didn't have anything else that could influence the measurements.

Note that that's why many of the numbers presented in the post should not be considered as absolute values but only
as a relative values between 3.6 and 4.0 setups.
My point is that when I say that saving the statistics took 2% of the interactions with the database, this '2%'
would probably be different in other environments and lower when the system does other things my system did not do.

> 
> also can you elaborate the following:
> 
> - HW definition?

Dell optiplex 780, 16GB of RAM, 4 CPUs with 4 cores-per-CPI
Both the engine and fake VDSM were running on that host.

> 
> - java HEAP Size?

I didn't change it manually - initial and maximum heap size of 3971M

> 
> - any postgres tuning ?

No, I kept the default settings.

> 
> 
> we are about to scale out the latest 4.0 RHEVM
> 
> any relevant changes that we should aware of?

Note that only part of these changes are in 4.0.
What bothers me and worth checking on 4.0 is the memory consumption -
in 4.0 we already cache VM statistics but many of the changes that probably lead
of the overall decrement of the memory consumption are missing there.
So it would be interesting to see what is the implication of storing more data
in memory in 4.0 - if it is problematic we can easily remove this cache as it
is rarely used in 4.0.

> 
> -Eldad
> 
> 
> 
> On 07/24/2016 05:15 PM, Arik Hadas wrote:
> > Hi all,
> >
> > I wrote a blog post that summarizes the recent improvements in the code
> > that monitors virtual machines in oVirt.
> > It describes the changes that were done and shows a comparison of different
> > aspects of oVirt with these changes
> > (based on the master branch) vs oVirt 3.6 without these changes:
> > http://ahadas.github.io/monitoring-improvements-in-ovirt/
> >
> > It is mostly a technical post.
> >
> > However, users might find it interesting as well.
> > Note that the measurements were taken on an environment with 1 host and
> > 6000 running VMs (in a pool).
> > Each VM had a network interface and was diskless.
> > Obviously, this is not a typical setup but it was useful to easily expose
> > some of weak sides of the monitoring.
> >
> > Your feedback is always welcome!
> >
> > Regards,
> > Arik
> > _______________________________________________
> > Devel mailing list
> > Devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
> 
>