[ovirt-devel] [VDSM] health: Introduce Vdsm health monitoring

[Francesco Romani]

----- Original Message -----
> From: "Nir Soffer" <nsoffer at redhat.com>
> To: "devel" <devel at ovirt.org>, "Francesco Romani" <fromani at redhat.com>
> Sent: Friday, January 15, 2016 9:52:13 AM
> Subject: Re: [VDSM] health: Introduce Vdsm health monitoring
> 
> I tested with https://gerrit.ovirt.org/#/c/51630
> 
> libvirt.virDomain object does not leak any more, but pthreading object
> are still leaking,
> and the number of uncollectible objects is still very high (12,915).
> 
> Each time we start and stop a vm, we leak about 6000 objects which are part
> of a reference cycle, and each time we create and leak 27 pthreading
> locks/conditions
> which are probably the reason for this leak (because they implement
> __del__ or reference object
> implementing __del__.

*Surely* this deserve more investigation, *most likley* there is more to be fixed,
and *surely* I'll invest more energy in research, but I think we should also apply
some grains of salt to the raw results gc gives us.
I've been running a test overnight like

- start 20 vms
- sleep 2m
- stop 20 vms
- sleep 30s

repeated 100 times, monitoring VmRSS (/proc/$VDSM_PID/status) and I didn't observed
any _significant_ leak.
The VmRSS growth was pretty low, almost flat. So something doesn't click here.

I mean, 6000 objects (at least) per vm each time *should* be noticeable after all!

Once 51630 is in good shape I'll broad the net and go hunt for more virt leaks.

Bests,

-- 
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani