[ovirt-users] 3.4: VDSM Memory consumption

[Dan Kenigsberg]

> On 01.09.2014 18:08, Dan Kenigsberg wrote:
> > On Mon, Sep 01, 2014 at 03:30:53PM +0000, Daniel Helgenberger wrote:
> >> Hello,
> >>
> >> in my LAB cluster I run into OOM conditions frequently because of a huge
> >> VDSM process. The memory stats from my nodes right now:
> >>
> >> Node A; running one VM:
> >>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
> >>  3465 vdsm       0 -20 18.6g 9.8g 8244 S 32.1 50.3  27265:21 vdsm                                                                                                                                                  
> >>  7439 qemu      20   0 5641m 4.1g 4280 S 22.9 20.9  12737:08 qemu-kvm                                                                                                                                              
> >>  2912 root      15  -5 2710m  35m 5968 S  0.0  0.2   0:04.76 supervdsmServer 
> >>
> >> Node B, running 3 VMs including HosedEngine:
> >>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
> >> 9079 vdsm       0 -20  9.9g 5.0g 7496 S 49.7 43.0  11858:06 vdsm                                                                                                                                                  
> >>  3347 qemu      20   0 7749m 1.8g 5264 S  4.3 15.8   3:25.71 qemu-kvm                                                                                                                                              
> >> 18463 qemu      20   0 3865m 415m 5516 R  1.6  3.5 359:15.24 qemu-kvm                                                                                                                                              
> >> 11755 qemu      20   0 3873m 276m 5336 S 80.5  2.3  21639:39 qemu-kvm
> >>
> >> Basically VDSM consumes more then all my VMs.
> >>
> >> I thought of VDSM as a 'supervisor' process for qemu-kvm?
> >>
> >> I attached recend vdsm logs as well as a screen shot.
> > Thanks for your report. It sounds like
> >
> >     Bug 1130045 - Very high memory consumption
> >
> > which we believe is due to python-ioprocess.

On Wed, Sep 24, 2014 at 03:39:25PM +0000, Daniel Helgenberger wrote:
> Hi Dan,
> 
> just to get this right, you yourself pointed me to the BZ. I was looking
> at the duplicate, since the metadata tags 3.4. Sorry my lack of
> knowlage, I really have no idea whatever there is an ioprocess python
> binding in 3.4 or not - I just see vdsmd resident size growing in 3.4.
> The top output below was from 3.4.3; I just upgraded to 3.4.4. But
> clearly vdsmd should not use 10GB RAM?

I'm sorry to have mislead you. The bug I refered to was indeed due to
ioprocess, and caused a very dramatic memory leak in 3.5. We have yet
another memory leak in 3.5.0, when managing gluster blocks
    Bug 1142647 - supervdsm leaks memory when using glusterfs

3.4.z does not use ioprocess, and do not have that gluster bug, so you
are seeing completely different and much older.

These leaks are not so easy to debug - but they are important. I'd love
if you open a BZ about it. Please specify the rate of the leak, when
does it happen (when a host has VMs? when a host is polled by Engine? On
nfs? iscsi?

What's `lsof -p pid-of-fat-vdsm`?

I think that Francesco has some debug patches to help nail it down - he
can provide smarter questions.

Dan.