Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, On 27.09.2014 13:18, Daniel Helgenberger wrote:
Hallo Dan,
I just opened a BZ aginst this [1]. Please let me know if I can be of further assistance. I stopped the cron script for now, so vdsm can 'grow nicely' (about 91.5 MB per 6hrs)
Cheers, [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148
On 24.09.2014 19:20, Dan Kenigsberg wrote:
On 01.09.2014 18:08, Dan Kenigsberg wrote:
On Mon, Sep 01, 2014 at 03:30:53PM +0000, Daniel Helgenberger wrote:
Hello,
in my LAB cluster I run into OOM conditions frequently because of a huge VDSM process. The memory stats from my nodes right now:
Node A; running one VM: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3465 vdsm 0 -20 18.6g 9.8g 8244 S 32.1 50.3 27265:21 vdsm 7439 qemu 20 0 5641m 4.1g 4280 S 22.9 20.9 12737:08 qemu-kvm 2912 root 15 -5 2710m 35m 5968 S 0.0 0.2 0:04.76 supervdsmServer
Node B, running 3 VMs including HosedEngine: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9079 vdsm 0 -20 9.9g 5.0g 7496 S 49.7 43.0 11858:06 vdsm 3347 qemu 20 0 7749m 1.8g 5264 S 4.3 15.8 3:25.71 qemu-kvm 18463 qemu 20 0 3865m 415m 5516 R 1.6 3.5 359:15.24 qemu-kvm 11755 qemu 20 0 3873m 276m 5336 S 80.5 2.3 21639:39 qemu-kvm
Basically VDSM consumes more then all my VMs.
I thought of VDSM as a 'supervisor' process for qemu-kvm?
I attached recend vdsm logs as well as a screen shot. Thanks for your report. It sounds like
Bug 1130045 - Very high memory consumption
which we believe is due to python-ioprocess. On Wed, Sep 24, 2014 at 03:39:25PM +0000, Daniel Helgenberger wrote: Hi Dan,
just to get this right, you yourself pointed me to the BZ. I was looking at the duplicate, since the metadata tags 3.4. Sorry my lack of knowlage, I really have no idea whatever there is an ioprocess python binding in 3.4 or not - I just see vdsmd resident size growing in 3.4. The top output below was from 3.4.3; I just upgraded to 3.4.4. But clearly vdsmd should not use 10GB RAM? I'm sorry to have mislead you. The bug I refered to was indeed due to ioprocess, and caused a very dramatic memory leak in 3.5. We have yet another memory leak in 3.5.0, when managing gluster blocks Bug 1142647 - supervdsm leaks memory when using glusterfs
3.4.z does not use ioprocess, and do not have that gluster bug, so you are seeing completely different and much older.
These leaks are not so easy to debug - but they are important. I'd love if you open a BZ about it. Please specify the rate of the leak, when does it happen (when a host has VMs? when a host is polled by Engine? On nfs? iscsi?
What's `lsof -p pid-of-fat-vdsm`?
I think that Francesco has some debug patches to help nail it down - he can provide smarter questions.
Dan.
-- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767