[ovirt-users] 3.4: VDSM Memory consumption
Daniel Helgenberger
daniel.helgenberger at m-box.de
Wed Sep 24 17:32:56 UTC 2014
On 24.09.2014 19:20, Dan Kenigsberg wrote:
>> On 01.09.2014 18:08, Dan Kenigsberg wrote:
>>> On Mon, Sep 01, 2014 at 03:30:53PM +0000, Daniel Helgenberger wrote:
>>>> Hello,
>>>>
>>>> in my LAB cluster I run into OOM conditions frequently because of a huge
>>>> VDSM process. The memory stats from my nodes right now:
>>>>
>>>> Node A; running one VM:
>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>>> 3465 vdsm 0 -20 18.6g 9.8g 8244 S 32.1 50.3 27265:21 vdsm
>>>> 7439 qemu 20 0 5641m 4.1g 4280 S 22.9 20.9 12737:08 qemu-kvm
>>>> 2912 root 15 -5 2710m 35m 5968 S 0.0 0.2 0:04.76 supervdsmServer
>>>>
>>>> Node B, running 3 VMs including HosedEngine:
>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>>> 9079 vdsm 0 -20 9.9g 5.0g 7496 S 49.7 43.0 11858:06 vdsm
>>>> 3347 qemu 20 0 7749m 1.8g 5264 S 4.3 15.8 3:25.71 qemu-kvm
>>>> 18463 qemu 20 0 3865m 415m 5516 R 1.6 3.5 359:15.24 qemu-kvm
>>>> 11755 qemu 20 0 3873m 276m 5336 S 80.5 2.3 21639:39 qemu-kvm
>>>>
>>>> Basically VDSM consumes more then all my VMs.
>>>>
>>>> I thought of VDSM as a 'supervisor' process for qemu-kvm?
>>>>
>>>> I attached recend vdsm logs as well as a screen shot.
>>> Thanks for your report. It sounds like
>>>
>>> Bug 1130045 - Very high memory consumption
>>>
>>> which we believe is due to python-ioprocess.
> On Wed, Sep 24, 2014 at 03:39:25PM +0000, Daniel Helgenberger wrote:
>> Hi Dan,
>>
>> just to get this right, you yourself pointed me to the BZ. I was looking
>> at the duplicate, since the metadata tags 3.4. Sorry my lack of
>> knowlage, I really have no idea whatever there is an ioprocess python
>> binding in 3.4 or not - I just see vdsmd resident size growing in 3.4.
>> The top output below was from 3.4.3; I just upgraded to 3.4.4. But
>> clearly vdsmd should not use 10GB RAM?
> I'm sorry to have mislead you.
No harm done, I am glad for your help anyway.
> The bug I refered to was indeed due to
> ioprocess, and caused a very dramatic memory leak in 3.5. We have yet
> another memory leak in 3.5.0, when managing gluster blocks
> Bug 1142647 - supervdsm leaks memory when using glusterfs
>
> 3.4.z does not use ioprocess, and do not have that gluster bug, so you
> are seeing completely different and much older.
>
> These leaks are not so easy to debug - but they are important. I'd love
> if you open a BZ about it. Please specify the rate of the leak, when
> does it happen (when a host has VMs? when a host is polled by Engine? On
> nfs? iscsi?
>
> What's `lsof -p pid-of-fat-vdsm`?
I just updated to 3.4.4 - so this is hard to tell. I've deactivated the
cron jobs for now. The problem started back in August with the supplied
yum transaction. I also attached the corresponding ovirt-report.
I will let you know if the problem persists and open a BZ if it is still
there...
# yum history info 28
Loaded plugins: fastestmirror
Transaction ID : 28
Begin time : Sat Aug 16 12:37:48 2014
Begin rpmdb : 436:1801c628ad395bb41146509ec1d42efd2f1c1c85
End time : 12:38:10 2014 (22 seconds)
End rpmdb : 436:b8d7f91fed480997ad891cb299a2483fe61a9d3a
User : root <root>
Return-Code : Success
Transaction performed with:
Installed rpm-4.8.0-37.el6.x86_64
@anaconda-CentOS-201311272149.x86_64/6.5
Installed yum-3.2.29-43.el6.centos.noarch @updates
Installed yum-metadata-parser-1.1.2-16.el6.x86_64
@anaconda-CentOS-201311272149.x86_64/6.5
Installed yum-plugin-fastestmirror-1.1.30-17.el6_5.noarch @updates
Packages Altered:
Updated vdsm-4.14.11-0.el6.x86_64
@ovirt-3.4-stable
Update 4.14.11.2-0.el6.x86_64
@ovirt-3.4-stable
Updated vdsm-cli-4.14.11-0.el6.noarch
@ovirt-3.4-stable
Update 4.14.11.2-0.el6.noarch
@ovirt-3.4-stable
Updated vdsm-hook-extnet-4.14.11-0.el6.noarch
@ovirt-3.4-stable
Update 4.14.11.2-0.el6.noarch
@ovirt-3.4-stable
Updated vdsm-hook-vmfex-dev-4.14.11-0.el6.noarch
@ovirt-3.4-stable
Update 4.14.11.2-0.el6.noarch
@ovirt-3.4-stable
Updated vdsm-python-4.14.11-0.el6.x86_64
@ovirt-3.4-stable
Update 4.14.11.2-0.el6.x86_64
@ovirt-3.4-stable
Updated vdsm-python-zombiereaper-4.14.11-0.el6.noarch
@ovirt-3.4-stable
Update 4.14.11.2-0.el6.noarch
@ovirt-3.4-stable
Updated vdsm-xmlrpc-4.14.11-0.el6.noarch
@ovirt-3.4-stable
Update 4.14.11.2-0.el6.noarch
@ovirt-3.4-stable
Scriptlet output:
1
2 Checking configuration status...
3
4
5 Running configure...
6
7 Done configuring modules to VDSM.
history info
>
> I think that Francesco has some debug patches to help nail it down - he
> can provide smarter questions.
>
> Dan.
>
>
--
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de www.monkeymen.tv
Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
-------------- next part --------------
A non-text attachment was scrubbed...
Name: single_host_resource_br2a.pdf
Type: application/pdf
Size: 135213 bytes
Desc: single_host_resource_br2a.pdf
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140924/a347a391/attachment-0001.pdf>
More information about the Users
mailing list