8:45 p.m.
Hi everybody.
So I ran into that high mem usage thing. The problem I have with patching
is that this is a live system so I can't do it mid day. Can anybody tell
me if it is possible to just restart the vdsm service or does the host have
to be in "maintenance mode" before restarting it? It is using gluster
storage, if that makes a difference as well.
Thanks,
--
*Michael Kleinpaste*
Senior Systems Administrator
SharperLending, LLC.
www.SharperLending.com
Michael.Kleinpaste(a)SharperLending.com
(509) 324-1230 Fax: (509) 324-1234
4:29 a.m.
--Apple-Mail=_B86C8BFC-E955-41B4-9601-CB6FF25A370A
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
If you=E2=80=99re using nfs mounts (even if they are gluster based), =
it=E2=80=99s safe to restart vdsmd, you=E2=80=99ll see it change status =
in ovirt, but your VMs will continue running. If you=E2=80=99re mounting =
gluster based storage as glusterfs shares directly (not over nfs), =
there=E2=80=99s another issue that will cause all your VMs to pause and =
the only way to recover is to stop them and restart them, but that=E2=80=99=
s going to happen to them anyway when vdsmd runs out of ram and =
crashes=E2=80=A6 Best solution is to migrate them yourself in this case, =
then restart and migrate back. Or live migrate them to NFS mounted =
storage so when vdsm crashes they don=E2=80=99t lock up, and clean up =
after you=E2=80=99ve had an opportunity to upgrade or patch.
Upgrade to 3.5.3 or later at your earliest opportunity, the mem leak is =
resolved there. Sounds like you already found the patch you can apply if =
upgrading isn=E2=80=99t an option, but it will still require you to =
restart your vdsms.
-Darrell
On Sep 10, 2015, at 1:45 PM, Michael Kleinpaste =
<michael.kleinpaste(a)sharperlending.com> wrote:
=20
Hi everybody.
=20
So I ran into that high mem usage thing. The problem I have with =
patching is that
this is a live system so I can't do it mid day. Can =
anybody tell me if it is possible to just restart the vdsm service or =
does the host have to be in "maintenance mode" before restarting it? It =
is using gluster storage, if that makes a difference as well.
=20
Thanks,
=20
--=20
Michael Kleinpaste
Senior Systems Administrator
SharperLending, LLC.
www.SharperLending.com <>
Michael.Kleinpaste(a)SharperLending.com
(509) 324-1230 Fax: (509) 324-1234
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_B86C8BFC-E955-41B4-9601-CB6FF25A370A
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">If you=E2=80=99re using nfs mounts (even if they are gluster =
based), it=E2=80=99s safe to restart vdsmd, you=E2=80=99ll see it change =
status in ovirt, but your VMs will continue running. If you=E2=80=99re =
mounting gluster based storage as glusterfs shares directly (not over =
nfs), there=E2=80=99s another issue that will cause all your VMs to =
pause and the only way to recover is to stop them and restart them, but =
that=E2=80=99s going to happen to them anyway when vdsmd runs out of ram =
and crashes=E2=80=A6 Best solution is to migrate them yourself in this =
case, then restart and migrate back. Or live migrate them to NFS mounted =
storage so when vdsm crashes they don=E2=80=99t lock up, and clean up =
after you=E2=80=99ve had an opportunity to upgrade or patch.<div =
class=3D""><br class=3D""></div><div
class=3D"">Upgrade to 3.5.3 or =
later at your earliest opportunity, the mem leak is resolved there. =
Sounds like you already found the patch you can apply if upgrading =
isn=E2=80=99t an option, but it will still require you to restart your =
vdsms.</div><div class=3D""><br
class=3D""></div><div class=3D""> =
-Darrell</div><div class=3D""><br
class=3D""><div><blockquote =
type=3D"cite" class=3D""><div class=3D"">On Sep 10,
2015, at 1:45 PM, =
Michael Kleinpaste <<a =
href=3D"mailto:michael.kleinpaste@sharperlending.com" =
class=3D"">michael.kleinpaste(a)sharperlending.com</a>&gt;
wrote:</div><br =
class=3D"Apple-interchange-newline"><div class=3D""><div
dir=3D"ltr" =
class=3D""><div class=3D"uyb8Gf" =
style=3D"font-size:13.2px;line-height:19.8px"><div
class=3D"F3hlO"><div =
dir=3D"ltr" class=3D"">Hi everybody.<br
class=3D""><br class=3D"">So I =
ran into that high mem usage thing. The problem I have with patching is =
that this is a live system so I can't do it mid day. Can anybody =
tell me if it is possible to just restart the vdsm service or does the =
host have to be in "maintenance mode" before restarting it? It is =
using gluster storage, if that makes a difference as well.<div =
class=3D""><br class=3D""></div><div
class=3D"">Thanks,</div><div =
class=3D""><br
class=3D""></div></div></div></div></div><div
dir=3D"ltr" =
class=3D"">-- <br class=3D""></div><div
dir=3D"ltr" class=3D""><b =
class=3D"">Michael Kleinpaste</b><br
class=3D""><span class=3D"">Senior =
Systems Administrator</span><br class=3D""><span =
class=3D"">SharperLending, LLC.</span><br
class=3D""><a =
class=3D"">www.SharperLending.com</a><br
class=3D""><span class=3D""><a =
href=3D"mailto:Michael.Kleinpaste@SharperLending.com" =
class=3D"">Michael.Kleinpaste(a)SharperLending.com</a></span><br
=
class=3D""><span class=3D"">(509) 324-1230 Fax:
(509) =
324-1234</span></div>
_______________________________________________<br class=3D"">Users =
mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org"
=
class=3D"">Users(a)ovirt.org</a><br =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br =
class=3D""></div></blockquote></div><br
class=3D""></div></body></html>=
--Apple-Mail=_B86C8BFC-E955-41B4-9601-CB6FF25A370A--
12:35 p.m.
Hello Michael,
I ran into the issue myself and can confirm restarting vdsm with nfs
mitgates the issue. I even had a cron job for that
On 11.09.2015 04:30, Darrell Budic wrote:
If you’re using nfs mounts (even if they are gluster based), it’s
safe to
restart vdsmd, you’ll see it change status in ovirt, but your VMs will continue
running. If you’re mounting gluster based storage as glusterfs shares directly
(not over nfs), there’s another issue that will cause all your VMs to pause and
the only way to recover is to stop them and restart them, but that’s going to
happen to them anyway when vdsmd runs out of ram and crashes… Best solution is
to migrate them yourself in this case, then restart and migrate back.
This is what
I have done. The easiest way to do so is to set the host in
maintenance, wait for the migration finishes and then restart vdsm. You
sould do this only at one host and then wait a while so you do not run
into OOM on the whole cluster at once.
Or live
migrate them to NFS mounted storage so when vdsm crashes they don’t
lock up, and
clean up after you’ve had an opportunity to upgrade or patch.
Upgrade to 3.5.3 or later at your earliest opportunity, the mem leak is resolved
there. Sounds like you already found the patch you can apply if upgrading isn’t
an option, but it will still require you to restart your vdsms.
I can confirm 3.5.3 finally solved the issue for us and VDSM keeps below
100MB RSS.
-Darrell
> On Sep 10, 2015, at 1:45 PM, Michael Kleinpaste
> <michael.kleinpaste(a)sharperlending.com
> <mailto:michael.kleinpaste@sharperlending.com>> wrote:
>
> Hi everybody.
>
> So I ran into that high mem usage thing. The problem I have with patching is
> that this is a live system so I can't do it mid day. Can anybody tell me if
> it is possible to just restart the vdsm service or does the host have to be in
> "maintenance mode" before restarting it? It is using gluster storage, if
that
> makes a difference as well.
>
> Thanks,
>
> --
> *Michael Kleinpaste*
> Senior Systems Administrator
> SharperLending, LLC.
> www.SharperLending.com
> Michael.Kleinpaste(a)SharperLending.com
> <mailto:Michael.Kleinpaste@SharperLending.com>
> (509) 324-1230 Fax: (509) 324-1234
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
--
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de www.monkeymen.tv
Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
3172
days inactive
3173
days old
2 comments
3 participants
participants (3)
-
Daniel Helgenberger -
Darrell Budic -
Michael Kleinpaste