10:57 a.m.
It looks like we are experiencing a bug in the version of glusterfs
included with ovirt 4.2.3. It looks like glusterfs 3.12.x has an issue
where it consumes large amounts of ram which has caused our HV report
storage errors and pause VMs.
https://bugzilla.redhat.com/show_bug.cgi?id=1496379
Is there a safe way to get glusterfs-fuse v3.13.x installed with ovirt
4.2.3 or do we have to live with this issue until future updates to
ovirt/glusterfs are released?
$ ssh hv1.domain.com "sudo grep glusterfs
/var/log/messages-* | grep -i kill "
/var/log/messages-20180610:Jun 6 13:49:54 hv1 kernel: Out of memory:
Kill process 15353 (glusterfs) score 630 or sacrifice child
/var/log/messages-20180610:Jun 6 13:49:54 hv1 kernel: Killed process
15353 (glusterfs) total-vm:33800604kB, anon-rss:31896632kB,
file-rss:840kB, shmem-rss:0kB
/var/log/messages-20180617:Jun 17 00:24:16 hv1 kernel: Out of memory:
Kill process 4072 (glusterfs) score 678 or sacrifice child
/var/log/messages-20180617:Jun 17 00:24:16 hv1 kernel: Killed process
4072 (glusterfs) total-vm:36159900kB, anon-rss:34338508kB,
file-rss:888kB, shmem-rss:0kB
We see this same issue occur on all hv in our cluster.
Edward Clay
Systems Administrator
The Hut Group<http://www.thehutgroup.com/>
Tel:
Email: edward.clay@uk2group.com<mailto:edward.clay@uk2group.com>
For the purposes of this email, the "company" means The Hut Group Limited, a
company registered in England and Wales (company number 6539496) whose registered office
is at Fifth Floor, Voyager House, Chicago Avenue, Manchester Airport, M90 3DQ and/or any
of its respective subsidiaries.
Confidentiality Notice
This e-mail is confidential and intended for the use of the named recipient only. If you
are not the intended recipient please notify us by telephone immediately on +44(0)1606
811888 or return it to us by e-mail. Please then delete it from your system and note that
any use, dissemination, forwarding, printing or copying is strictly prohibited. Any views
or opinions are solely those of the author and do not necessarily represent those of the
company.
Encryptions and Viruses
Please note that this e-mail and any attachments have not been encrypted. They may
therefore be liable to be compromised. Please also note that it is your responsibility to
scan this e-mail and any attachments for viruses. We do not, to the extent permitted by
law, accept any liability (whether in contract, negligence or otherwise) for any virus
infection and/or external compromise of security and/or confidentiality in relation to
transmissions sent by e-mail.
Monitoring
Activity and use of the company's systems is monitored to secure its effective use and
operation and for other lawful business purposes. Communications using these systems will
also be monitored and may be recorded to secure effective use and operation and for other
lawful business purposes.
hgvyjuv
3:54 a.m.
On Mon, Jun 18, 2018 at 11:57 AM, Edward Clay <edward.clay(a)uk2group.com>
wrote:
It looks like we are experiencing a bug in the version of glusterfs
included with ovirt 4.2.3. It looks like glusterfs 3.12.x has an issue
where it consumes large amounts of ram which has caused our HV report
storage errors and pause VMs.
https://bugzilla.redhat.com/show_bug.cgi?id=1496379
Hi,
The bug you have referenced is applicable to tiered volume type or volumes
with quota enabled. What's your volume configuration - can you provide
output of "gluster volume info" ?
Is there a safe way to get glusterfs-fuse v3.13.x installed with ovirt
4.2.3 or do we have to live with this issue until future updates to
ovirt/glusterfs are released?
$ ssh hv1.domain.com "sudo grep glusterfs
/var/log/messages-* | grep -i kill "
/var/log/messages-20180610:Jun 6 13:49:54 hv1 kernel: Out of memory:
Kill process 15353 (glusterfs) score 630 or sacrifice child
/var/log/messages-20180610:Jun 6 13:49:54 hv1 kernel: Killed process
15353 (glusterfs) total-vm:33800604kB, anon-rss:31896632kB,
file-rss:840kB, shmem-rss:0kB
/var/log/messages-20180617:Jun 17 00:24:16 hv1 kernel: Out of memory:
Kill process 4072 (glusterfs) score 678 or sacrifice child
/var/log/messages-20180617:Jun 17 00:24:16 hv1 kernel: Killed process
4072 (glusterfs) total-vm:36159900kB, anon-rss:34338508kB,
file-rss:888kB, shmem-rss:0kB
We see this same issue occur on all hv in our cluster.
Edward Clay
Systems Administrator
The Hut Group<http://www.thehutgroup.com/>
Tel:
Email: edward.clay@uk2group.com<mailto:edward.clay@uk2group.com>
For the purposes of this email, the "company" means The Hut Group Limited,
a company registered in England and Wales (company number 6539496) whose
registered office is at Fifth Floor, Voyager House, Chicago Avenue,
Manchester Airport, M90 3DQ and/or any of its respective subsidiaries.
Confidentiality Notice
This e-mail is confidential and intended for the use of the named
recipient only. If you are not the intended recipient please notify us by
telephone immediately on +44(0)1606 811888 or return it to us by e-mail.
Please then delete it from your system and note that any use,
dissemination, forwarding, printing or copying is strictly prohibited. Any
views or opinions are solely those of the author and do not necessarily
represent those of the company.
Encryptions and Viruses
Please note that this e-mail and any attachments have not been encrypted.
They may therefore be liable to be compromised. Please also note that it is
your responsibility to scan this e-mail and any attachments for viruses. We
do not, to the extent permitted by law, accept any liability (whether in
contract, negligence or otherwise) for any virus infection and/or external
compromise of security and/or confidentiality in relation to transmissions
sent by e-mail.
Monitoring
Activity and use of the company's systems is monitored to secure its
effective use and operation and for other lawful business purposes.
Communications using these systems will also be monitored and may be
recorded to secure effective use and operation and for other lawful
business purposes.
hgvyjuv
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/communit
y/about/community-guidelines/
List Archives: https://lists.ovirt.org/archiv
es/list/users(a)ovirt.org/message/WRYEBOLNHJZGKKJUNF77TJ7WMBS66ZYK/
4:39 a.m.
Hi,
Your glusterfs version is also necessary to check if its the same memory leak.
You have mentioned that you have used 3.12.x need to know which version is it.
The above bug was fixed with 3.12.2 if you have used 3.12.1 then the fix for
above will work. If you have been using a version higher than or equal to
3.12.2 then it might be some other leak.
If its 3.12.2 and above, can you check which particular process of gluster is
consuming more RAM and we need the statedumps of that particular process
taken at two points. The first one can be taken any time and the second after
a certain interval (like a day something as interval would be good). You can
create a bug and attach the statedumps there so the person how debugs it
will be able to make use of it and proceed faster.
More about statedumps :
https://github.com/gluster/glusterdocs/blob/master/Troubleshooting/stated...
filing a new bug : https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
Regards,
Hari.
On Tue, Jun 19, 2018 at 2:24 PM Sahina Bose <sabose(a)redhat.com> wrote:
On Mon, Jun 18, 2018 at 11:57 AM, Edward Clay <edward.clay(a)uk2group.com> wrote:
>
> It looks like we are experiencing a bug in the version of glusterfs
> included with ovirt 4.2.3. It looks like glusterfs 3.12.x has an issue
> where it consumes large amounts of ram which has caused our HV report
> storage errors and pause VMs.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1496379
Hi,
The bug you have referenced is applicable to tiered volume type or volumes with quota
enabled. What's your volume configuration - can you provide output of "gluster
volume info" ?
>
>
> Is there a safe way to get glusterfs-fuse v3.13.x installed with ovirt
> 4.2.3 or do we have to live with this issue until future updates to
> ovirt/glusterfs are released?
>
> $ ssh hv1.domain.com "sudo grep glusterfs
> /var/log/messages-* | grep -i kill "
> /var/log/messages-20180610:Jun 6 13:49:54 hv1 kernel: Out of memory:
> Kill process 15353 (glusterfs) score 630 or sacrifice child
> /var/log/messages-20180610:Jun 6 13:49:54 hv1 kernel: Killed process
> 15353 (glusterfs) total-vm:33800604kB, anon-rss:31896632kB,
> file-rss:840kB, shmem-rss:0kB
> /var/log/messages-20180617:Jun 17 00:24:16 hv1 kernel: Out of memory:
> Kill process 4072 (glusterfs) score 678 or sacrifice child
> /var/log/messages-20180617:Jun 17 00:24:16 hv1 kernel: Killed process
> 4072 (glusterfs) total-vm:36159900kB, anon-rss:34338508kB,
> file-rss:888kB, shmem-rss:0kB
>
>
> We see this same issue occur on all hv in our cluster.
>
> Edward Clay
> Systems Administrator
> The Hut Group<http://www.thehutgroup.com/>
>
> Tel:
> Email: edward.clay@uk2group.com<mailto:edward.clay@uk2group.com>
>
> For the purposes of this email, the "company" means The Hut Group Limited,
a company registered in England and Wales (company number 6539496) whose registered office
is at Fifth Floor, Voyager House, Chicago Avenue, Manchester Airport, M90 3DQ and/or any
of its respective subsidiaries.
>
> Confidentiality Notice
> This e-mail is confidential and intended for the use of the named recipient only. If
you are not the intended recipient please notify us by telephone immediately on +44(0)1606
811888 or return it to us by e-mail. Please then delete it from your system and note that
any use, dissemination, forwarding, printing or copying is strictly prohibited. Any views
or opinions are solely those of the author and do not necessarily represent those of the
company.
>
> Encryptions and Viruses
> Please note that this e-mail and any attachments have not been encrypted. They may
therefore be liable to be compromised. Please also note that it is your responsibility to
scan this e-mail and any attachments for viruses. We do not, to the extent permitted by
law, accept any liability (whether in contract, negligence or otherwise) for any virus
infection and/or external compromise of security and/or confidentiality in relation to
transmissions sent by e-mail.
>
> Monitoring
> Activity and use of the company's systems is monitored to secure its effective
use and operation and for other lawful business purposes. Communications using these
systems will also be monitored and may be recorded to secure effective use and operation
and for other lawful business purposes.
>
> hgvyjuv
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WRYEBOLNHJZ...
--
Regards,
Hari Gowtham.
2378
days inactive
2379
days old
2 comments
3 participants
participants (3)
-
Edward Clay -
Hari Gowtham -
Sahina Bose