Fwd: ** PROBLEM Service Alert: Resources file server/Resources Partition is WARNING **
Can you check what is causing it? we should have resolved space issue by applying retention script and dropping 4.1 ---------- Forwarded message --------- From: <nagios@monitoring.ovirt.org> Date: Mon, Sep 3, 2018 at 2:42 PM Subject: ** PROBLEM Service Alert: Resources file server/Resources Partition is WARNING ** To: <eedri@redhat.com> ***** Nagios ***** Notification Type: PROBLEM Service: Resources Partition Host: Resources file server Address: resources.ovirt.org State: WARNING Date/Time: Mon Sep 3 11:33:23 UTC 2018 Additional Info: DISK WARNING - free space: /srv/resources 166334 MB (14.46% inode=100%): -- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
Checking On Mon, Sep 3, 2018 at 3:06 PM Eyal Edri <eedri@redhat.com> wrote:
Can you check what is causing it? we should have resolved space issue by applying retention script and dropping 4.1
---------- Forwarded message --------- From: <nagios@monitoring.ovirt.org> Date: Mon, Sep 3, 2018 at 2:42 PM Subject: ** PROBLEM Service Alert: Resources file server/Resources Partition is WARNING ** To: <eedri@redhat.com>
***** Nagios *****
Notification Type: PROBLEM
Service: Resources Partition Host: Resources file server Address: resources.ovirt.org State: WARNING
Date/Time: Mon Sep 3 11:33:23 UTC 2018
Additional Info:
DISK WARNING - free space: /srv/resources 166334 MB (14.46% inode=100%):
--
Eyal edri
MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
2018-09-03 14:07 GMT+02:00 Ehud Yonasi <eyonasi@redhat.com>:
Checking
We are releasing 4.2.6 GA today, increase in usage may have been caused by backup copy of the release repo while updating it. I would consider increasing the inodes availability on that file system.
On Mon, Sep 3, 2018 at 3:06 PM Eyal Edri <eedri@redhat.com> wrote:
Can you check what is causing it? we should have resolved space issue by applying retention script and dropping 4.1
---------- Forwarded message --------- From: <nagios@monitoring.ovirt.org> Date: Mon, Sep 3, 2018 at 2:42 PM Subject: ** PROBLEM Service Alert: Resources file server/Resources Partition is WARNING ** To: <eedri@redhat.com>
***** Nagios *****
Notification Type: PROBLEM
Service: Resources Partition Host: Resources file server Address: resources.ovirt.org State: WARNING
Date/Time: Mon Sep 3 11:33:23 UTC 2018
Additional Info:
DISK WARNING - free space: /srv/resources 166334 MB (14.46% inode=100%):
--
Eyal edri
MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ message/C3YOLSRLODL47JIKDCD2SLBGYKJ4TWRC/
-- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f2000000RgRyAAK>
On Mon, Sep 3, 2018 at 3:12 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2018-09-03 14:07 GMT+02:00 Ehud Yonasi <eyonasi@redhat.com>:
Checking
We are releasing 4.2.6 GA today, increase in usage may have been caused by backup copy of the release repo while updating it. I would consider increasing the inodes availability on that file system.
Yea, this is in plan, we now have more storage availalble, there are talks about splitting the official builds and CI builds to be on different servers. Evgheni, is there additional available space on current storage we can allocate for resources?
On Mon, Sep 3, 2018 at 3:06 PM Eyal Edri <eedri@redhat.com> wrote:
Can you check what is causing it? we should have resolved space issue by applying retention script and dropping 4.1
---------- Forwarded message --------- From: <nagios@monitoring.ovirt.org> Date: Mon, Sep 3, 2018 at 2:42 PM Subject: ** PROBLEM Service Alert: Resources file server/Resources Partition is WARNING ** To: <eedri@redhat.com>
***** Nagios *****
Notification Type: PROBLEM
Service: Resources Partition Host: Resources file server Address: resources.ovirt.org State: WARNING
Date/Time: Mon Sep 3 11:33:23 UTC 2018
Additional Info:
DISK WARNING - free space: /srv/resources 166334 MB (14.46% inode=100%):
--
Eyal edri
MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/C3YOLSRLODL47J...
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f2000000RgRyAAK>
-- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
2018-09-03 14:20 GMT+02:00 Eyal Edri <eedri@redhat.com>:
On Mon, Sep 3, 2018 at 3:12 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2018-09-03 14:07 GMT+02:00 Ehud Yonasi <eyonasi@redhat.com>:
Checking
We are releasing 4.2.6 GA today, increase in usage may have been caused by backup copy of the release repo while updating it. I would consider increasing the inodes availability on that file system.
Yea, this is in plan, we now have more storage availalble, there are talks about splitting the official builds and CI builds to be on different servers.
Evgheni, is there additional available space on current storage we can allocate for resources?
There's plenty of disk space, only inodes are running out
On Mon, Sep 3, 2018 at 3:06 PM Eyal Edri <eedri@redhat.com> wrote:
Can you check what is causing it? we should have resolved space issue by applying retention script and dropping 4.1
---------- Forwarded message --------- From: <nagios@monitoring.ovirt.org> Date: Mon, Sep 3, 2018 at 2:42 PM Subject: ** PROBLEM Service Alert: Resources file server/Resources Partition is WARNING ** To: <eedri@redhat.com>
***** Nagios *****
Notification Type: PROBLEM
Service: Resources Partition Host: Resources file server Address: resources.ovirt.org State: WARNING
Date/Time: Mon Sep 3 11:33:23 UTC 2018
Additional Info:
DISK WARNING - free space: /srv/resources 166334 MB (14.46% inode=100%):
--
Eyal edri
MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ message/C3YOLSRLODL47JIKDCD2SLBGYKJ4TWRC/
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f2000000RgRyAAK>
--
Eyal edri
MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f2000000RgRyAAK>
First of all I'd like to clear things out with regards to the nagios status message: DISK OK - free space: /srv/resources 183768 MB (15.98% inode=100%): This shows free space and percentage, so 100% inode space is free and ~16% disk space is free. Currently this is around the threshold of 15% so we're seeing warnings. The free space situation has improved a lot since the tested retention patch was merged. It, however, only cleans up RPMs which results in node ISOs filling up their respective directory: # du -sh master/* 195G master/iso 135G master/rpm Also, the node ISO directory has corrupted directory names which is a known bug(OVIRT-2355) and in total there's more than a hundred ISOs stored inside: # du -sh master/iso/* 56G master/iso/ovirt-node-ng-installer 1.5G master/iso/ovirt-node-ng-installer-j-fc 9.6G master/iso/ovirt-node-ng-installer-master- 65G master/iso/ovirt-node-ng-installer-master-e 62G master/iso/ovirt-node-ng-installer-master-fc 3.0G master/iso/oVirt-toolsSetup We'll need to fix naming here and then create a retention script as well. For now I've kept the last 5 builds in each of these directories and that cleared 200GB (17% more free space). I believe this is the major factor contributing to disk usage at this point as each node ISO is 1.2GB in size. We can add disk space but there's not much point if it's going to be used for stale ISOs in misnamed directories. On Mon, Sep 3, 2018 at 2:23 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2018-09-03 14:20 GMT+02:00 Eyal Edri <eedri@redhat.com>:
On Mon, Sep 3, 2018 at 3:12 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2018-09-03 14:07 GMT+02:00 Ehud Yonasi <eyonasi@redhat.com>:
Checking
We are releasing 4.2.6 GA today, increase in usage may have been caused by backup copy of the release repo while updating it. I would consider increasing the inodes availability on that file system.
Yea, this is in plan, we now have more storage availalble, there are talks about splitting the official builds and CI builds to be on different servers.
Evgheni, is there additional available space on current storage we can allocate for resources?
There's plenty of disk space, only inodes are running out
On Mon, Sep 3, 2018 at 3:06 PM Eyal Edri <eedri@redhat.com> wrote:
Can you check what is causing it? we should have resolved space issue by applying retention script and dropping 4.1
---------- Forwarded message --------- From: <nagios@monitoring.ovirt.org> Date: Mon, Sep 3, 2018 at 2:42 PM Subject: ** PROBLEM Service Alert: Resources file server/Resources Partition is WARNING ** To: <eedri@redhat.com>
***** Nagios *****
Notification Type: PROBLEM
Service: Resources Partition Host: Resources file server Address: resources.ovirt.org State: WARNING
Date/Time: Mon Sep 3 11:33:23 UTC 2018
Additional Info:
DISK WARNING - free space: /srv/resources 166334 MB (14.46% inode=100%):
--
Eyal edri
MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/C3YOLSRLODL47J...
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f2000000RgRyAAK>
--
Eyal edri
MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f2000000RgRyAAK>
-- Regards, Evgheni Dereveanchin
participants (4)
-
Ehud Yonasi -
Evgheni Dereveanchin -
Eyal Edri -
Sandro Bonazzola