First of all I'd like to clear things out with regards to the nagios status
message:
DISK OK - free space: /srv/resources 183768 MB (15.98% inode=100%):
This shows free space and percentage, so 100% inode space is free and ~16%
disk space is free.
Currently this is around the threshold of 15% so we're seeing warnings.
The free space situation has improved a lot since the tested retention
patch was merged.
It, however, only cleans up RPMs which results in node ISOs filling up
their respective directory:
# du -sh master/*
195G master/iso
135G master/rpm
Also, the node ISO directory has corrupted directory names which is a known
bug(OVIRT-2355)
and in total there's more than a hundred ISOs stored inside:
# du -sh master/iso/*
56G master/iso/ovirt-node-ng-installer
1.5G master/iso/ovirt-node-ng-installer-j-fc
9.6G master/iso/ovirt-node-ng-installer-master-
65G master/iso/ovirt-node-ng-installer-master-e
62G master/iso/ovirt-node-ng-installer-master-fc
3.0G master/iso/oVirt-toolsSetup
We'll need to fix naming here and then create a retention script as well.
For now I've kept the last 5 builds in each of these directories and that
cleared 200GB (17% more free space).
I believe this is the major factor contributing to disk usage at this point
as each node ISO is 1.2GB in size.
We can add disk space but there's not much point if it's going to be used
for stale ISOs in misnamed directories.
On Mon, Sep 3, 2018 at 2:23 PM Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
2018-09-03 14:20 GMT+02:00 Eyal Edri <eedri(a)redhat.com>:
>
>
> On Mon, Sep 3, 2018 at 3:12 PM Sandro Bonazzola <sbonazzo(a)redhat.com>
> wrote:
>
>>
>>
>> 2018-09-03 14:07 GMT+02:00 Ehud Yonasi <eyonasi(a)redhat.com>:
>>
>>> Checking
>>>
>>
>> We are releasing 4.2.6 GA today, increase in usage may have been caused
>> by backup copy of the release repo while updating it.
>> I would consider increasing the inodes availability on that file system.
>>
>
> Yea, this is in plan, we now have more storage availalble, there are
> talks about splitting the official builds and CI builds to be on different
> servers.
>
> Evgheni, is there additional available space on current storage we can
> allocate for resources?
>
There's plenty of disk space, only inodes are running out
>
>
>>
>>
>>
>>>
>>> On Mon, Sep 3, 2018 at 3:06 PM Eyal Edri <eedri(a)redhat.com> wrote:
>>>
>>>> Can you check what is causing it? we should have resolved space issue
>>>> by applying retention script and dropping 4.1
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: <nagios(a)monitoring.ovirt.org>
>>>> Date: Mon, Sep 3, 2018 at 2:42 PM
>>>> Subject: ** PROBLEM Service Alert: Resources file server/Resources
>>>> Partition is WARNING **
>>>> To: <eedri(a)redhat.com>
>>>>
>>>>
>>>> ***** Nagios *****
>>>>
>>>> Notification Type: PROBLEM
>>>>
>>>> Service: Resources Partition
>>>> Host: Resources file server
>>>> Address:
resources.ovirt.org
>>>> State: WARNING
>>>>
>>>> Date/Time: Mon Sep 3 11:33:23 UTC 2018
>>>>
>>>> Additional Info:
>>>>
>>>> DISK WARNING - free space: /srv/resources 166334 MB (14.46%
>>>> inode=100%):
>>>>
>>>>
>>>> --
>>>>
>>>> Eyal edri
>>>>
>>>>
>>>> MANAGER
>>>>
>>>> RHV/CNV DevOps
>>>>
>>>> EMEA VIRTUALIZATION R&D
>>>>
>>>>
>>>> Red Hat EMEA <
https://www.redhat.com/>
>>>> <
https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>> <
https://redhat.com/trusted>
>>>> phone: +972-9-7692018 <+972%209-769-2018>
>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>
>>>
>>> _______________________________________________
>>> Infra mailing list -- infra(a)ovirt.org
>>> To unsubscribe send an email to infra-leave(a)ovirt.org
>>> Privacy Statement:
https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct:
>>>
https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>>
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/C3YOLSRLODL...
>>>
>>>
>>
>>
>> --
>>
>> SANDRO BONAZZOLA
>>
>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>>
>> Red Hat EMEA <
https://www.redhat.com/>
>>
>> sbonazzo(a)redhat.com
>> <
https://red.ht/sig>
>>
<
https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=70...
>>
>
>
> --
>
> Eyal edri
>
>
> MANAGER
>
> RHV/CNV DevOps
>
> EMEA VIRTUALIZATION R&D
>
>
> Red Hat EMEA <
https://www.redhat.com/>
> <
https://red.ht/sig> TRIED. TESTED. TRUSTED.
<
https://redhat.com/trusted>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <
https://www.redhat.com/>
sbonazzo(a)redhat.com
<
https://red.ht/sig>
<
https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=70...