OST Network suite is failing on "OSError: [Errno 28] No space left on device"

[Yedidyah Bar David]

On Tue, Mar 20, 2018 at 10:11 AM, Barak Korren <bkorren at redhat.com> wrote:
> On 20 March 2018 at 09:17, Yedidyah Bar David <didi at redhat.com> wrote:
>> On Mon, Mar 19, 2018 at 6:56 PM, Dominik Holler <dholler at redhat.com> wrote:
>>> Thanks Gal, I expect the problem is fixed until something eats
>>> all space in /dev/shm.
>>> But the usage of /dev/shm is logged in the output, so we would be able
>>> to detect the problem next time instantly.
>>>
>>> From my point of view it would be good to know why /dev/shm was full,
>>> to prevent this situation in future.
>>
>> Gal already wrote below - it was because some build failed to clean up
>> after itself.
>>
>> I don't know about this specific case, but I was told that I am
>> personally causing such issues by using the 'cancel' button, so I
>> sadly stopped. Sadly, because our CI system is quite loaded and when I
>> know that some build is useless, I wish to kill it and save some
>> load...
>>
>> Back to your point, perhaps we should make jobs check /dev/shm when
>> they _start_, and either alert/fail/whatever if it's not almost free,
>> or, if we know what we are doing, just remove stuff there? That might
>> be much easier than fixing things to clean up in end, and/or debugging
>> why this cleaning failed.
>
> Sure thing, patches to:
>
>     [jenkins repo]/jobs/confs/shell-scripts/cleanup_slave.sh
>
> Are welcome, we often find interesting stuff to add there...
>
> If constrained for time, please turn this comment into an orderly RFE in Jira...

Searched for '/dev/shm' and found way too many places to analyze them
all and add something to cleanup_slave to cover all.

Pushed this for now:

https://gerrit.ovirt.org/89215

>
> --
> Barak Korren
> RHV DevOps team , RHCE, RHCi
> Red Hat EMEA
> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted



-- 
Didi