OST Network suite is failing on "OSError: [Errno 28] No space left on device"

[Barak Korren]

On 20 March 2018 at 10:53, Yedidyah Bar David <didi at redhat.com> wrote:
> On Tue, Mar 20, 2018 at 10:11 AM, Barak Korren <bkorren at redhat.com> wrote:
>> On 20 March 2018 at 09:17, Yedidyah Bar David <didi at redhat.com> wrote:
>>> On Mon, Mar 19, 2018 at 6:56 PM, Dominik Holler <dholler at redhat.com> wrote:
>>>> Thanks Gal, I expect the problem is fixed until something eats
>>>> all space in /dev/shm.
>>>> But the usage of /dev/shm is logged in the output, so we would be able
>>>> to detect the problem next time instantly.
>>>>
>>>> From my point of view it would be good to know why /dev/shm was full,
>>>> to prevent this situation in future.
>>>
>>> Gal already wrote below - it was because some build failed to clean up
>>> after itself.
>>>
>>> I don't know about this specific case, but I was told that I am
>>> personally causing such issues by using the 'cancel' button, so I
>>> sadly stopped. Sadly, because our CI system is quite loaded and when I
>>> know that some build is useless, I wish to kill it and save some
>>> load...
>>>
>>> Back to your point, perhaps we should make jobs check /dev/shm when
>>> they _start_, and either alert/fail/whatever if it's not almost free,
>>> or, if we know what we are doing, just remove stuff there? That might
>>> be much easier than fixing things to clean up in end, and/or debugging
>>> why this cleaning failed.
>>
>> Sure thing, patches to:
>>
>>     [jenkins repo]/jobs/confs/shell-scripts/cleanup_slave.sh
>>
>> Are welcome, we often find interesting stuff to add there...
>>
>> If constrained for time, please turn this comment into an orderly RFE in Jira...
>
> Searched for '/dev/shm' and found way too many places to analyze them
> all and add something to cleanup_slave to cover all.

Where did you search?

>
> Pushed this for now:
>
> https://gerrit.ovirt.org/89215
>
>>
>> --
>> Barak Korren
>> RHV DevOps team , RHCE, RHCi
>> Red Hat EMEA
>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>
>
>
> --
> Didi



-- 
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted