OST Network suite is failing on "OSError: [Errno 28] No space left on device"
Gal Ben Haim
gbenhaim at redhat.com
Mon Mar 19 16:44:54 UTC 2018
I see that this failure happens a lot on "ovirt-srv19.phx.ovirt.org
<http://jenkins.ovirt.org/computer/ovirt-srv19.phx.ovirt.org>", and by
different projects that uses ansible.
Not sure it relates, but I've found (and removed) a stale lago environment
in "/dev/shm" that were created by ovirt-system-tests_he-basic-iscsi-suite
-master
<http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_he-basic-iscsi-suite-master/>
.
The stale environment caused the suite to not run in "/dev/shm".
The maximum number of semaphore on both ovirt-srv19.phx.ovirt.org
<http://jenkins.ovirt.org/computer/ovirt-srv19.phx.ovirt.org> and
ovirt-srv23.phx.ovirt.org
<http://jenkins.ovirt.org/computer/ovirt-srv19.phx.ovirt.org> (which run
the ansible suite with success) is 128.
On Mon, Mar 19, 2018 at 3:37 PM, Yedidyah Bar David <didi at redhat.com> wrote:
> Failed also here:
>
> http://jenkins.ovirt.org/job/ovirt-system-tests_master_
> check-patch-el7-x86_64/4540/
>
> The patch trigerring this affects many suites, and the job failed during
> ansible-suite-master .
>
> On Mon, Mar 19, 2018 at 3:10 PM, Eyal Edri <eedri at redhat.com> wrote:
>
>> Gal and Daniel are looking into it, strange its not affecting all suites.
>>
>> On Mon, Mar 19, 2018 at 2:11 PM, Dominik Holler <dholler at redhat.com>
>> wrote:
>>
>>> Looks like /dev/shm is run out of space.
>>>
>>> On Mon, 19 Mar 2018 13:33:28 +0200
>>> Leon Goldberg <lgoldber at redhat.com> wrote:
>>>
>>> > Hey, any updates?
>>> >
>>> > On Sun, Mar 18, 2018 at 10:44 AM, Edward Haas <ehaas at redhat.com>
>>> > wrote:
>>> >
>>> > > We are doing nothing special there, just executing ansible through
>>> > > their API.
>>> > >
>>> > > On Sun, Mar 18, 2018 at 10:42 AM, Daniel Belenky
>>> > > <dbelenky at redhat.com> wrote:
>>> > >
>>> > >> It's not a space issue. Other suites ran on that slave after your
>>> > >> suite successfully.
>>> > >> I think that the problem is the setting for max semaphores, though
>>> > >> I don't know what you're doing to reach that limit.
>>> > >>
>>> > >> [dbelenky at ovirt-srv18 ~]$ ipcs -ls
>>> > >>
>>> > >> ------ Semaphore Limits --------
>>> > >> max number of arrays = 128
>>> > >> max semaphores per array = 250
>>> > >> max semaphores system wide = 32000
>>> > >> max ops per semop call = 32
>>> > >> semaphore max value = 32767
>>> > >>
>>> > >>
>>> > >> On Sun, Mar 18, 2018 at 10:31 AM, Edward Haas <ehaas at redhat.com>
>>> > >> wrote:
>>> > >>> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suit
>>> e-master/
>>> > >>>
>>> > >>> On Sun, Mar 18, 2018 at 10:24 AM, Daniel Belenky
>>> > >>> <dbelenky at redhat.com> wrote:
>>> > >>>
>>> > >>>> Hi Edi,
>>> > >>>>
>>> > >>>> Are there any logs? where you're running the suite? may I have a
>>> > >>>> link?
>>> > >>>>
>>> > >>>> On Sun, Mar 18, 2018 at 8:20 AM, Edward Haas <ehaas at redhat.com>
>>> > >>>> wrote:
>>> > >>>>> Good morning,
>>> > >>>>>
>>> > >>>>> We are running in the OST network suite a test module with
>>> > >>>>> Ansible and it started failing during the weekend on "OSError:
>>> > >>>>> [Errno 28] No space left on device" when attempting to take a
>>> > >>>>> lock in the mutiprocessing python module.
>>> > >>>>>
>>> > >>>>> It smells like a slave resource problem, could someone help
>>> > >>>>> investigate this?
>>> > >>>>>
>>> > >>>>> Thanks,
>>> > >>>>> Edy.
>>> > >>>>>
>>> > >>>>> =================================== FAILURES
>>> > >>>>> =================================== ______________________
>>> > >>>>> test_ovn_provider_create_scenario _______________________
>>> > >>>>>
>>> > >>>>> os_client_config = None
>>> > >>>>>
>>> > >>>>> def test_ovn_provider_create_scenario(os_client_config):
>>> > >>>>> > _test_ovn_provider('create_scenario.yml')
>>> > >>>>>
>>> > >>>>> network-suite-master/tests/test_ovn_provider.py:68:
>>> > >>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>> > >>>>> _ _ _ _ _ _ _ _
>>> > >>>>> network-suite-master/tests/test_ovn_provider.py:78: in
>>> > >>>>> _test_ovn_provider playbook.run()
>>> > >>>>> network-suite-master/lib/ansiblelib.py:127: in run
>>> > >>>>> self._run_playbook_executor()
>>> > >>>>> network-suite-master/lib/ansiblelib.py:138: in
>>> > >>>>> _run_playbook_executor pbex =
>>> > >>>>> PlaybookExecutor(**self._pbex_args)
>>> /usr/lib/python2.7/site-packages/ansible/executor/playbook_e
>>> xecutor.py:60:
>>> > >>>>> in __init__ self._tqm = TaskQueueManager(inventory=inventory,
>>> > >>>>> variable_manager=variable_manager, loader=loader,
>>> > >>>>> options=options,
>>> > >>>>> passwords=self.passwords) /usr/lib/python2.7/site-packag
>>> es/ansible/executor/task_queue_manager.py:104:
>>> > >>>>> in __init__ self._final_q =
>>> > >>>>> multiprocessing.Queue() /usr/lib64/python2.7/multiproc
>>> essing/__init__.py:218:
>>> > >>>>> in Queue return
>>> > >>>>> Queue(maxsize) /usr/lib64/python2.7/multiproc
>>> essing/queues.py:63:
>>> > >>>>> in __init__ self._rlock =
>>> > >>>>> Lock() /usr/lib64/python2.7/multiprocessing/synchronize.py:147:
>>> > >>>>> in __init__ SemLock.__init__(self, SEMAPHORE, 1, 1) _ _ _ _ _ _
>>> > >>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>> > >>>>> _ _
>>> > >>>>>
>>> > >>>>> self = <Lock(owner=unknown)>, kind = 1, value = 1, maxvalue = 1
>>> > >>>>>
>>> > >>>>> def __init__(self, kind, value, maxvalue):
>>> > >>>>> > sl = self._semlock = _multiprocessing.SemLock(kind,
>>> > >>>>> > value, maxvalue)
>>> > >>>>> E OSError: [Errno 28] No space left on device
>>> > >>>>>
>>> > >>>>> /usr/lib64/python2.7/multiprocessing/synchronize.py:75: OSError
>>> > >>>>>
>>> > >>>>>
>>> > >>>>
>>> > >>>>
>>> > >>>> --
>>> > >>>>
>>> > >>>> DANIEL BELENKY
>>> > >>>>
>>> > >>>> RHV DEVOPS
>>> > >>>>
>>> > >>>
>>> > >>>
>>> > >>
>>> > >>
>>> > >> --
>>> > >>
>>> > >> DANIEL BELENKY
>>> > >>
>>> > >> RHV DEVOPS
>>> > >>
>>> > >
>>> > >
>>>
>>> _______________________________________________
>>> Infra mailing list
>>> Infra at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>
>>
>>
>>
>> --
>>
>> Eyal edri
>>
>>
>> MANAGER
>>
>> RHV DevOps
>>
>> EMEA VIRTUALIZATION R&D
>>
>>
>> Red Hat EMEA <https://www.redhat.com/>
>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>> phone: +972-9-7692018 <+972%209-769-2018>
>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>> _______________________________________________
>> Infra mailing list
>> Infra at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>>
>
>
> --
> Didi
>
--
*GAL bEN HAIM*
RHV DEVOPS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20180319/fcc88938/attachment.html>
More information about the Infra
mailing list