Re: OST Network suite is failing on "OSError: [Errno 28] No space left on device"

Failed also here: http://jenkins.ovirt.org/job/ovirt-system-tests_master_ check-patch-el7-x86_64/4540/ The patch trigerring this affects many suites, and the job failed during ansible-suite-master . On Mon, Mar 19, 2018 at 3:10 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > Gal and Daniel are looking into it, strange its not affecting all suites. > > On Mon, Mar 19, 2018 at 2:11 PM, Dominik Holler <dholler(a)redhat.com&gt; > wrote: > >> Looks like /dev/shm is run out of space. >> >> On Mon, 19 Mar 2018 13:33:28 +0200 >> Leon Goldberg <lgoldber(a)redhat.com&gt; wrote: >> >> > Hey, any updates? >> > >> > On Sun, Mar 18, 2018 at 10:44 AM, Edward Haas <ehaas(a)redhat.com&gt; >> > wrote: >> > >> > > We are doing nothing special there, just executing ansible through >> > > their API. >> > > >> > > On Sun, Mar 18, 2018 at 10:42 AM, Daniel Belenky >> > > <dbelenky(a)redhat.com&gt; wrote: >> > > >> > >> It's not a space issue. Other suites ran on that slave after your >> > >> suite successfully. >> > >> I think that the problem is the setting for max semaphores, though >> > >> I don't know what you're doing to reach that limit. >> > >> >> > >> [dbelenky@ovirt-srv18 ~]$ ipcs -ls >> > >> >> > >> ------ Semaphore Limits -------- >> > >> max number of arrays = 128 >> > >> max semaphores per array = 250 >> > >> max semaphores system wide = 32000 >> > >> max ops per semop call = 32 >> > >> semaphore max value = 32767 >> > >> >> > >> >> > >> On Sun, Mar 18, 2018 at 10:31 AM, Edward Haas <ehaas(a)redhat.com&gt; >> > >> wrote: >> > >>> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suit >> e-master/ >> > >>> >> > >>> On Sun, Mar 18, 2018 at 10:24 AM, Daniel Belenky >> > >>> <dbelenky(a)redhat.com&gt; wrote: >> > >>> >> > >>>> Hi Edi, >> > >>>> >> > >>>> Are there any logs? where you're running the suite? may I have a >> > >>>> link? >> > >>>> >> > >>>> On Sun, Mar 18, 2018 at 8:20 AM, Edward Haas <ehaas(a)redhat.com&gt; >> > >>>> wrote: >> > >>>>> Good morning, >> > >>>>> >> > >>>>> We are running in the OST network suite a test module with >> > >>>>> Ansible and it started failing during the weekend on "OSError: >> > >>>>> [Errno 28] No space left on device" when attempting to take a >> > >>>>> lock in the mutiprocessing python module. >> > >>>>> >> > >>>>> It smells like a slave resource problem, could someone help >> > >>>>> investigate this? >> > >>>>> >> > >>>>> Thanks, >> > >>>>> Edy. >> > >>>>> >> > >>>>> =================================== FAILURES >> > >>>>> =================================== ______________________ >> > >>>>> test_ovn_provider_create_scenario _______________________ >> > >>>>> >> > >>>>> os_client_config = None >> > >>>>> >> > >>>>> def test_ovn_provider_create_scenario(os_client_config): >> > >>>>> > _test_ovn_provider('create_scenario.yml') >> > >>>>> >> > >>>>> network-suite-master/tests/test_ovn_provider.py:68: >> > >>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >> > >>>>> _ _ _ _ _ _ _ _ >> > >>>>> network-suite-master/tests/test_ovn_provider.py:78: in >> > >>>>> _test_ovn_provider playbook.run() >> > >>>>> network-suite-master/lib/ansiblelib.py:127: in run >> > >>>>> self._run_playbook_executor() >> > >>>>> network-suite-master/lib/ansiblelib.py:138: in >> > >>>>> _run_playbook_executor pbex = >> > >>>>> PlaybookExecutor(**self._pbex_args) >> /usr/lib/python2.7/site-packages/ansible/executor/playbook_e >> xecutor.py:60: >> > >>>>> in __init__ self._tqm = TaskQueueManager(inventory=inventory, >> > >>>>> variable_manager=variable_manager, loader=loader, >> > >>>>> options=options, >> > >>>>> passwords=self.passwords) /usr/lib/python2.7/site-packag >> es/ansible/executor/task_queue_manager.py:104: >> > >>>>> in __init__ self._final_q = >> > >>>>> multiprocessing.Queue() /usr/lib64/python2.7/multiproc >> essing/__init__.py:218: >> > >>>>> in Queue return >> > >>>>> Queue(maxsize) /usr/lib64/python2.7/multiproc >> essing/queues.py:63: >> > >>>>> in __init__ self._rlock = >> > >>>>> Lock() /usr/lib64/python2.7/multiprocessing/synchronize.py:147: >> > >>>>> in __init__ SemLock.__init__(self, SEMAPHORE, 1, 1) _ _ _ _ _ _ >> > >>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >> > >>>>> _ _ >> > >>>>> >> > >>>>> self = <Lock(owner=unknown)>, kind = 1, value = 1, maxvalue = 1 >> > >>>>> >> > >>>>> def __init__(self, kind, value, maxvalue): >> > >>>>> > sl = self._semlock = _multiprocessing.SemLock(kind, >> > >>>>> > value, maxvalue) >> > >>>>> E OSError: [Errno 28] No space left on device >> > >>>>> >> > >>>>> /usr/lib64/python2.7/multiprocessing/synchronize.py:75: OSError >> > >>>>> >> > >>>>> >> > >>>> >> > >>>> >> > >>>> -- >> > >>>> >> > >>>> DANIEL BELENKY >> > >>>> >> > >>>> RHV DEVOPS >> > >>>> >> > >>> >> > >>> >> > >> >> > >> >> > >> -- >> > >> >> > >> DANIEL BELENKY >> > >> >> > >> RHV DEVOPS >> > >> >> > > >> > > >> >> _______________________________________________ >> Infra mailing list >> Infra(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/infra >> > > > > -- > > Eyal edri > > > MANAGER > > RHV DevOps > > EMEA VIRTUALIZATION R&D > > > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > phone: +972-9-7692018 <+972%209-769-2018> > irc: eedri (on #tlv #rhev-dev #rhev-integ) > > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > > -- Didi