Re: OST Network suite is failing on "OSError: [Errno 28] No space left on device"

We are doing nothing special there, just executing ansible through their API. On Sun, Mar 18, 2018 at 10:42 AM, Daniel Belenky <dbelenky(a)redhat.com&gt; wrote: > It's not a space issue. Other suites ran on that slave after your suite > successfully. > I think that the problem is the setting for max semaphores, though I > don't know what you're doing to reach that limit. > > [dbelenky@ovirt-srv18 ~]$ ipcs -ls > > ------ Semaphore Limits -------- > max number of arrays = 128 > max semaphores per array = 250 > max semaphores system wide = 32000 > max ops per semop call = 32 > semaphore max value = 32767 > > > On Sun, Mar 18, 2018 at 10:31 AM, Edward Haas <ehaas(a)redhat.com&gt; wrote: > >> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-master/ >> >> On Sun, Mar 18, 2018 at 10:24 AM, Daniel Belenky <dbelenky(a)redhat.com&gt; >> wrote: >> >>> Hi Edi, >>> >>> Are there any logs? where you're running the suite? may I have a link? >>> >>> On Sun, Mar 18, 2018 at 8:20 AM, Edward Haas <ehaas(a)redhat.com&gt; wrote: >>> >>>> Good morning, >>>> >>>> We are running in the OST network suite a test module with Ansible and >>>> it started failing during the weekend on "OSError: [Errno 28] No space left >>>> on device" when attempting to take a lock in the mutiprocessing python >>>> module. >>>> >>>> It smells like a slave resource problem, could someone help >>>> investigate this? >>>> >>>> Thanks, >>>> Edy. >>>> >>>> =================================== FAILURES =================================== >>>> ______________________ test_ovn_provider_create_scenario _______________________ >>>> >>>> os_client_config = None >>>> >>>> def test_ovn_provider_create_scenario(os_client_config): >>>> > _test_ovn_provider('create_scenario.yml') >>>> >>>> network-suite-master/tests/test_ovn_provider.py:68: >>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >>>> network-suite-master/tests/test_ovn_provider.py:78: in _test_ovn_provider >>>> playbook.run() >>>> network-suite-master/lib/ansiblelib.py:127: in run >>>> self._run_playbook_executor() >>>> network-suite-master/lib/ansiblelib.py:138: in _run_playbook_executor >>>> pbex = PlaybookExecutor(**self._pbex_args) >>>> /usr/lib/python2.7/site-packages/ansible/executor/playbook_executor.py:60: in __init__ >>>> self._tqm = TaskQueueManager(inventory=inventory, variable_manager=variable_manager, loader=loader, options=options, passwords=self.passwords) >>>> /usr/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py:104: in __init__ >>>> self._final_q = multiprocessing.Queue() >>>> /usr/lib64/python2.7/multiprocessing/__init__.py:218: in Queue >>>> return Queue(maxsize) >>>> /usr/lib64/python2.7/multiprocessing/queues.py:63: in __init__ >>>> self._rlock = Lock() >>>> /usr/lib64/python2.7/multiprocessing/synchronize.py:147: in __init__ >>>> SemLock.__init__(self, SEMAPHORE, 1, 1) >>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >>>> >>>> self = <Lock(owner=unknown)>, kind = 1, value = 1, maxvalue = 1 >>>> >>>> def __init__(self, kind, value, maxvalue): >>>> > sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue) >>>> E OSError: [Errno 28] No space left on device >>>> >>>> /usr/lib64/python2.7/multiprocessing/synchronize.py:75: OSError >>>> >>>> >>> >>> >>> -- >>> >>> DANIEL BELENKY >>> >>> RHV DEVOPS >>> >> >> > > > -- > > DANIEL BELENKY > > RHV DEVOPS >