<div dir="ltr">We are doing nothing special there, just executing ansible through their API.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Mar 18, 2018 at 10:42 AM, Daniel Belenky <span dir="ltr"><<a href="mailto:dbelenky@redhat.com" target="_blank">dbelenky@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>It's not a space issue. Other suites ran on that slave after your suite successfully.<br></div>I think that the problem is the setting for max semaphores, though I don't know what you're doing to reach that limit.<br><span style="font-family:monospace,monospace"><br>[dbelenky@ovirt-srv18 ~]$ ipcs -ls<br><br>------ Semaphore Limits --------<br>max number of arrays = 128<br>max semaphores per array = 250<br>max semaphores system wide = 32000<br>max ops per semop call = 32<br>semaphore max value = 32767<br></span></div><span style="font-family:monospace,monospace"></span><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Mar 18, 2018 at 10:31 AM, Edward Haas <span dir="ltr"><<a href="mailto:ehaas@redhat.com" target="_blank">ehaas@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><a href="http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-master/" target="_blank">http://jenkins.ovirt.org/job/o<wbr>virt-system-tests_network-suit<wbr>e-master/</a><br></div><div class="m_-8306623841702917503HOEnZb"><div class="m_-8306623841702917503h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Mar 18, 2018 at 10:24 AM, Daniel Belenky <span dir="ltr"><<a href="mailto:dbelenky@redhat.com" target="_blank">dbelenky@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Edi,<br><br></div>Are there any logs? where you're running the suite? may I have a link?<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Mar 18, 2018 at 8:20 AM, Edward Haas <span dir="ltr"><<a href="mailto:ehaas@redhat.com" target="_blank">ehaas@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Good morning,<br><br></div>We are running in the OST network suite a test module with Ansible and it started failing during the weekend on "OSError: [Errno 28] No space left on device" when attempting to take a lock in the mutiprocessing python module.<br><br></div>It smells like a slave resource problem, could someone help investigate this?<br><br></div>Thanks,<br></div><div><div class="m_-8306623841702917503m_2440596000458496649h5"><div><div class="m_-8306623841702917503m_2440596000458496649m_-8775056096712648942h5">Edy.<br><div><div><div><br><pre>==============================<wbr>===== FAILURES ==============================<wbr>=====
______________________ test_ovn_provider_create_scena<wbr>rio _______________________
os_client_config = None
def test_ovn_provider_create_scena<wbr>rio(os_client_config):
> _test_ovn_provider('create_sce<wbr>nario.yml')
network-suite-master/tests/tes<wbr>t_ovn_provider.py:68:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
network-suite-master/tests/tes<wbr>t_ovn_provider.py:78: in _test_ovn_provider
playbook.run()
network-suite-master/lib/ansib<wbr>lelib.py:127: in run
self._run_playbook_executor()
network-suite-master/lib/ansib<wbr>lelib.py:138: in _run_playbook_executor
pbex = PlaybookExecutor(**self._pbex_<wbr>args)
/usr/lib/python2.7/site-packag<wbr>es/ansible/executor/playbook_e<wbr>xecutor.py:60: in __init__
self._tqm = TaskQueueManager(inventory=inv<wbr>entory, variable_manager=variable_mana<wbr>ger, loader=loader, options=options, passwords=self.passwords)
/usr/lib/python2.7/site-packag<wbr>es/ansible/executor/task_queue<wbr>_manager.py:104: in __init__
self._final_q = multiprocessing.Queue()
/usr/lib64/python2.7/multiproc<wbr>essing/__init__.py:218: in Queue
return Queue(maxsize)
/usr/lib64/python2.7/multiproc<wbr>essing/queues.py:63: in __init__
self._rlock = Lock()
/usr/lib64/python2.7/multiproc<wbr>essing/synchronize.py:147: in __init__
SemLock.__init__(self, SEMAPHORE, 1, 1)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <Lock(owner=unknown)>, kind = 1, value = 1, maxvalue = 1
def __init__(self, kind, value, maxvalue):
> sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
E OSError: [Errno 28] No space left on device
/usr/lib64/python2.7/multiproc<wbr>essing/synchronize.py:75: OSError</pre></div></div></div></div></div></div></div></div><span class="m_-8306623841702917503m_2440596000458496649HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="m_-8306623841702917503m_2440596000458496649HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_-8306623841702917503m_2440596000458496649m_-8775056096712648942gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><span><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>DANIEL</span> <span>BELENKY</span><br></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span style="font-weight:normal">RHV DEVOPS</span><br></p></span></div></div></div></div></div>
</font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="m_-8306623841702917503gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><span><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>DANIEL</span> <span>BELENKY</span><br></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span style="font-weight:normal">RHV DEVOPS</span><br></p></span></div></div></div></div></div>
</div>
</div></div></blockquote></div><br></div>