<div dir="ltr">Hello.<div><br></div><div>Today around 11 am the add host for master started to consistently fail.</div><div><br></div><div>We found the following in /var/log/messages (as I understand it might get there through journald as the host deploy log claims that vdsmd unit startup failed):</div><div><br></div><div><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: Traceback (most recent call last):
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: return tool_command[cmd]["command"](*args)
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/network.py", line 48, in upgrade_networks
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: netupgrade.upgrade()
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 55, in upgrade
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: _create_unified_configuration(rconfig, NetInfo(netinfo(vdsmnets)))
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 133, in _create_unified_configuration
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: RunningConfig.store()
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", line 204, in store
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: _store_net_config()
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", line 281, in _store_net_config
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: utils.rmTree(real_old_safeconf_dir)
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 125, in rmTree
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: shutil.rmtree(directoryToRemove)
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib64/python2.7/shutil.py", line 232, in rmtree
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: onerror(os.path.islink, path, sys.exc_info())
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib64/python2.7/shutil.py", line 230, in rmtree
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: raise OSError("Cannot call rmtree on a symbolic link")
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: OSError: Cannot call rmtree on a symbolic link
May 9 07:01:25 lago-basic-suite-master-host1 systemd: vdsm-network.service: control process exited, code=exited status=1</pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"><br></pre>The full logs of the run you can find in Jenkins artifacts:<pre style="word-wrap:break-word"><font color="#000000"><span style="white-space:pre-wrap"><a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6589/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/">http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6589/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/</a><br></span></font></pre><pre style="word-wrap:break-word"><br></pre><pre style="word-wrap:break-word">Any vdsm patches merged that might cause this? </pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">-- <br></pre><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><font color="#888888"><div>Anton Marchukov<br>Senior Software Engineer - <span><span><font color="#888888"><span><span><font color="#888888"><span><span><font color="#888888">RHEV CI - </font></span></span></font></span></span>Red Hat</font></span></span><br><br></div></font></span></div></div></div></div>
</div></div>