Today around 11 am the add host for master started to consistently fail.

We found the following in /var/log/messages (as I understand it might get there through journald as the host deploy log claims that vdsmd unit startup failed):

May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: Traceback (most recent call last):
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: return tool_command[cmd]["command"](*args)
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/network.py", line 48, in upgrade_networks
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: netupgrade.upgrade()
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 55, in upgrade
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: _create_unified_configuration(rconfig, NetInfo(netinfo(vdsmnets)))
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 133, in _create_unified_configuration
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: RunningConfig.store()
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", line 204, in store
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: _store_net_config()
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", line 281, in _store_net_config
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: utils.rmTree(real_old_safeconf_dir)
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 125, in rmTree
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: shutil.rmtree(directoryToRemove)
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib64/python2.7/shutil.py", line 232, in rmtree
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: onerror(os.path.islink, path, sys.exc_info())
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib64/python2.7/shutil.py", line 230, in rmtree
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: raise OSError("Cannot call rmtree on a symbolic link")
May  9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: OSError: Cannot call rmtree on a symbolic link
May  9 07:01:25 lago-basic-suite-master-host1 systemd: vdsm-network.service: control process exited, code=exited status=1

The full logs of the run you can find in Jenkins artifacts:

http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6589/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/

Any vdsm patches merged that might cause this? 

--