2:40 p.m.
Hello.
Today around 11 am the add host for master started to consistently fail.
We found the following in /var/log/messages (as I understand it might get
there through journald as the host deploy log claims that vdsmd unit
startup failed):
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: Traceback
(most recent call last):
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File
"/usr/bin/vdsm-tool", line 219, in main
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: return
tool_command[cmd]["command"](*args)
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/tool/network.py", line 48, in
upgrade_networks
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: netupgrade.upgrade()
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line
55, in upgrade
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool:
_create_unified_configuration(rconfig, NetInfo(netinfo(vdsmnets)))
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line
133, in _create_unified_configuration
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: RunningConfig.store()
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py",
line 204, in store
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: _store_net_config()
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py",
line 281, in _store_net_config
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool:
utils.rmTree(real_old_safeconf_dir)
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/utils.py", line 125, in rmTree
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool:
shutil.rmtree(directoryToRemove)
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File
"/usr/lib64/python2.7/shutil.py", line 232, in rmtree
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool:
onerror(os.path.islink, path, sys.exc_info())
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File
"/usr/lib64/python2.7/shutil.py", line 230, in rmtree
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: raise
OSError("Cannot call rmtree on a symbolic link")
May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: OSError:
Cannot call rmtree on a symbolic link
May 9 07:01:25 lago-basic-suite-master-host1 systemd:
vdsm-network.service: control process exited, code=exited status=1
The full logs of the run you can find in Jenkins artifacts:
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6589/art...
Any vdsm patches merged that might cause this?
--
Anton Marchukov
Senior Software Engineer - RHEV CI - Red Hat