[ovirt-devel] [ OST Failure Report ] [ oVirt master][09-05-2017][ 002_bootstrap.add_hosts ]

Hello. Today around 11 am the add host for master started to consistently fail. We found the following in /var/log/messages (as I understand it might get there through journald as the host deploy log claims that vdsmd unit startup failed): May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: Traceback (most recent call last): May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: return tool_command[cmd]["command"](*args) May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/network.py", line 48, in upgrade_networks May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: netupgrade.upgrade() May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 55, in upgrade May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: _create_unified_configuration(rconfig, NetInfo(netinfo(vdsmnets))) May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 133, in _create_unified_configuration May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: RunningConfig.store() May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", line 204, in store May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: _store_net_config() May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", line 281, in _store_net_config May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: utils.rmTree(real_old_safeconf_dir) May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 125, in rmTree May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: shutil.rmtree(directoryToRemove) May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib64/python2.7/shutil.py", line 232, in rmtree May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: onerror(os.path.islink, path, sys.exc_info()) May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib64/python2.7/shutil.py", line 230, in rmtree May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: raise OSError("Cannot call rmtree on a symbolic link") May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: OSError: Cannot call rmtree on a symbolic link May 9 07:01:25 lago-basic-suite-master-host1 systemd: vdsm-network.service: control process exited, code=exited status=1 The full logs of the run you can find in Jenkins artifacts: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6589/art... Any vdsm patches merged that might cause this? -- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat