Hello. Today around 11 am the add host for master started to consistently fail. We found the following in /var/log/messages (as I understand it might get there through journald as the host deploy log claims that vdsmd unit startup failed): May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: Traceback (most recent call last): May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: return tool_command[cmd]["command"](*args) May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/network.py", line 48, in upgrade_networks May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: netupgrade.upgrade() May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 55, in upgrade May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: _create_unified_configuration(rconfig, NetInfo(netinfo(vdsmnets))) May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 133, in _create_unified_configuration May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: RunningConfig.store() May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", line 204, in store May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: _store_net_config() May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", line 281, in _store_net_config May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: utils.rmTree(real_old_safeconf_dir) May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 125, in rmTree May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: shutil.rmtree(directoryToRemove) May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib64/python2.7/shutil.py", line 232, in rmtree May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: onerror(os.path.islink, path, sys.exc_info()) May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File "/usr/lib64/python2.7/shutil.py", line 230, in rmtree May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: raise OSError("Cannot call rmtree on a symbolic link") May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: OSError: Cannot call rmtree on a symbolic link May 9 07:01:25 lago-basic-suite-master-host1 systemd: vdsm-network.service: control process exited, code=exited status=1 The full logs of the run you can find in Jenkins artifacts: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6589/art... Any vdsm patches merged that might cause this? -- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

On Tue, May 9, 2017 at 3:44 PM, Piotr Kliczewski <piotr.kliczewski(a)gmail.com&gt; wrote: > It seems to be broken by this patch [1] > > > [1] https://gerrit.ovirt.org/#/c/76603 Indeed. Please revert for now [1], then investigate. [1] https://gerrit.ovirt.org/76626 > > On Tue, May 9, 2017 at 2:40 PM, Anton Marchukov <amarchuk(a)redhat.com&gt; wrote: >> Hello. >> >> Today around 11 am the add host for master started to consistently fail. >> >> We found the following in /var/log/messages (as I understand it might get >> there through journald as the host deploy log claims that vdsmd unit startup >> failed): >> >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: Traceback (most >> recent call last): >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File >> "/usr/bin/vdsm-tool", line 219, in main >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: return >> tool_command[cmd]["command"](*args) >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File >> "/usr/lib/python2.7/site-packages/vdsm/tool/network.py", line 48, in >> upgrade_networks >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: >> netupgrade.upgrade() >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File >> "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 55, in >> upgrade >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: >> _create_unified_configuration(rconfig, NetInfo(netinfo(vdsmnets))) >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File >> "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line 133, in >> _create_unified_configuration >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: >> RunningConfig.store() >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File >> "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", line >> 204, in store >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: _store_net_config() >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File >> "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", line >> 281, in _store_net_config >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: >> utils.rmTree(real_old_safeconf_dir) >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File >> "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 125, in rmTree >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: >> shutil.rmtree(directoryToRemove) >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File >> "/usr/lib64/python2.7/shutil.py", line 232, in rmtree >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: >> onerror(os.path.islink, path, sys.exc_info()) >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File >> "/usr/lib64/python2.7/shutil.py", line 230, in rmtree >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: raise >> OSError("Cannot call rmtree on a symbolic link") >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: OSError: Cannot >> call rmtree on a symbolic link >> May 9 07:01:25 lago-basic-suite-master-host1 systemd: vdsm-network.service: >> control process exited, code=exited status=1 >> >> >> The full logs of the run you can find in Jenkins artifacts: >> >> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/6589/artifact/exported-artifacts/basic-suit-master- el7/test_logs/basic-suite-master/post-002_bootstrap.py/ >> >> >> Any vdsm patches merged that might cause this? >> >> -- >> >> Anton Marchukov >> Senior Software Engineer - RHEV CI - Red Hat >> >> >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel -- Didi

CC-ing Edi. On Tue, May 9, 2017 at 3:46 PM, Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > On Tue, May 9, 2017 at 3:44 PM, Piotr Kliczewski > <piotr.kliczewski(a)gmail.com&gt; wrote: > > It seems to be broken by this patch [1] > > > > > > [1] https://gerrit.ovirt.org/#/c/76603 > > Indeed. Please revert for now [1], then investigate. > > [1] https://gerrit.ovirt.org/76626 > > > > > On Tue, May 9, 2017 at 2:40 PM, Anton Marchukov <amarchuk(a)redhat.com&gt; > wrote: > >> Hello. > >> > >> Today around 11 am the add host for master started to consistently > fail. > >> > >> We found the following in /var/log/messages (as I understand it might > get > >> there through journald as the host deploy log claims that vdsmd unit > startup > >> failed): > >> > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: Traceback > (most > >> recent call last): > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File > >> "/usr/bin/vdsm-tool", line 219, in main > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: return > >> tool_command[cmd]["command"](*args) > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File > >> "/usr/lib/python2.7/site-packages/vdsm/tool/network.py", line 48, in > >> upgrade_networks > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: > >> netupgrade.upgrade() > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File > >> "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line > 55, in > >> upgrade > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: > >> _create_unified_configuration(rconfig, NetInfo(netinfo(vdsmnets))) > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File > >> "/usr/lib/python2.7/site-packages/vdsm/network/netupgrade.py", line > 133, in > >> _create_unified_configuration > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: > >> RunningConfig.store() > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File > >> "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", > line > >> 204, in store > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: > _store_net_config() > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File > >> "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", > line > >> 281, in _store_net_config > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: > >> utils.rmTree(real_old_safeconf_dir) > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File > >> "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 125, in rmTree > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: > >> shutil.rmtree(directoryToRemove) > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File > >> "/usr/lib64/python2.7/shutil.py", line 232, in rmtree > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: > >> onerror(os.path.islink, path, sys.exc_info()) > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: File > >> "/usr/lib64/python2.7/shutil.py", line 230, in rmtree > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: raise > >> OSError("Cannot call rmtree on a symbolic link") > >> May 9 07:01:25 lago-basic-suite-master-host1 vdsm-tool: OSError: > Cannot > >> call rmtree on a symbolic link > >> May 9 07:01:25 lago-basic-suite-master-host1 systemd: > vdsm-network.service: > >> control process exited, code=exited status=1 > >> > >> > >> The full logs of the run you can find in Jenkins artifacts: > >> > >> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma > ster/6589/artifact/exported-artifacts/basic-suit-master-el7/ > test_logs/basic-suite-master/post-002_bootstrap.py/ > >> > >> > >> Any vdsm patches merged that might cause this? > >> > >> -- > >> > >> Anton Marchukov > >> Senior Software Engineer - RHEV CI - Red Hat > >> > >> > >> _______________________________________________ > >> Devel mailing list > >> Devel(a)ovirt.org > >> http://lists.ovirt.org/mailman/listinfo/devel > > _______________________________________________ > > Devel mailing list > > Devel(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/devel > > > > -- > Didi > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel > > >