On Fri, May 13, 2016 at 3:53 PM, Charles Tassell <ctassell@gmail.com> wrote:
Hi Gervais,
Okay, I see two problems: there are some leftover direcyories causing issues and for some reason VDSM seems to be trying to bind to a port something is already running on (probably an older version of VDSM.) Try removing the duplicate dirs (rmdir /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286 and /rhev/data-center/mnt - if they aren't empty don't rm -rf them because they might be mounted from your production servers. Just mv -i them to /root or somewhere.)
You should not touch directories under /run/vdsm/storage or /rhev/data-center, they should not have any negative effect on the system.
Next shutdown the vdsm service with "service vdsm stop" (I think, might be service stop vdsm, I don't use CentOS much) and kill any running vdsm processes (ps ax |grep vdsm) The error that I saw was:
MainThread::ERROR::2016-05-13 08:58:38,262::clientIF::128::vds::(__init__) failed to init clientIF, shutting down storage dispatcher MainThread::ERROR::2016-05-13 08:58:38,289::vdsm::171::vds::(run) Exception raised Traceback (most recent call last): File "/usr/share/vdsm/vdsm", line 169, in run serve_clients(log) File "/usr/share/vdsm/vdsm", line 102, in serve_clients cif = clientIF.getInstance(irs, log, scheduler) File "/usr/share/vdsm/clientIF.py", line 193, in getInstance cls._instance = clientIF(irs, log, scheduler) File "/usr/share/vdsm/clientIF.py", line 123, in __init__ self._createAcceptor(host, port) File "/usr/share/vdsm/clientIF.py", line 201, in _createAcceptor port, sslctx) File "/usr/share/vdsm/protocoldetector.py", line 170, in __init__ sock = _create_socket(host, port) File "/usr/share/vdsm/protocoldetector.py", line 40, in _create_socket server_socket.bind(addr[0][4]) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) error: [Errno 98] Address already in use
Please open a bug - this should never happen. https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm oVirt team: Infra Severity: High Vdsm must set the socket option socket.SO_REUSEADDR before binding. Ensuring that only one vdsm instance is running must be done elsewhere if needed, for example using a lock file. Nir