7:45 p.m.
Hello all,
We recently had a network event where we lost access to our storage for a
period of time. The Cluster basically shut down all our VM's and in the
process we had three HV's that went offline and would not communicate
properly with the cluster.
We have since completely reinstalled CentOS on the hosts and attempted to
install them into the cluster with no joy. We've gotten to the point where
we generally get an error message in the web gui:
Stage: Misc Configuration
Host hv-ausa-02 installation failed. Command returned failure code 1 during
SSH session 'root(a)10.1.90.154'.
the following is what we are seeing in the messages log:
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error :
authentication failed: authentication failed
Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231:
error : virNetSASLSessionListMechanisms:390 : internal error: cannot list
SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in
server.c near line 1757)
Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231:
error : remoteDispatchAuthSaslInit:3411 : authentication failed:
authentication failed
Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15226:
error : virNetSocketReadWire:1808 : End of file while reading data:
Input/output error
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error :
authentication failed: authentication failed
Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: 15233:
error : virNetSASLSessionListMechanisms:390 : internal error: cannot list
SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in
server.c near line 1757)
Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15233:
error : remoteDispatchAuthSaslInit:3411 : authentication failed:
authentication failed
Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15226:
error : virNetSocketReadWire:1808 : End of file while reading data:
Input/output error
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error :
authentication failed: authentication failed
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call last):
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219,
in main
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return
tool_command[cmd]["command"](*args)
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/tool/upgrade_300_networks.py", line
83, in upgrade_networks
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks()
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in networks
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get()
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 159, in
get
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection()
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 95, in
_open_qemu_connection
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return utils.retry(libvirtOpen,
timeout=10, sleep=0.2)
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in retry
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func()
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise
libvirtError('virConnectOpenAuth() failed')
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication failed:
authentication failed
Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control process
exited, code=exited status=1
Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop Server
Manager network restoration.
Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual Desktop
Server Manager.
Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed with
result 'dependency'.
Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service entered
failed state.
Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed.
Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root.
Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root.
Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root.
Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root.
Can someone point me in the right direction to resolve this - it seems to
be a SASL issue perhaps?
***
*Mark Steele*
CIO / VP Technical Operations | TelVue Corporation
TelVue - We Share Your Vision
16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054
800.885.8886 x128 | msteele(a)telvue.com | http://www.telvue.com
twitter: http://twitter.com/telvue | facebook:
https://www.facebook.com/telvue