Hello all, We recently had a network event where we lost access to our storage for a period of time. The Cluster basically shut down all our VM's and in the process we had three HV's that went offline and would not communicate properly with the cluster. We have since completely reinstalled CentOS on the hosts and attempted to install them into the cluster with no joy. We've gotten to the point where we generally get an error message in the web gui: Stage: Misc Configuration Host hv-ausa-02 installation failed. Command returned failure code 1 during SSH session 'root@10.1.90.154'. the following is what we are seeing in the messages log: Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: 15233: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15233: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call last): Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return tool_command[cmd]["command"](*args) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/upgrade_300_networks.py", line 83, in upgrade_networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 159, in get Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 95, in _open_qemu_connection Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return utils.retry(libvirtOpen, timeout=10, sleep=0.2) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in retry Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise libvirtError('virConnectOpenAuth() failed') Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control process exited, code=exited status=1 Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop Server Manager network restoration. Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual Desktop Server Manager. Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed with result 'dependency'. Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service entered failed state. Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root. Can someone point me in the right direction to resolve this - it seems to be a SASL issue perhaps? *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue
On Feb 16, 2018 6:47 PM, "Mark Steele" <msteele@telvue.com> wrote: Hello all, We recently had a network event where we lost access to our storage for a period of time. The Cluster basically shut down all our VM's and in the process we had three HV's that went offline and would not communicate properly with the cluster. We have since completely reinstalled CentOS on the hosts and attempted to install them into the cluster with no joy. We've gotten to the point where we generally get an error message in the web gui: Which EL release and which oVirt release are you using? My guess would be latest EL, with an older oVirt? Y. Stage: Misc Configuration Host hv-ausa-02 installation failed. Command returned failure code 1 during SSH session 'root@10.1.90.154'. the following is what we are seeing in the messages log: Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: 15233: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15233: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call last): Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return tool_command[cmd]["command"](* args) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site- packages/vdsm/tool/upgrade_300_networks.py", line 83, in upgrade_networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site- packages/vdsm/libvirtconnection.py", line 159, in get Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site- packages/vdsm/libvirtconnection.py", line 95, in _open_qemu_connection Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return utils.retry(libvirtOpen, timeout=10, sleep=0.2) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in retry Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise libvirtError('virConnectOpenAuth() failed') Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control process exited, code=exited status=1 Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop Server Manager network restoration. Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual Desktop Server Manager. Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed with result 'dependency'. Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service entered failed state. Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root. Can someone point me in the right direction to resolve this - it seems to be a SASL issue perhaps? *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www. facebook.com/telvue _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
We are using CentOS Linux release 7.0.1406 (Core) and oVirt Engine Version: 3.5.0.1-1.el6 We have four other hosts that are running this same configuration already. I took one host out of the cluster (forcefully) that was working and now it will not add back in either - throwing the same SASL error. We are looking at downgrading libvirt as I've seen that somewhere else - is there another version of RH I should be trying? I have a host I can put it on. *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Fri, Feb 16, 2018 at 3:31 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Feb 16, 2018 6:47 PM, "Mark Steele" <msteele@telvue.com> wrote:
Hello all,
We recently had a network event where we lost access to our storage for a period of time. The Cluster basically shut down all our VM's and in the process we had three HV's that went offline and would not communicate properly with the cluster.
We have since completely reinstalled CentOS on the hosts and attempted to install them into the cluster with no joy. We've gotten to the point where we generally get an error message in the web gui:
Which EL release and which oVirt release are you using? My guess would be latest EL, with an older oVirt? Y.
Stage: Misc Configuration Host hv-ausa-02 installation failed. Command returned failure code 1 during SSH session 'root@10.1.90.154'.
the following is what we are seeing in the messages log:
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: 15233: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15233: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call last): Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return tool_command[cmd]["command"](* args) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packa ges/vdsm/tool/upgrade_300_networks.py", line 83, in upgrade_networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packa ges/vdsm/libvirtconnection.py", line 159, in get Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packa ges/vdsm/libvirtconnection.py", line 95, in _open_qemu_connection Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return utils.retry(libvirtOpen, timeout=10, sleep=0.2) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in retry Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise libvirtError('virConnectOpenAuth() failed') Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control process exited, code=exited status=1 Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop Server Manager network restoration. Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual Desktop Server Manager. Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed with result 'dependency'. Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service entered failed state. Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root.
Can someone point me in the right direction to resolve this - it seems to be a SASL issue perhaps?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Fri, Feb 16, 2018 at 11:14 PM, Mark Steele <msteele@telvue.com> wrote:
We are using CentOS Linux release 7.0.1406 (Core) and oVirt Engine Version: 3.5.0.1-1.el6
You are seeing https://bugzilla.redhat.com/show_bug.cgi?id=1444426 , which is a result of a default change of libvirt and was fixed in later versions of oVirt than the one you are using. See patch https://gerrit.ovirt.org/#/c/76934/ for how it was fixed, you can probably configure it manually. Y.
We have four other hosts that are running this same configuration already. I took one host out of the cluster (forcefully) that was working and now it will not add back in either - throwing the same SASL error.
We are looking at downgrading libvirt as I've seen that somewhere else - is there another version of RH I should be trying? I have a host I can put it on.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www. facebook.com/telvue
On Fri, Feb 16, 2018 at 3:31 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Feb 16, 2018 6:47 PM, "Mark Steele" <msteele@telvue.com> wrote:
Hello all,
We recently had a network event where we lost access to our storage for a period of time. The Cluster basically shut down all our VM's and in the process we had three HV's that went offline and would not communicate properly with the cluster.
We have since completely reinstalled CentOS on the hosts and attempted to install them into the cluster with no joy. We've gotten to the point where we generally get an error message in the web gui:
Which EL release and which oVirt release are you using? My guess would be latest EL, with an older oVirt? Y.
Stage: Misc Configuration Host hv-ausa-02 installation failed. Command returned failure code 1 during SSH session 'root@10.1.90.154'.
the following is what we are seeing in the messages log:
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: 15233: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15233: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call last): Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return tool_command[cmd]["command"](*args) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packa ges/vdsm/tool/upgrade_300_networks.py", line 83, in upgrade_networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packa ges/vdsm/libvirtconnection.py", line 159, in get Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packa ges/vdsm/libvirtconnection.py", line 95, in _open_qemu_connection Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return utils.retry(libvirtOpen, timeout=10, sleep=0.2) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in retry Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise libvirtError('virConnectOpenAuth() failed') Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control process exited, code=exited status=1 Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop Server Manager network restoration. Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual Desktop Server Manager. Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed with result 'dependency'. Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service entered failed state. Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root.
Can someone point me in the right direction to resolve this - it seems to be a SASL issue perhaps?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Thank you very much! Question - is upgrading the ovirt installation a matter of just upgrading the engine? Or are there changes that are pushed down to each host / vm? *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Sat, Feb 17, 2018 at 2:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Feb 16, 2018 at 11:14 PM, Mark Steele <msteele@telvue.com> wrote:
We are using CentOS Linux release 7.0.1406 (Core) and oVirt Engine Version: 3.5.0.1-1.el6
You are seeing https://bugzilla.redhat.com/show_bug.cgi?id=1444426 , which is a result of a default change of libvirt and was fixed in later versions of oVirt than the one you are using. See patch https://gerrit.ovirt.org/#/c/76934/ for how it was fixed, you can probably configure it manually. Y.
We have four other hosts that are running this same configuration already. I took one host out of the cluster (forcefully) that was working and now it will not add back in either - throwing the same SASL error.
We are looking at downgrading libvirt as I've seen that somewhere else - is there another version of RH I should be trying? I have a host I can put it on.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Fri, Feb 16, 2018 at 3:31 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Feb 16, 2018 6:47 PM, "Mark Steele" <msteele@telvue.com> wrote:
Hello all,
We recently had a network event where we lost access to our storage for a period of time. The Cluster basically shut down all our VM's and in the process we had three HV's that went offline and would not communicate properly with the cluster.
We have since completely reinstalled CentOS on the hosts and attempted to install them into the cluster with no joy. We've gotten to the point where we generally get an error message in the web gui:
Which EL release and which oVirt release are you using? My guess would be latest EL, with an older oVirt? Y.
Stage: Misc Configuration Host hv-ausa-02 installation failed. Command returned failure code 1 during SSH session 'root@10.1.90.154'.
the following is what we are seeing in the messages log:
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: 15233: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15233: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call last): Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return tool_command[cmd]["command"](*args) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packa ges/vdsm/tool/upgrade_300_networks.py", line 83, in upgrade_networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packa ges/vdsm/libvirtconnection.py", line 159, in get Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packa ges/vdsm/libvirtconnection.py", line 95, in _open_qemu_connection Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return utils.retry(libvirtOpen, timeout=10, sleep=0.2) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in retry Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise libvirtError('virConnectOpenAuth() failed') Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control process exited, code=exited status=1 Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop Server Manager network restoration. Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual Desktop Server Manager. Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed with result 'dependency'. Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service entered failed state. Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root.
Can someone point me in the right direction to resolve this - it seems to be a SASL issue perhaps?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
For a proper upgrade there are specific steps that you follow for each host and the engine. I usually upgrade the hosts first then the engine. If you have spare resources so as to put hosts at maintenance then the upgrade should be seamless. Also i think you need to go strp by step: 3.5 -> 3.6 -> 4.0 ... etc In case you have a similar test setup you may try it first there. On Feb 17, 2018 14:10, "Mark Steele" <msteele@telvue.com> wrote:
Thank you very much!
Question - is upgrading the ovirt installation a matter of just upgrading the engine? Or are there changes that are pushed down to each host / vm?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 <800%20885%208886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www. facebook.com/telvue
On Sat, Feb 17, 2018 at 2:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Feb 16, 2018 at 11:14 PM, Mark Steele <msteele@telvue.com> wrote:
We are using CentOS Linux release 7.0.1406 (Core) and oVirt Engine Version: 3.5.0.1-1.el6
You are seeing https://bugzilla.redhat.com/show_bug.cgi?id=1444426 , which is a result of a default change of libvirt and was fixed in later versions of oVirt than the one you are using. See patch https://gerrit.ovirt.org/#/c/76934/ for how it was fixed, you can probably configure it manually. Y.
We have four other hosts that are running this same configuration already. I took one host out of the cluster (forcefully) that was working and now it will not add back in either - throwing the same SASL error.
We are looking at downgrading libvirt as I've seen that somewhere else - is there another version of RH I should be trying? I have a host I can put it on.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Fri, Feb 16, 2018 at 3:31 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Feb 16, 2018 6:47 PM, "Mark Steele" <msteele@telvue.com> wrote:
Hello all,
We recently had a network event where we lost access to our storage for a period of time. The Cluster basically shut down all our VM's and in the process we had three HV's that went offline and would not communicate properly with the cluster.
We have since completely reinstalled CentOS on the hosts and attempted to install them into the cluster with no joy. We've gotten to the point where we generally get an error message in the web gui:
Which EL release and which oVirt release are you using? My guess would be latest EL, with an older oVirt? Y.
Stage: Misc Configuration Host hv-ausa-02 installation failed. Command returned failure code 1 during SSH session 'root@10.1.90.154'.
the following is what we are seeing in the messages log:
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: 15233: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15233: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call last): Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return tool_command[cmd]["command"](*args) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/upgrade_300_networks.py", line 83, in upgrade_networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 159, in get Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 95, in _open_qemu_connection Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return utils.retry(libvirtOpen, timeout=10, sleep=0.2) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in retry Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise libvirtError('virConnectOpenAuth() failed') Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control process exited, code=exited status=1 Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop Server Manager network restoration. Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual Desktop Server Manager. Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed with result 'dependency'. Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service entered failed state. Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root.
Can someone point me in the right direction to resolve this - it seems to be a SASL issue perhaps?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Thank you Alex. I guess the first step is to get my existing hosts back into the cluster. I'm going to try to manually apply the patch that Yaniv sent over to see if I can get them back in. Mark *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Sat, Feb 17, 2018 at 7:20 AM, Alex K <rightkicktech@gmail.com> wrote:
For a proper upgrade there are specific steps that you follow for each host and the engine.
I usually upgrade the hosts first then the engine. If you have spare resources so as to put hosts at maintenance then the upgrade should be seamless. Also i think you need to go strp by step: 3.5 -> 3.6 -> 4.0 ... etc
In case you have a similar test setup you may try it first there.
On Feb 17, 2018 14:10, "Mark Steele" <msteele@telvue.com> wrote:
Thank you very much!
Question - is upgrading the ovirt installation a matter of just upgrading the engine? Or are there changes that are pushed down to each host / vm?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <800%20885%208886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Sat, Feb 17, 2018 at 2:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Feb 16, 2018 at 11:14 PM, Mark Steele <msteele@telvue.com> wrote:
We are using CentOS Linux release 7.0.1406 (Core) and oVirt Engine Version: 3.5.0.1-1.el6
You are seeing https://bugzilla.redhat.com/show_bug.cgi?id=1444426 , which is a result of a default change of libvirt and was fixed in later versions of oVirt than the one you are using. See patch https://gerrit.ovirt.org/#/c/76934/ for how it was fixed, you can probably configure it manually. Y.
We have four other hosts that are running this same configuration already. I took one host out of the cluster (forcefully) that was working and now it will not add back in either - throwing the same SASL error.
We are looking at downgrading libvirt as I've seen that somewhere else - is there another version of RH I should be trying? I have a host I can put it on.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Fri, Feb 16, 2018 at 3:31 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Feb 16, 2018 6:47 PM, "Mark Steele" <msteele@telvue.com> wrote:
Hello all,
We recently had a network event where we lost access to our storage for a period of time. The Cluster basically shut down all our VM's and in the process we had three HV's that went offline and would not communicate properly with the cluster.
We have since completely reinstalled CentOS on the hosts and attempted to install them into the cluster with no joy. We've gotten to the point where we generally get an error message in the web gui:
Which EL release and which oVirt release are you using? My guess would be latest EL, with an older oVirt? Y.
Stage: Misc Configuration Host hv-ausa-02 installation failed. Command returned failure code 1 during SSH session 'root@10.1.90.154'.
the following is what we are seeing in the messages log:
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: 15233: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15233: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call last): Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return tool_command[cmd]["command"](*args) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/upgrade_300_networks.py", line 83, in upgrade_networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 159, in get Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 95, in _open_qemu_connection Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return utils.retry(libvirtOpen, timeout=10, sleep=0.2) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in retry Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise libvirtError('virConnectOpenAuth() failed') Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control process exited, code=exited status=1 Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop Server Manager network restoration. Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual Desktop Server Manager. Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed with result 'dependency'. Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service entered failed state. Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root.
Can someone point me in the right direction to resolve this - it seems to be a SASL issue perhaps?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Yaniv, I have one of my developers assisting me and we are continuing to run into issues. This is a note from him: Hi, I'm trying to add a host to ovirt, but I'm running into package dependency problems. I have existing hosts that are working and integrated properly, and inspecting those, I am able to match the packages between the new host and the existing, but when I then try to add the new host to ovirt, it fails on reinstall because it's trying to install packages that are later versions. does the installation run list from ovirt-release35 002-1 have unspecified versions? The working hosts use libvirt-1.1.1-29, and vdsm-4.16.7, but it's trying to install vdsm-4.16.30, which requires a higher version of libvirt, at which point, the installation fails. is there some way I can specify which package versions the ovirt install procedure uses? or better yet, skip the package management step entirely? *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Sat, Feb 17, 2018 at 2:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Feb 16, 2018 at 11:14 PM, Mark Steele <msteele@telvue.com> wrote:
We are using CentOS Linux release 7.0.1406 (Core) and oVirt Engine Version: 3.5.0.1-1.el6
You are seeing https://bugzilla.redhat.com/show_bug.cgi?id=1444426 , which is a result of a default change of libvirt and was fixed in later versions of oVirt than the one you are using. See patch https://gerrit.ovirt.org/#/c/76934/ for how it was fixed, you can probably configure it manually. Y.
We have four other hosts that are running this same configuration already. I took one host out of the cluster (forcefully) that was working and now it will not add back in either - throwing the same SASL error.
We are looking at downgrading libvirt as I've seen that somewhere else - is there another version of RH I should be trying? I have a host I can put it on.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Fri, Feb 16, 2018 at 3:31 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Feb 16, 2018 6:47 PM, "Mark Steele" <msteele@telvue.com> wrote:
Hello all,
We recently had a network event where we lost access to our storage for a period of time. The Cluster basically shut down all our VM's and in the process we had three HV's that went offline and would not communicate properly with the cluster.
We have since completely reinstalled CentOS on the hosts and attempted to install them into the cluster with no joy. We've gotten to the point where we generally get an error message in the web gui:
Which EL release and which oVirt release are you using? My guess would be latest EL, with an older oVirt? Y.
Stage: Misc Configuration Host hv-ausa-02 installation failed. Command returned failure code 1 during SSH session 'root@10.1.90.154'.
the following is what we are seeing in the messages log:
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: 15233: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15233: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call last): Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return tool_command[cmd]["command"](*args) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packa ges/vdsm/tool/upgrade_300_networks.py", line 83, in upgrade_networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packa ges/vdsm/libvirtconnection.py", line 159, in get Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packa ges/vdsm/libvirtconnection.py", line 95, in _open_qemu_connection Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return utils.retry(libvirtOpen, timeout=10, sleep=0.2) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in retry Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise libvirtError('virConnectOpenAuth() failed') Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control process exited, code=exited status=1 Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop Server Manager network restoration. Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual Desktop Server Manager. Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed with result 'dependency'. Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service entered failed state. Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root.
Can someone point me in the right direction to resolve this - it seems to be a SASL issue perhaps?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
At this point I'm wondering if there is anyone in the community that freelances and would be willing to provide remote support to resolve this issue? We are running with 1/2 our normal hosts, and not being able to add anymore back into the cluster is a serious problem. Best regards, *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Sat, Feb 17, 2018 at 12:53 PM, Mark Steele <msteele@telvue.com> wrote:
Yaniv,
I have one of my developers assisting me and we are continuing to run into issues. This is a note from him:
Hi, I'm trying to add a host to ovirt, but I'm running into package dependency problems. I have existing hosts that are working and integrated properly, and inspecting those, I am able to match the packages between the new host and the existing, but when I then try to add the new host to ovirt, it fails on reinstall because it's trying to install packages that are later versions. does the installation run list from ovirt-release35 002-1 have unspecified versions? The working hosts use libvirt-1.1.1-29, and vdsm-4.16.7, but it's trying to install vdsm-4.16.30, which requires a higher version of libvirt, at which point, the installation fails. is there some way I can specify which package versions the ovirt install procedure uses? or better yet, skip the package management step entirely?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www. facebook.com/telvue
On Sat, Feb 17, 2018 at 2:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Feb 16, 2018 at 11:14 PM, Mark Steele <msteele@telvue.com> wrote:
We are using CentOS Linux release 7.0.1406 (Core) and oVirt Engine Version: 3.5.0.1-1.el6
You are seeing https://bugzilla.redhat.com/show_bug.cgi?id=1444426 , which is a result of a default change of libvirt and was fixed in later versions of oVirt than the one you are using. See patch https://gerrit.ovirt.org/#/c/76934/ for how it was fixed, you can probably configure it manually. Y.
We have four other hosts that are running this same configuration already. I took one host out of the cluster (forcefully) that was working and now it will not add back in either - throwing the same SASL error.
We are looking at downgrading libvirt as I've seen that somewhere else - is there another version of RH I should be trying? I have a host I can put it on.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Fri, Feb 16, 2018 at 3:31 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Feb 16, 2018 6:47 PM, "Mark Steele" <msteele@telvue.com> wrote:
Hello all,
We recently had a network event where we lost access to our storage for a period of time. The Cluster basically shut down all our VM's and in the process we had three HV's that went offline and would not communicate properly with the cluster.
We have since completely reinstalled CentOS on the hosts and attempted to install them into the cluster with no joy. We've gotten to the point where we generally get an error message in the web gui:
Which EL release and which oVirt release are you using? My guess would be latest EL, with an older oVirt? Y.
Stage: Misc Configuration Host hv-ausa-02 installation failed. Command returned failure code 1 during SSH session 'root@10.1.90.154'.
the following is what we are seeing in the messages log:
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: 15233: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15233: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call last): Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return tool_command[cmd]["command"](*args) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/upgrade_300_networks.py", line 83, in upgrade_networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 159, in get Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 95, in _open_qemu_connection Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return utils.retry(libvirtOpen, timeout=10, sleep=0.2) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in retry Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise libvirtError('virConnectOpenAuth() failed') Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control process exited, code=exited status=1 Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop Server Manager network restoration. Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual Desktop Server Manager. Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed with result 'dependency'. Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service entered failed state. Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root.
Can someone point me in the right direction to resolve this - it seems to be a SASL issue perhaps?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Is it possible that the HostedEngine became corrupted somehow and that is preventing us from adding hosts? Is creating a new hosted engine an option? *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Mon, Feb 19, 2018 at 9:55 AM, Mark Steele <msteele@telvue.com> wrote:
At this point I'm wondering if there is anyone in the community that freelances and would be willing to provide remote support to resolve this issue?
We are running with 1/2 our normal hosts, and not being able to add anymore back into the cluster is a serious problem.
Best regards,
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www. facebook.com/telvue
On Sat, Feb 17, 2018 at 12:53 PM, Mark Steele <msteele@telvue.com> wrote:
Yaniv,
I have one of my developers assisting me and we are continuing to run into issues. This is a note from him:
Hi, I'm trying to add a host to ovirt, but I'm running into package dependency problems. I have existing hosts that are working and integrated properly, and inspecting those, I am able to match the packages between the new host and the existing, but when I then try to add the new host to ovirt, it fails on reinstall because it's trying to install packages that are later versions. does the installation run list from ovirt-release35 002-1 have unspecified versions? The working hosts use libvirt-1.1.1-29, and vdsm-4.16.7, but it's trying to install vdsm-4.16.30, which requires a higher version of libvirt, at which point, the installation fails. is there some way I can specify which package versions the ovirt install procedure uses? or better yet, skip the package management step entirely?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Sat, Feb 17, 2018 at 2:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Feb 16, 2018 at 11:14 PM, Mark Steele <msteele@telvue.com> wrote:
We are using CentOS Linux release 7.0.1406 (Core) and oVirt Engine Version: 3.5.0.1-1.el6
You are seeing https://bugzilla.redhat.com/show_bug.cgi?id=1444426 , which is a result of a default change of libvirt and was fixed in later versions of oVirt than the one you are using. See patch https://gerrit.ovirt.org/#/c/76934/ for how it was fixed, you can probably configure it manually. Y.
We have four other hosts that are running this same configuration already. I took one host out of the cluster (forcefully) that was working and now it will not add back in either - throwing the same SASL error.
We are looking at downgrading libvirt as I've seen that somewhere else - is there another version of RH I should be trying? I have a host I can put it on.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Fri, Feb 16, 2018 at 3:31 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Feb 16, 2018 6:47 PM, "Mark Steele" <msteele@telvue.com> wrote:
Hello all,
We recently had a network event where we lost access to our storage for a period of time. The Cluster basically shut down all our VM's and in the process we had three HV's that went offline and would not communicate properly with the cluster.
We have since completely reinstalled CentOS on the hosts and attempted to install them into the cluster with no joy. We've gotten to the point where we generally get an error message in the web gui:
Which EL release and which oVirt release are you using? My guess would be latest EL, with an older oVirt? Y.
Stage: Misc Configuration Host hv-ausa-02 installation failed. Command returned failure code 1 during SSH session 'root@10.1.90.154'.
the following is what we are seeing in the messages log:
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: 15233: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15233: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call last): Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return tool_command[cmd]["command"](*args) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/upgrade_300_networks.py", line 83, in upgrade_networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 159, in get Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 95, in _open_qemu_connection Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return utils.retry(libvirtOpen, timeout=10, sleep=0.2) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in retry Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise libvirtError('virConnectOpenAuth() failed') Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control process exited, code=exited status=1 Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop Server Manager network restoration. Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual Desktop Server Manager. Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed with result 'dependency'. Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service entered failed state. Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root.
Can someone point me in the right direction to resolve this - it seems to be a SASL issue perhaps?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Tue, Feb 20, 2018 at 12:52 PM, Mark Steele <msteele@telvue.com> wrote:
Is it possible that the HostedEngine became corrupted somehow and that is preventing us from adding hosts?
I doubt that. I still suspect the libvirt auth. issue. Nevertheless, as commented more than once, you are running on somewhat old version with a recent CentOS version. Not sure this combination is tested or anyone's running it.
Is creating a new hosted engine an option?
You could backup and restore to a new HE. Y.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www. facebook.com/telvue
On Mon, Feb 19, 2018 at 9:55 AM, Mark Steele <msteele@telvue.com> wrote:
At this point I'm wondering if there is anyone in the community that freelances and would be willing to provide remote support to resolve this issue?
We are running with 1/2 our normal hosts, and not being able to add anymore back into the cluster is a serious problem.
Best regards,
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Sat, Feb 17, 2018 at 12:53 PM, Mark Steele <msteele@telvue.com> wrote:
Yaniv,
I have one of my developers assisting me and we are continuing to run into issues. This is a note from him:
Hi, I'm trying to add a host to ovirt, but I'm running into package dependency problems. I have existing hosts that are working and integrated properly, and inspecting those, I am able to match the packages between the new host and the existing, but when I then try to add the new host to ovirt, it fails on reinstall because it's trying to install packages that are later versions. does the installation run list from ovirt-release35 002-1 have unspecified versions? The working hosts use libvirt-1.1.1-29, and vdsm-4.16.7, but it's trying to install vdsm-4.16.30, which requires a higher version of libvirt, at which point, the installation fails. is there some way I can specify which package versions the ovirt install procedure uses? or better yet, skip the package management step entirely?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Sat, Feb 17, 2018 at 2:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Feb 16, 2018 at 11:14 PM, Mark Steele <msteele@telvue.com> wrote:
We are using CentOS Linux release 7.0.1406 (Core) and oVirt Engine Version: 3.5.0.1-1.el6
You are seeing https://bugzilla.redhat.com/show_bug.cgi?id=1444426 , which is a result of a default change of libvirt and was fixed in later versions of oVirt than the one you are using. See patch https://gerrit.ovirt.org/#/c/76934/ for how it was fixed, you can probably configure it manually. Y.
We have four other hosts that are running this same configuration already. I took one host out of the cluster (forcefully) that was working and now it will not add back in either - throwing the same SASL error.
We are looking at downgrading libvirt as I've seen that somewhere else - is there another version of RH I should be trying? I have a host I can put it on.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Fri, Feb 16, 2018 at 3:31 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Feb 16, 2018 6:47 PM, "Mark Steele" <msteele@telvue.com> wrote:
Hello all,
We recently had a network event where we lost access to our storage for a period of time. The Cluster basically shut down all our VM's and in the process we had three HV's that went offline and would not communicate properly with the cluster.
We have since completely reinstalled CentOS on the hosts and attempted to install them into the cluster with no joy. We've gotten to the point where we generally get an error message in the web gui:
Which EL release and which oVirt release are you using? My guess would be latest EL, with an older oVirt? Y.
Stage: Misc Configuration Host hv-ausa-02 installation failed. Command returned failure code 1 during SSH session 'root@10.1.90.154'.
the following is what we are seeing in the messages log:
Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15231: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: 15233: error : virNetSASLSessionListMechanisms:390 : internal error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: Internal Error -4 in server.c near line 1757) Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15233: error : remoteDispatchAuthSaslInit:3411 : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: 15226: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call last): Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return tool_command[cmd]["command"](*args) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/upgrade_300_networks.py", line 83, in upgrade_networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in networks Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 159, in get Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 95, in _open_qemu_connection Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return utils.retry(libvirtOpen, timeout=10, sleep=0.2) Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in retry Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func() Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise libvirtError('virConnectOpenAuth() failed') Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication failed: authentication failed Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control process exited, code=exited status=1 Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop Server Manager network restoration. Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual Desktop Server Manager. Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed with result 'dependency'. Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service entered failed state. Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root. Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root.
Can someone point me in the right direction to resolve this - it seems to be a SASL issue perhaps?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
We have found a resolution to this issue which is a bit convoluted - and still does not explain why this started in the first place. Once we have prepped a HV server to be added (all the NIC's are ready, selinux, networkmanager, firewalld, etc have been disabled, we have to do the following: yum install librbd1 rm /etc/yum.repos.d/CentOS-Base.repo (yes... delete the base repo...) vi /etc/yum.repos.d/CentOS-Vault.repo add the following: [vault] name=CentOS-$releasever - Extras #mirrorlist=http://vault.centos.org/?release=$releasever&arch=$basearch&repo=extras baseurl=http://vault.centos.org/centos/7.0.1406/os/x86_64/ gpgcheck=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 vi /etc/yum.repos.d/ovirt-3.5-dependencies.repo edit baseurls as show below: [ovirt-3.5-glusterfs-epel] name=GlusterFS is a clustered file-system capable of scaling to several petabytes. baseurl=https://download.gluster.org/pub/gluster/glusterfs/old-releases/3.6/3.6.1/RH... enabled=1 skip_if_unavailable=1 gpgcheck=0 [ovirt-3.5-glusterfs-noarch-epel] name=GlusterFS is a clustered file-system capable of scaling to several petabytes. baseurl=http://download.gluster.org/pub/gluster/glusterfs/old-releases/3.6/3.6.1/RHE... #baseurl=http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-$rel... enabled=1 skip_if_unavailable=1 gpgcheck=0 yum remove ovirt-release35 yum remove vdsm yum remove libvirt yum install ovirt-release35-002-1 yum install libvirt-1.1.1-29.el7 yum install vdsm-4.16.7-1.gitdb83943.el7 We can then successfully add the HV into the cluster. This is using CentOS 7.0.1406 (Core) as the host OS and oVirt 3.5.0.1-1.el6 If anyone has any questions please feel free to ask. *** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 800.885.8886 x128 | msteele@telvue.com | http://www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook.com/telvue On Tue, Feb 20, 2018 at 11:24 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Tue, Feb 20, 2018 at 12:52 PM, Mark Steele <msteele@telvue.com> wrote:
Is it possible that the HostedEngine became corrupted somehow and that is preventing us from adding hosts?
I doubt that. I still suspect the libvirt auth. issue. Nevertheless, as commented more than once, you are running on somewhat old version with a recent CentOS version. Not sure this combination is tested or anyone's running it.
Is creating a new hosted engine an option?
You could backup and restore to a new HE. Y.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Mon, Feb 19, 2018 at 9:55 AM, Mark Steele <msteele@telvue.com> wrote:
At this point I'm wondering if there is anyone in the community that freelances and would be willing to provide remote support to resolve this issue?
We are running with 1/2 our normal hosts, and not being able to add anymore back into the cluster is a serious problem.
Best regards,
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Sat, Feb 17, 2018 at 12:53 PM, Mark Steele <msteele@telvue.com> wrote:
Yaniv,
I have one of my developers assisting me and we are continuing to run into issues. This is a note from him:
Hi, I'm trying to add a host to ovirt, but I'm running into package dependency problems. I have existing hosts that are working and integrated properly, and inspecting those, I am able to match the packages between the new host and the existing, but when I then try to add the new host to ovirt, it fails on reinstall because it's trying to install packages that are later versions. does the installation run list from ovirt-release35 002-1 have unspecified versions? The working hosts use libvirt-1.1.1-29, and vdsm-4.16.7, but it's trying to install vdsm-4.16.30, which requires a higher version of libvirt, at which point, the installation fails. is there some way I can specify which package versions the ovirt install procedure uses? or better yet, skip the package management step entirely?
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Sat, Feb 17, 2018 at 2:32 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Feb 16, 2018 at 11:14 PM, Mark Steele <msteele@telvue.com> wrote:
We are using CentOS Linux release 7.0.1406 (Core) and oVirt Engine Version: 3.5.0.1-1.el6
You are seeing https://bugzilla.redhat.com/show_bug.cgi?id=1444426 , which is a result of a default change of libvirt and was fixed in later versions of oVirt than the one you are using. See patch https://gerrit.ovirt.org/#/c/76934/ for how it was fixed, you can probably configure it manually. Y.
We have four other hosts that are running this same configuration already. I took one host out of the cluster (forcefully) that was working and now it will not add back in either - throwing the same SASL error.
We are looking at downgrading libvirt as I've seen that somewhere else - is there another version of RH I should be trying? I have a host I can put it on.
*** *Mark Steele* CIO / VP Technical Operations | TelVue Corporation TelVue - We Share Your Vision 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// www.telvue.com twitter: http://twitter.com/telvue | facebook: https://www.facebook .com/telvue
On Fri, Feb 16, 2018 at 3:31 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
> > > On Feb 16, 2018 6:47 PM, "Mark Steele" <msteele@telvue.com> wrote: > > Hello all, > > We recently had a network event where we lost access to our storage > for a period of time. The Cluster basically shut down all our VM's and in > the process we had three HV's that went offline and would not communicate > properly with the cluster. > > We have since completely reinstalled CentOS on the hosts and > attempted to install them into the cluster with no joy. We've gotten to the > point where we generally get an error message in the web gui: > > > Which EL release and which oVirt release are you using? My guess > would be latest EL, with an older oVirt? > Y. > > > Stage: Misc Configuration > Host hv-ausa-02 installation failed. Command returned failure code 1 > during SSH session 'root@10.1.90.154'. > > the following is what we are seeing in the messages log: > > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : > authentication failed: authentication failed > Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: > 15231: error : virNetSASLSessionListMechanisms:390 : internal > error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: > Internal Error -4 in server.c near line 1757) > Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: > 15231: error : remoteDispatchAuthSaslInit:3411 : authentication > failed: authentication failed > Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.761+0000: > 15226: error : virNetSocketReadWire:1808 : End of file while reading data: > Input/output error > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : > authentication failed: authentication failed > Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.962+0000: > 15233: error : virNetSASLSessionListMechanisms:390 : internal > error: cannot list SASL mechanisms -4 (SASL(-4): no mechanism available: > Internal Error -4 in server.c near line 1757) > Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: > 15233: error : remoteDispatchAuthSaslInit:3411 : authentication > failed: authentication failed > Feb 16 11:39:53 hv-ausa-02 libvirtd: 2018-02-16 16:39:53.963+0000: > 15226: error : virNetSocketReadWire:1808 : End of file while reading data: > Input/output error > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirt: XML-RPC error : > authentication failed: authentication failed > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: Traceback (most recent call > last): > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File "/usr/bin/vdsm-tool", > line 219, in main > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return > tool_command[cmd]["command"](*args) > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File > "/usr/lib/python2.7/site-packages/vdsm/tool/upgrade_300_networks.py", > line 83, in upgrade_networks > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: networks = netinfo.networks() > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File > "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 112, in > networks > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = libvirtconnection.get() > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File > "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line > 159, in get > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: conn = _open_qemu_connection() > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File > "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line > 95, in _open_qemu_connection > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return > utils.retry(libvirtOpen, timeout=10, sleep=0.2) > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File > "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1108, in > retry > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: return func() > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: File > "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in > openAuth > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: if ret is None:raise > libvirtError('virConnectOpenAuth() failed') > Feb 16 11:39:53 hv-ausa-02 vdsm-tool: libvirtError: authentication > failed: authentication failed > Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service: control > process exited, code=exited status=1 > Feb 16 11:39:53 hv-ausa-02 systemd: Failed to start Virtual Desktop > Server Manager network restoration. > Feb 16 11:39:53 hv-ausa-02 systemd: Dependency failed for Virtual > Desktop Server Manager. > Feb 16 11:39:53 hv-ausa-02 systemd: Job vdsmd.service/start failed > with result 'dependency'. > Feb 16 11:39:53 hv-ausa-02 systemd: Unit vdsm-network.service > entered failed state. > Feb 16 11:39:53 hv-ausa-02 systemd: vdsm-network.service failed. > Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 10 of user root. > Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 10 of user root. > Feb 16 11:40:01 hv-ausa-02 systemd: Started Session 11 of user root. > Feb 16 11:40:01 hv-ausa-02 systemd: Starting Session 11 of user root. > > Can someone point me in the right direction to resolve this - it > seems to be a SASL issue perhaps? > > *** > *Mark Steele* > CIO / VP Technical Operations | TelVue Corporation > TelVue - We Share Your Vision > 16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054 > <https://maps.google.com/?q=16000+Horizon+Way,+Suite+100+%7C+Mt.+Laurel,+NJ+08054&entry=gmail&source=g> > 800.885.8886 x128 <(800)%20885-8886> | msteele@telvue.com | http:// > www.telvue.com > twitter: http://twitter.com/telvue | facebook: https://www.facebook > .com/telvue > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > >
participants (3)
-
Alex K -
Mark Steele -
Yaniv Kaul