To update the thread here after some offline discussion with Piotr, the problem was that vdsmd had stopped running and wasn't able to start. It wasn't reported in the logs that I could see, but I noticed it after trying to restart it at Piotr's suggestion, after running a 'systemctl'. A 'journalctl -xe' reported:

[root@kvm-ldn-02 vdsm]# journalctl -xe
-- Unit vdsmd.service has failed.
--
-- The result is failed.
May 05 15:15:15 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: vdsm: Running check_is_configured
May 05 15:15:16 kvm-ldn-02.ldn.org sasldblistusers2[19047]: DIGEST-MD5 common mech free
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: Error:
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: One of the modules is not configured to work with VDSM.
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: To configure the module use the following:
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: 'vdsm-tool configure [--module module-name]'.
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: If all modules are not configured try to use:
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: 'vdsm-tool configure --force'
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: (The force flag will stop the module's service and start it
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: afterwards automatically to load the new configuration.)
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: Current revision of multipath.conf detected, preserving
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: libvirt is already configured for vdsm
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: Modules sebool are not configured
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: vdsm: stopped during execute check_is_configured task (task returned with error code 1).
May 05 15:15:16 kvm-ldn-02.ldn.org systemd[1]: vdsmd.service: control process exited, code=exited status=1
May 05 15:15:16 kvm-ldn-02.ldn.org systemd[1]: Failed to start Virtual Desktop Server Manager.
-- Subject: Unit vdsmd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--

So I ran a 'vdsm-tool configure --module sebool', and then I was able to start vdsmd and mom-vdsm. This fixed the connectivity issue and the node is now back online.

Many thanks to Piotr for all his help.

Cheers,

Cam

On Mon, May 2, 2016 at 3:35 PM, Cam Mac <iucounu@gmail.com> wrote:
Hi Piotr,

Attached are the vdsm log, the engine log and the supervdsm log. I've attached them as a .tgz. 

I noticed it is complaining about configuring an interface in one of the node logs. It shows as up in the engine web GUI though (and on the command line).

Thanks for the help.

-Cam

On Mon, May 2, 2016 at 1:38 PM, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Cam,

Please provide engine and failing vdsm logs.

Thanks,
Piotr

On Sun, May 1, 2016 at 4:05 PM, Cam Mac <iucounu@gmail.com> wrote:
> Hi,
>
> I have a two node + engine ovirt setup, and I was having problems
> doing a live migration between nodes. I looked in the vdsm logs and
> noticed selinux errors, so I checked the selinux config, and both the
> ovirt-engine host and one of the nodes had selinux disabled. So I
> thought I would enable it on these two hosts, as it is officially
> supported anyway. I started with the node, and put it into maintenance
> mode, which interestingly, migrated the VMs off to the other node
> without issue. After modifying the selinux config, I then rebooted
> that node, which came back up. I then tried to activate the node but
> it fails and marks it as unresponsive.
>
> ----------8<----------
>
> 2016-04-28 16:34:31,326 INFO
> [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
> Reactor) [29acb18b] Connecting to
> kvm-ldn-02/172.16.23.12
> 2016-04-28 16:34:31,327 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
> (DefaultQuartzScheduler_Worker-32) [ac322cb] Command
> 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02,
> VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
> hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa',
> vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})'
> execution failed:
> org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
> failed
> 2016-04-28 16:34:31,327 ERROR
> [org.ovirt.engine.core.vdsbroker.HostMonitoring]
> (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds
> runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
> Connection failed
> 2016-04-28 16:34:31,327 ERROR
> [org.ovirt.engine.core.vdsbroker.HostMonitoring]
> (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception:
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
> org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
> failed
>         at
> org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
> [vdsbroker.jar:]
>         at
> org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
> [vdsbroker.jar:]
>         at
> org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
> [vdsbroker.jar:]
>         at
> org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
> [dal.jar:]
>         at
> org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
> [vdsbroker.jar:]
>         at
> org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652)
> [vdsbroker.jar:]
>         at
> org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
> [vdsbroker.jar:]
>         at
> org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
> [vdsbroker.jar:]
>         at
> org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227)
> [vdsbroker.jar:]
>         at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown
> Source) [:1.8.0_71]
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0_71]
>         at java.lang.reflect.Method.invoke(Method.java:497)
> [rt.jar:1.8.0_71]
>         at
> org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
> [scheduler.jar:]
>         at
> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
> [scheduler.jar:]
>         at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
> [quartz.jar:]
>         at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
> [quartz.jar:]
> Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
> Connection failed
>         at
> org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157)
> [vdsm-jsonrpc-java-client.jar:]
>         at
> org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114)
> [vdsm-jsonrpc-java-client.jar:]
>         at
> org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73)
> [vdsm-jsonrpc-java-client.jar:]
>         at
> org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:68)
> [vdsbroker.jar:]
>         at
> org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268)
> [vdsbroker.jar:]
>         at
> org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15)
> [vdsbroker.jar:]
>         at
> org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110)
> [vdsbroker.jar:]
>         ... 14 more
>
> ----------8<----------
>
> Any ideas?
>
> Thanks for any help,
>
> Cam
>
> _______________________________________________
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>