[ovirt-users] node unresponsive after reboot

Thu May 5 14:40:49 UTC 2016

To update the thread here after some offline discussion with Piotr, the
problem was that vdsmd had stopped running and wasn't able to start. It
wasn't reported in the logs that I could see, but I noticed it after trying
to restart it at Piotr's suggestion, after running a 'systemctl'. A
'journalctl -xe' reported:

[root at kvm-ldn-02 vdsm]# journalctl -xe
-- Unit vdsmd.service has failed.
-- 
-- The result is failed.
May 05 15:15:15 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: vdsm:
Running check_is_configured
May 05 15:15:16 kvm-ldn-02.ldn.org sasldblistusers2[19047]: DIGEST-MD5
common mech free
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: Error:
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: One of the
modules is not configured to work with VDSM.
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: To
configure the module use the following:
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: 'vdsm-tool
configure [--module module-name]'.
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: If all
modules are not configured try to use:
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: 'vdsm-tool
configure --force'
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: (The force
flag will stop the module's service and start it
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: afterwards
automatically to load the new configuration.)
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: Current
revision of multipath.conf detected, preserving
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: libvirt is
already configured for vdsm
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: Modules
sebool are not configured
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: vdsm:
stopped during execute check_is_configured task (task returned with error
code 1).
May 05 15:15:16 kvm-ldn-02.ldn.org systemd[1]: vdsmd.service: control
process exited, code=exited status=1
May 05 15:15:16 kvm-ldn-02.ldn.org systemd[1]: Failed to start Virtual
Desktop Server Manager.
-- Subject: Unit vdsmd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 

So I ran a 'vdsm-tool configure --module sebool', and then I was able to
start vdsmd and mom-vdsm. This fixed the connectivity issue and the node is
now back online.

Many thanks to Piotr for all his help.

Cheers,

Cam

On Mon, May 2, 2016 at 3:35 PM, Cam Mac <iucounu at gmail.com> wrote:

> Hi Piotr,
>
> Attached are the vdsm log, the engine log and the supervdsm log. I've
> attached them as a .tgz.
>
> I noticed it is complaining about configuring an interface in one of the
> node logs. It shows as up in the engine web GUI though (and on the command
> line).
>
> Thanks for the help.
>
> -Cam
>
> On Mon, May 2, 2016 at 1:38 PM, Piotr Kliczewski <
> piotr.kliczewski at gmail.com> wrote:
>
>> Cam,
>>
>> Please provide engine and failing vdsm logs.
>>
>> Thanks,
>> Piotr
>>
>> On Sun, May 1, 2016 at 4:05 PM, Cam Mac <iucounu at gmail.com> wrote:
>> > Hi,
>> >
>> > I have a two node + engine ovirt setup, and I was having problems
>> > doing a live migration between nodes. I looked in the vdsm logs and
>> > noticed selinux errors, so I checked the selinux config, and both the
>> > ovirt-engine host and one of the nodes had selinux disabled. So I
>> > thought I would enable it on these two hosts, as it is officially
>> > supported anyway. I started with the node, and put it into maintenance
>> > mode, which interestingly, migrated the VMs off to the other node
>> > without issue. After modifying the selinux config, I then rebooted
>> > that node, which came back up. I then tried to activate the node but
>> > it fails and marks it as unresponsive.
>> >
>> > ----------8<----------
>> >
>> > 2016-04-28 16:34:31,326 INFO
>> > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
>> > Reactor) [29acb18b] Connecting to
>> > kvm-ldn-02/172.16.23.12
>> > 2016-04-28 16:34:31,327 ERROR
>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Command
>> > 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02,
>> > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
>> > hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa',
>> > vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})'
>> > execution failed:
>> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
>> > failed
>> > 2016-04-28 16:34:31,327 ERROR
>> > [org.ovirt.engine.core.vdsbroker.HostMonitoring]
>> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds
>> > runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
>> > Connection failed
>> > 2016-04-28 16:34:31,327 ERROR
>> > [org.ovirt.engine.core.vdsbroker.HostMonitoring]
>> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception:
>> > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
>> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
>> > failed
>> >         at
>> >
>> org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
>> > [vdsbroker.jar:]
>> >         at
>> >
>> org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
>> > [vdsbroker.jar:]
>> >         at
>> >
>> org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
>> > [vdsbroker.jar:]
>> >         at
>> > org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
>> > [dal.jar:]
>> >         at
>> >
>> org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
>> > [vdsbroker.jar:]
>> >         at
>> >
>> org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652)
>> > [vdsbroker.jar:]
>> >         at
>> >
>> org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
>> > [vdsbroker.jar:]
>> >         at
>> >
>> org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
>> > [vdsbroker.jar:]
>> >         at
>> > org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227)
>> > [vdsbroker.jar:]
>> >         at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown
>> > Source) [:1.8.0_71]
>> >         at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > [rt.jar:1.8.0_71]
>> >         at java.lang.reflect.Method.invoke(Method.java:497)
>> > [rt.jar:1.8.0_71]
>> >         at
>> >
>> org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
>> > [scheduler.jar:]
>> >         at
>> > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
>> > [scheduler.jar:]
>> >         at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>> > [quartz.jar:]
>> >         at
>> >
>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>> > [quartz.jar:]
>> > Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
>> > Connection failed
>> >         at
>> >
>> org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157)
>> > [vdsm-jsonrpc-java-client.jar:]
>> >         at
>> >
>> org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114)
>> > [vdsm-jsonrpc-java-client.jar:]
>> >         at
>> > org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73)
>> > [vdsm-jsonrpc-java-client.jar:]
>> >         at
>> >
>> org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:68)
>> > [vdsbroker.jar:]
>> >         at
>> >
>> org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268)
>> > [vdsbroker.jar:]
>> >         at
>> >
>> org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15)
>> > [vdsbroker.jar:]
>> >         at
>> >
>> org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110)
>> > [vdsbroker.jar:]
>> >         ... 14 more
>> >
>> > ----------8<----------
>> >
>> > Any ideas?
>> >
>> > Thanks for any help,
>> >
>> > Cam
>> >
>> > _______________________________________________
>> > Users mailing list
>> > Users at ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/users
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160505/96c0fd2e/attachment-0001.html>